Walker lore randomly changes. It was established that walkers would simply walk around walls if they couldn't see or hear anything on the other side. Here, walkers surround the walled neighborhood and stay in place for days, even when no one has been within hundreds of feet of a walled section. The episode even contradicts itself because (after just a few hours) walkers had forgotten all about someone they knew was hiding just out of arm's reach.