There are many reasons. First, the Samurai have burned the Bandits homes and possibly food stores. Most likely, all the Bandits escaped with just the clothes on their back, their horses, and swords. All the supplies they needed, even to rebuild their fort, are in the village. Second, they wanted revenge, although, at one point we see two Bandits try to desert and be killed for their troubles. So, while some of the Bandits did want to just leave the village alone, it really was the Bandit Leader, with his ability to scare the others into obeying, who decides they are in it to the finish. Third, during these times there would have been a feeling to fulfill one's role. Japanese culture is conformist and it would have been shocking, even insulting to the Bandit Leader that samurai and villagers were acting outside their roles and working together. The Bandits role should be to pillage the village, and they try desperately to fulfill that role, maybe more so because their foes are not.