Poles are pretty weak on open maps, where their farming eco is easy to pressure and they are pretty strong on closed maps, where they can set up their farming rather peacefully.
If they were able to garrison villagers in them (like Khmer garrisoning in their houses) they would have some extra space to hide in and maybe wait until the army arrives. This would make them a bit easier to play on open maps and would not really change them on closed maps.
I think having space for 5 villagers would be sufficient since this also is the population space it gives.