The problems here is a downgrade from Age of Empries 2 DE mechanics with AOE IV. When a garrisoned Town Center or Castle (or tower) gets destroyed with villagers inside, they predictably come out at top right corner. This is really easy to surround and crowd leading to a bloodbath for the archers.
This for one gives a massive advantage to players who are defending with friendly zones to the North East against attack move cavalry and infantry. One team in a top right vs bottom left matchup will get garrisons lost and the other will save the villagers more easily. Really it makes sense just to set a direction and have them come out on the side towards the rally. They’ve already given feudal and castle attacks more effectiveness with ram assembly by units so let the old defensive villager juggling resume a bit.