The title explains it, I guess.
Can’t imagine this to be a feature, as it makes things quite unbalanced/random.
When a tc, tower, village, keep or similar is under attack, the outcome of the fight depends on which side of the map you start, pretty much.
It can be to the defender’s advantage or disadvantage and makes the game less competitive, since your units will always exit the building towards the top right corner (in default cam setting).
There is a workaround for villagers giving them a command before pressing the “seek shelter” button, to force them to change direction on ungarrison command.
So, after all, the code for units to exit to a different side exists.
I think similar (unwanted?) mechanics exist for rams, have to test this again.
If not on the list to be fixed problems yet, please add.