Can't do the garrison trick for the action of building (video attached)

Age of Empires IV IV - Report a Bug
1

How to reproduce:

  1. Ask Villager to build something.
  2. Use Seek Shelter for the Villager to garrison in the Town Center.
  3. Use Return all Villagers to work (from Seek Shelter).
  4. Notice the Villager goes out of the Town Center, but is idling.

This trick is often used to bypass having to walk around the Town Center. It works for gathering resources, but doesn’t seem to work for building structures.