Why do my villagers always 'pool' instead of building something?

Hi, I have played the game about 350 hours, about half the time whenever I try to build something, the villagers don’t build it, instead they stand over the structure framework idle. Whenever I want to build something, am I supposed to spam click several times in order to get them to actually build it? I attached a recording to show what I’m talking about. It is extremely frustrating. You think you built a granary and are gathering from it, only to find out your villagers are all idle standing around an unbuilt granary structure. Happens many many times per game, every game.
villagerskeeppooling.mythrec (6.3 MB)

That IS weird. I haven’t seen that to be a problem. Yeah, around 2:15 or so there’s a villager who just stands inside the granary outline, but doesn’t build it.