Villagers Stop Building

Just started playing multiplayer, could be lag, but Ive got a fairly decent comp for it…
If I select villagers to go build something, on some multiplayer servers the villys will just go over the spot where they’re supposed to build then not build anything. They do register as idle and its a matter of reselecting them to build on the spot they were on. Just annoying really.