Selection of vils not working properly

All times I played the game in different modes and with different civs when I select one or more vils and try to select a building to be built I have to try 2 or 3 times before the vils really start doing what they are supposed to.
Resources are more then enough in all tries.