Sometimes idle villagers are not selectable using the hotkey.
A way to reliably reproduce this bug is assigning villagers to an already occuppied farm. They reach the farm, become idle and then are no longer selectable by the hotkey.
1 Like