So this is a minor but very irritating issue IDK its intentional or not
Villagers will get involved into any fight happening nearby! and you cant stop them from doing that. if you re-task them, they will again start attacking after few seconds.
Example :
If an Explorer is fighting to a treasure near hunts. The villager will not work untill the treasure guardian is dead or the villager is dead.
This is same for ur unit fighting an enemy unit nearby.
IMO They should just mind their business if they are not attacked upon. Also they should ignore the fight if a military unit is fighting nearby (Just build the wall!! that musk can fight hussar on his own!)