This also annoys me, I would propose Ctrl+Click to be the keybind, as it basically has that functionality already. If you select a Villager and a Scout for example Ctrl+Click on the Scout drops all Villagers from the selection.
I would also put out a reminder about the TC garrisoning whereby the villagers stand in a line like hockey substitutes and you can only ungarrison to the north of the TC unless I have missed something?
There definitely needs to be more flexibility with ungarrisoning.