It’s probably this bug: "Back to work" is fully unresponsive if vills were garrisoned with command queue (shift). [RtoR DLC]
Meaning it’s not always unresponsive, but only when the garrisoned units did so after a “shift” command, such as “move here, then garrison”. I reported it almost a year ago and it’s still there 