English White Tower not firing at enemy units unless garrisoned

Was playing English vs HRE. Rushed the enemy and built the White Tower inside their base where it began firing upon enemy units, for some unknown reason the rate of fire drops and eventually stops entirely unless i garrison my workers inside the tower.
There was enemy units all around the tower but when 0 units were garrisoned it would not fire at all.