The armory is necessary because it affects a wide range of units and not just your own units, such as mersenaries or natives. The base stats of units can be reduced accordingly as explained above. If they never have an arsenal, at least give them this: