Byzantine Castles Cause Framerate Drop?

Not sure if this is unique to me or not, but having a Byzantine castle anywhere on the map (even out of view, concealed by fog of war) causes my framerate to drop from 50-60 to about 5-10. Destroying the last castle instantly fixes the problem.