Leave castles alone please. Unique units are just there to make the game look cool for the most part. You’re not meant to build these cool epic structures and nice little little castles everywhere. This isn’t a city builder game. You are meant to rush.
You meant to end the game in feudal age with a scout rush/archer rush then GG in 1v1 on a very open map.
If you want "epic’ battles just make a team game of black forest or michi. Plenty of unique units in those games. Or play some cool scenario with loads of castles and very rich resources. Leave castles as a rare building.