Maybe Town Centers and Towers could have slightly increased damage against cavalry, for example a small +1 or +2 damage bonus vs cavalry.
The reason is that cavalry, especially heavy cavalry, can dive even TC-protected woodlines or nearby economy, kill villagers, and escape while taking very little damage. This is not only because of their stats, but also because their movement speed lets them spend much less time under TC or tower fire compared to infantry.
Cavalry can also chase down villagers because of their speed, while infantry usually cannot do that effectively. Because of this, it feels more logical from a game design perspective that infantry should have a stronger role in pushes and siege situations, while cavalry remains better for mobility, raids, and map control.
A small anti-cavalry damage bonus could help compensate for cavalry’s speed advantage without removing cavalry’s identity. The goal is not to make defensive structures hard-counter cavalry, but to make repeated dives under TC and tower fire slightly more punishable.
This would probably not remove the current cavalry meta by itself, but it could be a small step toward making infantry-based armies and foot-based pushes more viable.
For context, I usually play around Solo Ranked Diamond 3 / Conqueror 1, so this is not really about struggling to defend cavalry raids personally. It is more about the overall balance and how cavalry mobility currently interacts with defensive structures.