Cavalry and horse archers just run around my spearmen and ignore bowmen since they deal 0 damage.
Horse archers are even worse. nothing can get close, they just kite.
Perhaps horses and the rider could get separate health and you can shoot down the horse so they knights have to walk with they lose their horse?
You can’t engage cavalry directly with your spearmen and archers are not a hard-counter. At no point should you overreact (try chasing or taking an engagement) against mounted units if you haven’t got any appropriate counters, and never chase horse archers; those guys are designed to punish that activity.
In the early game, you want to use what little you’ve got as a deterrent to cavalry harassment. Keep your spearmen / archers around your villagers, tower up near gold / food, especially when RNG gave you really unsafe resource positions.
In the mid-late game, get proper counters like crossbows and better meat-shield spearmen / M@As and use thoughtfully placed walls around the map to limit the number of effective paths mounted units can make use of against you. As for the late game, 10 bombards can also help against horse archers, but I digress.
That is a bit too large of a change for the Dev team to invest into at this point, and it could open up its own list of problems.