One of the most common mistakes with archers. It’s only worth waiting for upgrades if you just beat the opponent with your uptime. Then you should wait until the upgrades kick in.
And then you meet onagers and mangonels
If your opponent is spamming cav from 2 stables and you’re going crossbows, consider adding more production buildings to get ahead in numbers if you have the eco to keep ranges busy producing non-stop, this is essential unless you want to mix in pikes/monks as a counter, since equal numbers of crossbows will not do well against cavalry for reasons already stated here.