Spearmen and walls both compete for the same resource, that being wood so if you’re facing knights you need to put even more workers on wood for botht he spearmen and the walls… The Knights from the Rus and French can just go along the outside of your walls and throw torches to slowly wear stuff down forcing you to take workers off something to repair it and also spend more wood. While they can spend all their wood on rams since knights don’t cost wood.
Then when they get through your defenses they can also help destroy your defensive buildings by throwing torches.
If you have spent the resources on a wall in, then you should just be safe from knights until they spend all the resources to get a ram up. And even then I think that spearmen should be able to throw torches over the palisade at the ram so that to make a good attack they need archers, knights and rams, but that is a different discussion.