I played as Franks, and I tried to rush the Viking with cheap Castle drop and knights, and failed terribly, because consider the Viking economic spike, which means more effective economy and military production. By the time I started the raid, I was greeted by a nasty surprise: they had a Castle too, along with pikes, knights and some rams.
He reached Castle Age before me so yea. Is Vikings that hard to raid in the early game? At this point, I realized I messed with the wrong civ.
Part of it would be a matter of game decisions. Traditionally with Franks you would want to pressure the enemy as hard as possible in the early Feudal Age, in an attempt to crash their economy, and possibly end the game. The mode of entry into Feudal combat is complex as you have a variety of choices for starting off and transitioning to other forms of play. Scouting is a lifesaver. Note that while Frankish Scouts are optimized for early Feudal combat does not mean that they are the best starting choices. It is best to work your way into learning them under fire. It is valid to use a Fast Castle strategy, but civilizations like the Vikings will get more out of it because they are economically oriented. Sometimes the only way you can stand a chance against an opponent is to outeco them, and the Vikings player knows that you have terrifying cavalry.
A smart opponent will not be idle and will work to wall off your entry points, bait you into their town center, or control your ability to attack certain points. Do not be afraid to use towers in your initial attack to tie them down (But do so judiciously, when the map favors towers). Remember that they have faster villagers than you, so chasing them down with MAA will not be easy, but that does not mean that they might not be useful when looking at your choices.
Typically, the Castle Age is viewed more through the lens of being mid-game and divided into sections with different modes of play throughout them.
Castle drop is only effective when you can drop it at a critical spot where you can block gold piles and probably a TC, otherwise it will be considered as a failure because you pulled many villagers that will delay your eco a lot.
It is all about strategy and skill difference. Franks are faster than Viking if you FC