It’s not really speculation and “arm chair analysis” but based on experience when playing the game and watching replays. AoE 2’s casting tools do a good job of indicating the LOS of each player to show what they can see and how they could react.
As @thieftdp8498 pointed out, AoE 2 was released in 1999 and forward castles are around since day 1.
In reality, you only have a limited place to drop a forward castle on an open map because your opponent propably already walled parts of their economy off and continued to expand their base towards the enemy, so the economy should be save unless you place farms close to your walls.
As the Viper vs Daut showmatch on Acropolish was brought up earlier to show how “op” castles are, let me add this breakdown by Twest which just confirmes what we’ve already said how a castle is not the end of the world:
And as OP previously mentioned AoE 3’s protection zones, let me drop this screenshot where I tried to build a Fort (the AoE 3 equivalent of a castle) near the opponent’s base:
Notice how the tooltip says “too close” and “first Town Center” - Ensemble didn’t want to kill forward plays entirely, but wanted to give players some room to breathe so Teuton Deathstars and Persian TC Drop can’t happen. That’s why I’m perfectly fine to build the Fort below the cliff which still is close to the opponent’s base. Not to mention that Forts themselves are already “nerfed” compared to castles bc they’re only available via a home city card. So that’s the AoE 3 version of Castles costing 650 Stone.