I recently played a game like this on Arena. My ally was Khmer, he went Fast Castle into a single Ballista Elephant. I played with a generic civ, I also went Fast Castle into a couple of knights. He had to cut exactly one tree and I bring in my Knights.
My opponent didn’t even have Loom!
I don’t think this is OP though. I mean, you could also just make two petards with your Castle to destroy a single piece of Stone Wall. Heck, you don’t even need a Castle, you can just build a Siege Tower in Castle Age!
And while the strategy is strong on low ELO (such as I), good players will know this strategy and then it’s easy to counter. If your opponent is Khmer, pay extra attention to scouting. If he arrives with a Ballista Elephant, just add a few wall wall tiles or a couple of buildings. The Khmer player, on the other hand, has to invest 650 stone into a Castle.