That is true for Watch towers, Guard towers (especially when garrisoned) will take down Mangonels through vills repairing (unless there’s like, 4 vills repairing in which case it’s a big resource drain).
To be fair, Elephant Archers don’t really struggle with sniping Siege. The point of ranged units is to have a mass, and once you do have a couple of Elephant Archers out on the field, you can pretty reliably snipe Mangonels already. One could also argue that their faster shooting is basically the anti-Siege bonus. Ironically, I’d say their Castle UT is also great against Siege, as you can comfortably tank a couple of Mangonel shots and then have them healed up over time. 11
In these situations I don’t think one should be afraid of Dravidian stable being that barren, as when you’re only against Mangonels, a Scout Cavalry no matter the upgrades will counter the Mangonels. A couple of them and you can still just run in and snipe well enough while your other army tackles whatever else the enemy is making.