As demuslim is repeatedly saying it, I agree, when you anyhow destroy a piece of wall, only 1 unit of it falls (when there is no gate). And your units trying to go through it are in an extreme choke point. If you destroyed it with rams, your own rams will block your units in that small choke point. You basically cant get through if enemy is blocking it with even few units. You either have to destroy more and more, to create a wider opening, or attack where the gate is. (but again, there is not a gate all the time).
Destroying wall segment should take down at least 2 of them or 3, even if it is not a gate.