My suggestion:
Wall cost scales with length from 5-10, pillars are free with 0 construction time.
Construction time is doubled on wall segments to make up for that.
The overall wall cost and construction time is now about as high as before if you don’t delete the pillars.
Stone walls are a separate building available in fortress age (maybe some of the defensive cards makes them available earlier). The construction time is 2x as high. They cost coin in addition to wood so the price is double.
Generally:
Fortifications (maybe all buildings) get -33% construction time around TC and +33% construction time everywhere else until Fortress Age. Than constructing defences is twice as fast around your base than in a forward position. (Could be nice for all AoE games tbh.)