On rocky unbuild-able terrain I suggest that buildings be build-able on them at 25% build speed - in return the building’s defense increases by 25-100% as a contrast to sandy terrain weakening buildings.
Where as on sandy terrain build speed could be quickened in exchange for the weakened structure
Another option would be to introduce a stone-like terrain /solid flat rock type that buildings may be built on while the current unbuild-able terrain stays as is.
Big nono.
You’d make rocky hills literally unpushable, considering how hard is already to push a castle on a hill with masonry
technically speaking this already exists
you can’t build on the actual cliff terrain, but you can build on the top side of the cliff, and anything attacking up a cliffside suffers the same damage reduction as attacking up a hill.
that and the associated effect of not being able to approach the base of the building from the direction of the cliffside are two effects already making cliffs effective fortification terrain.