Perhaps you have misunderstood this mechanic.
I don’t know if my English expresses my idea properly.
Manor can be deleted at any time by any personal reason at no cost, just like other buildings.
The difference is that rebuilding a manor that you delete cannot provide settlers.
It will be a counter there, starting at 20, meaning the default 20 free settlers will be spawned through the manors. As they are spawned from the manors by, the number will decrease until the number is 0 when the manor no longer spawns settlers. Ideally, it will be 0 when you exactly have all 20 manors.
Once your manor is destroyed by enemies, the counter will increment back, meaning your new manor is sure to a spawn new settler. For example, you built 8 manors, 5 destroyed by the enemy, and 3 remained. The counter will be 12+5=17, which means that the next 17 manors will spawn settlers, and you’ll get a total of 25 free settlers in this game.
The above performance is no different from the current gameplay.
The only difference is that once the manor is deleted by Yourself, this counter does Not Increase to prevent abuse. For example, you built 8 manors, 5 deleted by yourself, 1 destroyed by the enemy, and 2 remained. The counter will still be 12+1=13, which means that the next 13 manors will still be able to spawn settlers, but the 14th manor and beyond will not be able, and you will only get 21 free settlers in total. If the enemy hadn’t destroyed that 1 manor, so the counter will keep 12 and it would not be able to spawn settlers from the 13th, and there would still be only 20 free settlers in total.
Deletion is a self-determined action, not a forced situation like being destroyed. The previous manors have fulfilled the benefits, so even if the later estates can only provide population, it is the player’s own responsibility to bear the consequences.