Just destroyed every castle house in my town :( Any tips?

Ok, reluctantly, I destroyed every house (all castle!) in my town. I was very proud of my completed castle town with all slots filled.

Why did I do it? I realized after doing the math that to level up my town from level 9 to 10 would take probably 300+ hours of grinding, if not more. It would take a huge amount of 300 and 400 prestige upgrades to equal 36,500 prestige to get a level 10 town (and then level 10 blacksmith).

So, the best solution was to destroy all houses and then rebuild destroy, as suggested by others.

The first house was over 5000 gold to destroy but each next one gets less, down to 25 to destroy the last one. I also destroyed my level 5 workshop to move the workshop. Rebuilt another workshop in a different location which will also help with prestige.

Lumber is my shortage at the moment, and marble. Trying to decide if I should build wood (111 ptg), stone (125 ptg), or castle (160 ptg) houses for prestige. Any tips?

Pic