Why don't we just knock down buildings with expensive rents?

All these luxury towers keep increasing rents so why are no parties advocating knocking them down? I don't care if they're replaced with affordable housing, just get rid of them!

I did hear there may be unintended consequences but I can't imagine what they might be. If we just get rid of flats with high rents, average rents would mathematically have to fall.

I'm most happy with the Greens who want to ensure that nothing but affordable housing is built but why is no one advocating going further and reversing the mistake of allowing non-affordable housing to be built?