Temple destruction

Can anyone maybe explain me how the temple destruction works, cause it feels like, the more time i spend inside a temple the more rooms get destroyed. But when i leave the temple allots instantly it destroys only about 1 room. I completely bricked my temple btw.