Do buildings just de-upgrade or disappear in your Settlements?

I've been having a hard time keeping track of the upgrades and stuff in my settlements and I swear I've upgraded some buildings like 4 times to B-rank and I swear some of the buildings are missing when I come back.