Why would you *not* castle?

This isn’t a rhetorical question but something I’ve been wondering for a long time. I searched this sub and found a lot of “why would you castle,” but I can’t find the opposite. I’m a poor (but improving) and very defensive player, and I’m not sure why I shouldn’t just castle every time it’s possible.