How do you handle deploying AI models in strict on-prem client environments?

A lot of the standard tutorials for building AI apps assume you can just plug into an external cloud API or use managed cloud services. But I am realizing that in actual enterprise sectors like banking or healthcare, clients completely refuse to let their data leave their own secure servers.

For anyone working as an FDE or implementation engineer, what does your stack look like when you have to deploy open-source models completely on-prem? Are you mostly wrestling with local hardware constraints, or is it more of a security compliance nightmare?