Will large LLMs become accessible on-prem? The cost/benefit math isn't adding up.

I'm at a hardware SME where we write system software. Management is wary of public cloud tools leaking source code, so on-prem is the only option. A few of us run local models. I have an RTX 4070 mobile with 8GB VRAM running starcoder2:7b under ollama. It's useful but nowhere near as good as Copilot.

I'm thinking about persuading management to invest in on-prem hardware. Options range from roughly 4k for a dual RTX 3090 setup, which barely runs 70b models, to enterprise-grade systems that cost six figures. The math is brutal when you factor in power, cooling, and maintenance.
But there's also the software layer. Even if we buy hardware, we still need a way to route requests across models, cache responses, and observe performance. Most of the tooling assumes you're calling cloud APIs. Self-hosting means building that infrastructure yourself or finding something that works in air-gapped environments.

Has anyone successfully deployed on-prem LLM infrastructure at a mid-sized company? What was the actual cost and what broke? Did you find any tooling that made self-hosting manageable?