Cost break even between LLM APIs and self hosted RTX 6000 Pro clusters for sustained inference

Hi all,

I am trying to estimate the cost break even point between frontier model APIs, cloud GPU rentals, and a self hosted RTX 6000 Pro based cluster for sustained LLM inference.

Target workload:

• A few thousand users

• Peak concurrency around 256 requests per minute

• Heavy use of tool calls and multi step agent workflows

• Stable daily traffic

• Qwen 235b for the llm and various voice models (asr, tts, ..)

Hardware configuration under consideration:

• 2 servers

• 8x RTX 6000 Pro per server (16 GPUs total)

When I estimate token based API usage at this scale, monthly costs increase very quickly. When I estimate long term AWS GPU rental at near 24/7 utilization, the yearly cost approaches the full hardware purchase price.

On many subreddits it is often stated that APIs are almost always cheaper and that local hosting is mainly for other reasons such as privacy or control. I am trying to understand under what concrete workload assumptions that statement remains true.

For those who run sustained production inference on RTX 6000 class GPUs, at what utilization level or traffic profile do APIs or long term cloud rentals remain more cost effective than owning the hardware?