I can't wrap my head around API inferencing costs per token. Especially input cost.

Okay I am wondering if someone can explain to me the thing hiding in plain sight... because I just can't wrap my head around it.

Go to API OpenRouter and pick any model. As an example I will use a typical cost of $0.50 million tokens input / $1.50 million token output.

Okay.

Now we know that in many cases (like you developers hooking up API directly to CLI coding assistants) we usually have long running conversations with LLMs. You send a prompt. LLM reponds back. You send another prompt... etc etc

Not only this, a single LLM turn makes multiple tool calls. For each tool call the ENTIRE conversation is sent back via along with the tool call results, processed, and returned along with the next tool call.

What you get is an eye watering usage bill - easily using a million tokens in a single LLM turn (according to my OpenRouter billing). Of course, OpenRouter just pass on the bill from whatever provider its using with a small percentage fee.

Now here is part I can't seem to reconcile:

• What about prompt / prefix caching? vLLM docs literally call it a free lunch that eliminates pretty much all compute cost of previously seen token. But apparently, only Anthropic, OpenAI, and to some extent Google "opt in". So why do other providers not take this into account?!
• Is input token cost realistic? I've seen claims that when run locally input tokens are calculated up to thousands of times faster than output. So why so little difference between the input and ouput cost in API pricing? Of course, I understand that the more input tokens are added, the higher the compute per output token, but this is drastically reduced with KV caching.

I am sorry if this is pretty obvious for someone out there but I haven't been able to try self hosting any models (no hardware locally, and haven't gotten around to trying runpod or rented GPUs yet. I am just wondering if there is a pretty obvious answer I am missing here.