I’ve been using Claude Code heavily for development and wanted to better understand how prompt caching affects real API costs in longer coding sessions.

Disclosure: I’m affiliated with ByteUnion AI, one of the gateways included below. This is not an independent benchmark, and I’m not presenting it as proof that one provider is better than another. I’m sharing the setup and numbers mainly to get feedback on the methodology.

For 7 days, I ran roughly 15.6K matched Claude Code API requests across three routes:

Anthropic direct API

A larger API gateway

ByteUnion AI

All three exposed the same Claude model. I wasn’t testing model quality. I was mainly looking at the delivery layer: cache behavior, reliability, latency, and billed cost.

Approximate usage per route:

545K uncached input tokens

21M cache creation tokens

870M cache read tokens

6.57M output tokens

\~5,194 attempted responses

A request was counted as successful if it returned a completed stream without an API or transport error. Latency refers to complete response time, not time to first token.

The main thing I took away is that headline input/output token pricing doesn’t explain the whole bill for Claude Code workloads.

With hundreds of millions of cache-read tokens, cache creation and cache-read pricing can meaningfully change the economics of a long coding session.

The larger gateway also had the lowest cache-hit rate and highest P95 latency in this test. One possible explanation is weaker cache affinity when requests move between upstream routes, but I don’t have provider-side routing data, so that’s only a hypothesis.

Limitations: I didn’t measure output quality, task correctness, model provenance, silent fallback, multiple regions, or high-concurrency behavior. A successful API response also doesn’t necessarily mean the coding task was completed successfully.

For people running LLM workloads through different gateways:

Do you factor cache-hit rate and cache-read behavior into cost comparisons, or do you mostly look at headline token pricing, latency, and rate limits?

What would make this comparison more useful or credible?

Prompt caching across LLM gateways: notes from ~15.6K Claude Code API requests