Caching a stable prefix

What is being cached

When a model reads your prompt, it computes internal key and value tensors for every token — the KV cache. That work is the prefill, and it dominates the cost and latency of a long prompt.

Prompt caching stores those tensors for a prefix of your input and reuses them on the next call that starts with the same tokens. A hit skips the arithmetic entirely, which is why the discount is large rather than marginal.

The numbers, roughly

Exact multipliers change every few months; the shape does not. Check your provider's current pricing page rather than a course.

• Anthropic: explicit, via cache_control markers. A cache write costs about 1.25x the base input rate; a read costs about 0.1x — a 90% discount. Default lifetime five minutes, refreshed on each hit, with a longer option at a higher write price.
• OpenAI: automatic, no markers. Cached input tokens are discounted substantially (historically 50%, more on some models). No write premium, which makes it strictly free money, and correspondingly less controllable.
• Google: both an explicit cache with per-token-hour storage pricing, and implicit caching on recent models.
• DeepSeek and several open-weight hosts: cache hits priced around a tenth of misses.
• Self-hosted (vLLM, SGLang, TensorRT-LLM): automatic prefix caching is on or one flag away. There is no billing, only throughput — and the throughput gain is the same mechanism.

Do the arithmetic once

A support assistant: 20,000 tokens of stable system prompt, tool schemas and policy documents, then a 500-token question. 100,000 calls a month. Base rate $3 per million input tokens.

Without caching: 20,500 × 100,000 = 2.05 billion tokens = $6,150.

With caching at a 95% hit rate, writes at 1.25x and reads at 0.1x:

• writes: 5,000 × 20,000 = 100M at $3.75/M = $375
• reads: 95,000 × 20,000 = 1.9B at $0.30/M = $570
• variable tail: 100,000 × 500 at $3/M = $150

Total $1,095. About 5.6x. Push the stable share higher — a 100k-token document queried repeatedly — and 10x is ordinary. The latency win comes free: time to first token on a hit typically drops by more than half on long prompts.

The rule that decides whether you get any of it

The cache is prefix-exact. Matching starts at token one and stops at the first difference. Everything after the difference is recomputed.

So one changed byte near the top costs you the entire benefit. The usual culprits:

# every one of these is a cache miss on every single call
SYSTEM = f"Current time: {datetime.now()}\n..."      # changes per second
SYSTEM = f"Request ID: {uuid4()}\n..."               # changes per request
SYSTEM = f"You are helping {user.name}.\n..."        # changes per user
tools  = sorted_by(random.shuffle(tool_list))        # changes per call
json.dumps(schema)                                   # key order not guaranteed

That last one bites quietly. If you serialise tool schemas from a dict without sort_keys=True, the byte order can vary across processes and your cache hit rate becomes a function of which worker handled the request.

Order by stability

1. system prompt              never changes
2. tool definitions           changes on deploy
3. static few-shot examples   changes on deploy
4. large shared documents     changes on ingest
--------------------------------- cache breakpoint
5. conversation history       grows, append-only
6. current date, user context per call
7. retrieved chunks           per call
8. the user's question        per call

History sits at position 5 because it is append-only: turn 12 leaves turns 1 through 11 byte-identical, so a growing conversation keeps hitting the cache for everything before the newest turn. Prepend anything to history — a running summary, a re-sorted list — and you break that.

Things that will surprise you

• Minimum length. Many providers will not cache below about 1,024 tokens. A short prompt gets nothing.
• Short lifetimes. Five minutes is common. Low-traffic endpoints pay the write premium repeatedly and save nothing; you can end up slightly worse off. If a route gets one call every ten minutes, do not cache it, or use a longer TTL tier and check the sums.
• Scope. Caches are per API key or organisation. They are not shared between customers, and a cache hit does not reveal anything across accounts.
• Images and files count. A stable image in the prefix caches like text.

Verify, do not assume

Every major API returns cache token counts in the response usage. Log them, and put the hit rate on a dashboard next to spend.

u = response.usage
log.info("cache_read=%s cache_write=%s uncached=%s",
         u.cache_read_input_tokens, u.cache_creation_input_tokens, u.input_tokens)

A hit rate that reads 3% when you expected 95% is the single most common silent cost bug in production LLM systems, and it takes one log line to find.