Prompt caching, and the ordering rule that pays for itself

The waste that caching removes

A support assistant sends the same 1,800 tokens on every single call: system prompt, tool definitions, eight examples, a policy document. Only the last 200 tokens (the customer’s message) differ. Across a million calls you have paid for 1.8 billion identical tokens.

Prompt caching removes most of that bill. The mechanism is worth understanding because it dictates how you must write the prompt.

When a model processes your input it computes a key and value vector for every token (the KV cache) and this prefill is most of the cost and latency of a request with a long prompt. Because attention is causal, the KV entries for the first 1,800 tokens depend only on those tokens. If the next request begins with exactly the same 1,800 tokens, that work can be reused verbatim.

The rule that follows

The match is on an exact prefix, byte for byte, from position zero. One changed character anywhere in the prefix invalidates everything after it.

So the ordering of your prompt is now an engineering constraint, not a style choice:

1. System prompt and tool definitions: identical on every call
2. Few-shot examples: identical
3. Large static documents: identical per document
4. Conversation history: grows, but only at the end
5. The current user turn: always different

Get this backwards and you get nothing. A prompt that opens with Today is 2026-09-06, 14:32:11. has a new prefix every second and will never hit the cache. Put the timestamp after the static block, or round it to the day, or pass it as part of the user turn.

The same applies to a user’s name, a session id, or a randomised greeting anywhere near the top. This is the single most common reason a team enables caching and sees no saving.

The numbers

Rates vary by provider, but the shape is consistent:

• Cache write: around 1.25× the normal input price, paid once when the prefix is first stored
• Cache read: around 0.1× the normal input price, a 90 per cent discount
• Time to live: typically five minutes, refreshed on each hit; longer tiers exist at a higher write price

Work an example. 1,800 static tokens plus 200 variable, at $3 per million input.

• Uncached: 2,000 × $3/M = $0.006 per call. A million calls: $6,000.
• Cached, with a good hit rate: 1,800 × $0.30/M + 200 × $3/M = $0.00114. A million calls: $1,140, plus a negligible number of writes.

That is an 80 per cent reduction on a feature nobody rewrote. Latency falls too, because prefill is skipped: commonly a 30 to 60 per cent cut in time to first token on prompts of this size.

Where it silently fails

Traffic too sparse. A five-minute TTL means fewer than one request every five minutes gives you writes and no reads, which is more expensive than not caching. Caching pays at sustained volume, not on a prototype.

Minimum length. Providers set a floor (often around 1,024 tokens) below which nothing is cached. A 400-token prompt gets no benefit no matter how static it is.

A moving history. In a chat, appending turns keeps the prefix stable and the cache warm. But summarising or trimming old turns rewrites the middle of the prompt and invalidates everything downstream. If you trim, trim rarely and in large chunks, so you pay the re-write once rather than every turn.

Tool definitions that reorder. If your tool list is built from a dictionary or a set, its serialisation order may change between processes. Sort it. A cache miss caused by non-deterministic JSON key order is a miserable bug to find.

Verify it, do not assume it

The response tells you the truth. Log cache_read_input_tokens and cache_creation_input_tokens on every call and put the ratio on a dashboard.

u = r.usage
hit = u.cache_read_input_tokens / max(1, u.cache_read_input_tokens
                                       + u.cache_creation_input_tokens
                                       + u.input_tokens)
metrics.gauge("llm.cache_hit_ratio", hit, tags=[f"feature:{feature}"])

A hit ratio that was 0.85 last week and is 0.02 today means somebody added a variable to the top of the prompt. Without the metric, you find out from the invoice a month later.

One last thing worth saying plainly: caching changes nothing about the output. Same tokens in, same distribution out. It is a pure cost and latency optimisation, which makes it unusual: most of the decisions in this course are trade-offs, and this one is close to free if you order the prompt correctly on the first day.