Prompt caching, and the ordering it demands

What is being cached

Module two established that attention caches each token’s keys and values so they are not recomputed at every generation step. Prompt caching extends the idea across requests: if two requests begin with the same tokens, the keys and values for that shared prefix are identical, so the second request can start from the stored state instead of recomputing prefill.

Nothing about the model changes. It is the same computation, skipped because the answer is already known.

What it saves

Two distinct savings, and both are large where prompts are long and repetitive:

Latency. Prefill on a 20,000-token prompt is the dominant part of time to first token. A cache hit removes most of it.

Money. Providers price cached input tokens well below fresh ones: discounts of the order of 50 to 90 per cent are typical, with the exact figures varying by provider and changing over time, so check the current pricing page rather than trusting a number in a course.

For an application with a long, stable system prompt and short user messages, this is frequently the single largest cost reduction available, and it requires no change to model or quality.

The rule: identical prefix, byte for byte

The cache is keyed on the exact token sequence from the start. One different character anywhere in the prefix and everything after that point is a miss.

Which produces one design rule with several consequences:

Order your prompt from most stable to most variable.

1. system prompt and policy      never changes         cacheable
2. tool definitions              changes on release    cacheable
3. few-shot examples             changes rarely        cacheable
4. retrieved documents           changes per request   not cacheable
5. conversation history          grows                 partly cacheable
6. current user message          always new            not cacheable

And one rule about what not to do:

Never put anything variable near the top. A timestamp, a session id, a user’s name, a randomised greeting: any of these at the start of the system prompt destroys the cache for the entire request, every request. This is a real and common mistake, and its symptom is exactly the one in module two’s check: costs and latency that are inexplicably high given the prompt size. If you need the time, put it at the end, next to the user message.

Growing conversations cache well

A multi-turn conversation has a naturally growing shared prefix: turn 5’s prompt begins with everything from turns 1 to 4. Providers with automatic caching handle this well, and each turn only pays fresh prefill for what is new.

The thing that breaks it is editing history: summarising old turns, or dropping the oldest message from a sliding window. Both rewrite the prefix and invalidate everything. If you must trim, trim in large infrequent steps rather than every turn, so you pay the full cost occasionally instead of continuously.

Details that vary and must be checked

Caches have a time to live, often measured in minutes, so traffic that arrives in bursts benefits far more than traffic spread thinly. Some providers cache automatically, some require an explicit marker in the request, and some charge a small premium to write a cache entry, which means caching a prefix used only once costs more than not caching it.

None of these are things to memorise. They are things to look up for your provider before you design around them, and to re-check, because they change.

Self-hosting

If you run your own serving stack, this is prefix caching and vLLM has it built in. With paged attention, two requests sharing a prefix can point at the same physical memory blocks, which saves memory as well as compute. Both are free and open source, and for a workload with a long shared system prompt the effect on throughput can be several-fold.

The measurement

Most providers report cached and uncached token counts in the response. Log both, and compute your hit rate. A hit rate you assumed was 90 per cent and is actually 4 per cent is a common and expensive discovery, and it always has a specific cause: usually a single variable field sitting somewhere near the top of a prompt that nobody has read in months.