Ordering the request so the cache can hit

One rule, applied ruthlessly

A prefix cache matches from the first token forward and stops at the first difference. Everything after that difference is recomputed and recharged. So the layout of a request is decided by one question asked of every block: how often does this change?

The order, most stable first:

1. Role, policy, output contract: changes on a deploy.
2. Tool definitions and schemas: changes on a deploy.
3. Static exemplars: changes on a deploy.
4. Organisation- or tenant-level context: changes weekly, shared by many users.
5. User profile or preferences: changes rarely, but is per-user.
6. Conversation history: append-only, so stable up to its current end.
7. Retrieved passages: different every request.
8. The question: different every request.

Blocks 1 to 3 are shared by every user of your product, which makes them the most valuable cache entry you will ever have. Block 5 splits the cache per user, which is fine and much less valuable. Blocks 7 and 8 will never cache, and that is correct: they should be last.

The cache killers, in order of how often they occur

Every one of these is a real bug that has cost a real team real money.

A timestamp at the top. “Today is 2026-09-08 14:22:11.” Changes every second, invalidating the entire prefix on every call. Fix: truncate to the day, and move it below the stable block.

A session or request id in the system prompt. Unique per call by definition. It belongs in your logs, not in the window.

The user’s name in the greeting. Splits the cache per user for a block that could have been shared by everyone. Move personalisation below the shared prefix.

Non-deterministic serialisation. Tool schemas built from a dictionary, serialised with whatever key order the runtime produced. Identical content, different bytes, no cache hit. Fix:

tools_json = json.dumps(TOOLS, sort_keys=True, separators=(",", ":"))

Randomised example order. Someone shuffles few-shot examples to combat position bias. Defensible for quality, fatal for caching. Fix the seed per deploy, not per request.

Trailing whitespace drift. A template that emits an extra newline when a variable is empty. Invisible in review, changes the token sequence.

All six are caught by the same check: hash the rendered prefix on every request and log the hash. If the number of distinct hashes per hour is much larger than the number of prompt versions you have deployed, something variable is above the boundary.

Where the boundary goes

Draw an explicit line in the assembler between the cacheable region and the variable one, and make it visible in the code rather than implied by ordering.

STABLE  = [role_block, tools_block, exemplars_block]   # cache breakpoint here
VARIABLE = [org_block, user_block, history, passages, question]

Where your provider supports explicit cache breakpoints, put one at the end of STABLE and possibly a second after history, since history is append-only and its prefix is reusable across turns of the same conversation. Where the provider caches automatically, the ordering alone does the work.

The second breakpoint on history is the one people miss, and in a chat product it is worth more than the first, because it addresses the quadratic billing from the previous lesson directly.

Minimum lengths and expiry

Two operational facts that decide whether any of this applies.

There is a minimum cacheable prefix, commonly around a thousand tokens. A 400-token system prompt caches nothing at all, and no amount of ordering changes that. If your stable block is short, either accept that caching is not your lever, or reconsider whether more of your context could be made stable.

Entries expire, often after a few minutes of not being used, sometimes with an option to extend for a higher write price. This makes cache value a function of your traffic shape. Steady traffic keeps entries warm; bursty traffic pays the write premium repeatedly for entries that expire between bursts.

The break-even, in one calculation

A cache write costs about 1.25 times base input; a cache read about 0.1 times. So writing and then reading once costs 1.35 against 2.0 for two uncached calls, already a saving. Writing and never reading costs 1.25 against 1.0, a 25% loss.

The break-even is therefore just under one and a half uses of the same prefix within its lifetime. Almost any shared system prompt clears that easily. A per-user prefix for a user who sends one message a day does not.

That single calculation tells you which blocks are worth caching and which are not, and it is worth doing with your provider’s actual multipliers rather than the illustrative ones here.