{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"designing-a-stable-prefix","title":"Ordering the request so the cache can hit","url":"https://addaly.com/learn/context-engineering/designing-a-stable-prefix","position":43,"of":87,"minutes":9,"answer":"Lay the request out strictly from most stable to most variable, put the cache breakpoint at the end of the shared block and a second after append-only history.","keyIdea":"Lay the request out strictly from most stable to most variable, put the cache breakpoint at the end of the shared block and a second after append-only history, and hash the rendered prefix in your logs so an accidental timestamp or a reordered dictionary cannot silently destroy the discount."},"body":{"format":"markdown","text":"## One rule, applied ruthlessly\n\nA prefix cache matches from the first token forward and stops at the first difference. Everything after that difference is recomputed and recharged. So the layout of a request is decided by one question asked of every block: **how often does this change?**\n\nThe order, most stable first:\n\n1. Role, policy, output contract: changes on a deploy.\n2. Tool definitions and schemas: changes on a deploy.\n3. Static exemplars: changes on a deploy.\n4. Organisation- or tenant-level context: changes weekly, shared by many users.\n5. User profile or preferences: changes rarely, but is per-user.\n6. Conversation history: append-only, so stable up to its current end.\n7. Retrieved passages: different every request.\n8. The question: different every request.\n\nBlocks 1 to 3 are shared by every user of your product, which makes them the most valuable cache entry you will ever have. Block 5 splits the cache per user, which is fine and much less valuable. Blocks 7 and 8 will never cache, and that is correct: they should be last.\n\n> Figure: One request, laid out most stable first. Layers, top to bottom: Role, policy, output contract (Changes on a deploy, shared by every user: the most valuable cache entry you will own); Tool definitions, sorted by name (Changes on a deploy. Sort them, or a dictionary reordering costs the whole discount); Static exemplars (Changes on a deploy. Fix the shuffle seed per deploy, never per request); Cache breakpoint (Everything above is shared; below it the cache splits further at each layer); Tenant context, then user (Weekly, then rarely. Per-user from here down, which is fine and worth much less); History, then a second breakpoint (Append-only, so its prefix survives every turn. This is the breakpoint people miss); Passages, then the question (Different every request. Nothing here was ever going to cache). A cache matches from the first token and stops at the first difference, so every block's position is decided by one question: how often does this change. Hash the rendered prefix and log it. If the number of distinct hashes in an hour is far larger than the number of versions you have deployed, something variable has crept upwards.\n\n## The cache killers, in order of how often they occur\n\nEvery one of these is a real bug that has cost a real team real money.\n\n**A timestamp at the top.** \"Today is 2026-09-08 14:22:11.\" Changes every second, invalidating the entire prefix on every call. Fix: truncate to the day, and move it below the stable block.\n\n**A session or request id in the system prompt.** Unique per call by definition. It belongs in your logs, not in the window.\n\n**The user's name in the greeting.** Splits the cache per user for a block that could have been shared by everyone. Move personalisation below the shared prefix.\n\n**Non-deterministic serialisation.** Tool schemas built from a dictionary, serialised with whatever key order the runtime produced. Identical content, different bytes, no cache hit. Fix:\n\n```python\ntools_json = json.dumps(TOOLS, sort_keys=True, separators=(\",\", \":\"))\n```\n\n**Randomised example order.** Someone shuffles few-shot examples to combat position bias. Defensible for quality, fatal for caching. Fix the seed per deploy, not per request.\n\n**Trailing whitespace drift.** A template that emits an extra newline when a variable is empty. Invisible in review, changes the token sequence.\n\nAll six are caught by the same check: hash the rendered prefix on every request and log the hash. If the number of distinct hashes per hour is much larger than the number of prompt versions you have deployed, something variable is above the boundary.\n\n## Where the boundary goes\n\nDraw an explicit line in the assembler between the cacheable region and the variable one, and make it visible in the code rather than implied by ordering.\n\n```python\nSTABLE  = [role_block, tools_block, exemplars_block]   # cache breakpoint here\nVARIABLE = [org_block, user_block, history, passages, question]\n```\n\nWhere your provider supports explicit cache breakpoints, put one at the end of `STABLE` and possibly a second after history, since history is append-only and its prefix is reusable across turns of the same conversation. Where the provider caches automatically, the ordering alone does the work.\n\nThe second breakpoint on history is the one people miss, and in a chat product it is worth more than the first, because it addresses the quadratic billing from the previous lesson directly.\n\n## Minimum lengths and expiry\n\nTwo operational facts that decide whether any of this applies.\n\n**There is a minimum cacheable prefix**, commonly around a thousand tokens. A 400-token system prompt caches nothing at all, and no amount of ordering changes that. If your stable block is short, either accept that caching is not your lever, or reconsider whether more of your context could be made stable.\n\n**Entries expire**, often after a few minutes of not being used, sometimes with an option to extend for a higher write price. This makes cache value a function of your traffic shape. Steady traffic keeps entries warm; bursty traffic pays the write premium repeatedly for entries that expire between bursts.\n\n## The break-even, in one calculation\n\nA cache write costs about 1.25 times base input; a cache read about 0.1 times. So writing and then reading once costs 1.35 against 2.0 for two uncached calls, already a saving. Writing and never reading costs 1.25 against 1.0, a 25% loss.\n\nThe break-even is therefore just under one and a half uses of the same prefix within its lifetime. Almost any shared system prompt clears that easily. A per-user prefix for a user who sends one message a day does not.\n\nThat single calculation tells you which blocks are worth caching and which are not, and it is worth doing with your provider's actual multipliers rather than the illustrative ones here."},"check":{"question":"A team adds `Session: 8f21ac` to the top of their 1,400-token system prompt for debugging. The bill rises noticeably. Why?","options":[{"text":"The extra tokens are billed at the cache-write premium on every call.","correct":false,"tempting":"Names the right premium but the wrong scope; the cost is not the six added tokens but the 1,400 that stopped being cached."},{"text":"Session ids break the provider's request deduplication, so identical requests are billed twice.","correct":false,"tempting":"Invents a deduplication feature; providers bill per request regardless of similarity."},{"text":"The prefix now differs on every call, so nothing after the first token can be read from cache.","correct":true},{"text":"The system prompt fell below the minimum cacheable length once the id was inserted.","correct":false,"tempting":"Adding text increases length; the minimum-length rule cuts the other way."}]},"prev":{"slug":"caching-the-prefix","title":"Caching a stable prefix","url":"https://addaly.com/learn/context-engineering/caching-the-prefix"},"next":{"slug":"cache-hit-rate-as-a-metric","title":"Measuring the cache instead of believing in it","url":"https://addaly.com/learn/context-engineering/cache-hit-rate-as-a-metric"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Ordering the request so the cache can hit', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/designing-a-stable-prefix (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Ordering the request so the cache can hit","courseTitle":"Context Engineering","module":{"index":5,"slug":"cost-cache-and-latency","title":"Cost, caching and latency"},"answer":null,"keyIdea":"Lay the request out strictly from most stable to most variable, put the cache breakpoint at the end of the shared block and a second after append-only history, and hash the rendered prefix in your logs so an accidental timestamp or a reordered dictionary cannot silently destroy the discount.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"One request, laid out most stable first","layers":[{"label":"Role, policy, output contract","detail":"Changes on a deploy, shared by every user: the most valuable cache entry you will own"},{"label":"Tool definitions, sorted by name","detail":"Changes on a deploy. Sort them, or a dictionary reordering costs the whole discount"},{"label":"Static exemplars","detail":"Changes on a deploy. Fix the shuffle seed per deploy, never per request"},{"label":"Cache breakpoint","detail":"Everything above is shared; below it the cache splits further at each layer"},{"label":"Tenant context, then user","detail":"Weekly, then rarely. Per-user from here down, which is fine and worth much less"},{"label":"History, then a second breakpoint","detail":"Append-only, so its prefix survives every turn. This is the breakpoint people miss"},{"label":"Passages, then the question","detail":"Different every request. Nothing here was ever going to cache"}],"caption":"A cache matches from the first token and stops at the first difference, so every block’s position is decided by one question: how often does this change. Hash the rendered prefix and log it. If the number of distinct hashes in an hour is far larger than the number of versions you have deployed, something variable has crept upwards."},"alt":"Figure: One request, laid out most stable first. Layers, top to bottom: Role, policy, output contract (Changes on a deploy, shared by every user: the most valuable cache entry you will own); Tool definitions, sorted by name (Changes on a deploy. Sort them, or a dictionary reordering costs the whole discount); Static exemplars (Changes on a deploy. Fix the shuffle seed per deploy, never per request); Cache breakpoint (Everything above is shared; below it the cache splits further at each layer); Tenant context, then user (Weekly, then rarely. Per-user from here down, which is fine and worth much less); History, then a second breakpoint (Append-only, so its prefix survives every turn. This is the breakpoint people miss); Passages, then the question (Different every request. Nothing here was ever going to cache). A cache matches from the first token and stops at the first difference, so every block’s position is decided by one question: how often does this change. Hash the rendered prefix and log it. If the number of distinct hashes in an hour is far larger than the number of versions you have deployed, something variable has crept upwards."}},"blocks":[{"t":"h2","id":"one-rule-applied-ruthlessly","runs":[{"text":"One rule, applied ruthlessly"}]},{"t":"p","runs":[{"text":"A prefix cache matches from the first token forward and stops at the first difference. Everything after that difference is recomputed and recharged. So the layout of a request is decided by one question asked of every block: "},{"text":"how often does this change?","b":true}]},{"t":"p","runs":[{"text":"The order, most stable first:"}]},{"t":"ol","start":1,"items":[[{"text":"Role, policy, output contract: changes on a deploy."}],[{"text":"Tool definitions and schemas: changes on a deploy."}],[{"text":"Static exemplars: changes on a deploy."}],[{"text":"Organisation- or tenant-level context: changes weekly, shared by many users."}],[{"text":"User profile or preferences: changes rarely, but is per-user."}],[{"text":"Conversation history: append-only, so stable up to its current end."}],[{"text":"Retrieved passages: different every request."}],[{"text":"The question: different every request."}]]},{"t":"p","runs":[{"text":"Blocks 1 to 3 are shared by every user of your product, which makes them the most valuable cache entry you will ever have. Block 5 splits the cache per user, which is fine and much less valuable. Blocks 7 and 8 will never cache, and that is correct: they should be last."}]},{"t":"h2","id":"the-cache-killers-in-order-of-how-often-they-occur","runs":[{"text":"The cache killers, in order of how often they occur"}]},{"t":"p","runs":[{"text":"Every one of these is a real bug that has cost a real team real money."}]},{"t":"p","runs":[{"text":"A timestamp at the top.","b":true},{"text":" “Today is 2026-09-08 14:22:11.” Changes every second, invalidating the entire prefix on every call. Fix: truncate to the day, and move it below the stable block."}]},{"t":"p","runs":[{"text":"A session or request id in the system prompt.","b":true},{"text":" Unique per call by definition. It belongs in your logs, not in the window."}]},{"t":"p","runs":[{"text":"The user’s name in the greeting.","b":true},{"text":" Splits the cache per user for a block that could have been shared by everyone. Move personalisation below the shared prefix."}]},{"t":"p","runs":[{"text":"Non-deterministic serialisation.","b":true},{"text":" Tool schemas built from a dictionary, serialised with whatever key order the runtime produced. Identical content, different bytes, no cache hit. Fix:"}]},{"t":"code","lang":"python","label":"Python","text":"tools_json = json.dumps(TOOLS, sort_keys=True, separators=(\",\", \":\"))"},{"t":"p","runs":[{"text":"Randomised example order.","b":true},{"text":" Someone shuffles few-shot examples to combat position bias. Defensible for quality, fatal for caching. Fix the seed per deploy, not per request."}]},{"t":"p","runs":[{"text":"Trailing whitespace drift.","b":true},{"text":" A template that emits an extra newline when a variable is empty. Invisible in review, changes the token sequence."}]},{"t":"p","runs":[{"text":"All six are caught by the same check: hash the rendered prefix on every request and log the hash. If the number of distinct hashes per hour is much larger than the number of prompt versions you have deployed, something variable is above the boundary."}]},{"t":"h2","id":"where-the-boundary-goes","runs":[{"text":"Where the boundary goes"}]},{"t":"p","runs":[{"text":"Draw an explicit line in the assembler between the cacheable region and the variable one, and make it visible in the code rather than implied by ordering."}]},{"t":"code","lang":"python","label":"Python","text":"STABLE  = [role_block, tools_block, exemplars_block]   # cache breakpoint here\nVARIABLE = [org_block, user_block, history, passages, question]"},{"t":"p","runs":[{"text":"Where your provider supports explicit cache breakpoints, put one at the end of "},{"text":"STABLE","code":true},{"text":" and possibly a second after history, since history is append-only and its prefix is reusable across turns of the same conversation. Where the provider caches automatically, the ordering alone does the work."}]},{"t":"p","runs":[{"text":"The second breakpoint on history is the one people miss, and in a chat product it is worth more than the first, because it addresses the quadratic billing from the previous lesson directly."}]},{"t":"h2","id":"minimum-lengths-and-expiry","runs":[{"text":"Minimum lengths and expiry"}]},{"t":"p","runs":[{"text":"Two operational facts that decide whether any of this applies."}]},{"t":"p","runs":[{"text":"There is a minimum cacheable prefix","b":true},{"text":", commonly around a thousand tokens. A 400-token system prompt caches nothing at all, and no amount of ordering changes that. If your stable block is short, either accept that caching is not your lever, or reconsider whether more of your context could be made stable."}]},{"t":"p","runs":[{"text":"Entries expire","b":true},{"text":", often after a few minutes of not being used, sometimes with an option to extend for a higher write price. This makes cache value a function of your traffic shape. Steady traffic keeps entries warm; bursty traffic pays the write premium repeatedly for entries that expire between bursts."}]},{"t":"h2","id":"the-break-even-in-one-calculation","runs":[{"text":"The break-even, in one calculation"}]},{"t":"p","runs":[{"text":"A cache write costs about 1.25 times base input; a cache read about 0.1 times. So writing and then reading once costs 1.35 against 2.0 for two uncached calls, already a saving. Writing and never reading costs 1.25 against 1.0, a 25% loss."}]},{"t":"p","runs":[{"text":"The break-even is therefore just under one and a half uses of the same prefix within its lifetime. Almost any shared system prompt clears that easily. A per-user prefix for a user who sends one message a day does not."}]},{"t":"p","runs":[{"text":"That single calculation tells you which blocks are worth caching and which are not, and it is worth doing with your provider’s actual multipliers rather than the illustrative ones here."}]}],"check":{"question":[{"text":"A team adds "},{"text":"Session: 8f21ac","code":true},{"text":" to the top of their 1,400-token system prompt for debugging. The bill rises noticeably. Why?"}],"options":[{"text":[{"text":"The extra tokens are billed at the cache-write premium on every call."}],"correct":false,"tempting":[{"text":"Names the right premium but the wrong scope; the cost is not the six added tokens but the 1,400 that stopped being cached."}]},{"text":[{"text":"Session ids break the provider’s request deduplication, so identical requests are billed twice."}],"correct":false,"tempting":[{"text":"Invents a deduplication feature; providers bill per request regardless of similarity."}]},{"text":[{"text":"The prefix now differs on every call, so nothing after the first token can be read from cache."}],"correct":true,"tempting":null},{"text":[{"text":"The system prompt fell below the minimum cacheable length once the id was inserted."}],"correct":false,"tempting":[{"text":"Adding text increases length; the minimum-length rule cuts the other way."}]}],"order":[1,2,3,0],"why":null},"prev":{"slug":"caching-the-prefix","title":"Caching a stable prefix","minutes":8},"next":{"slug":"cache-hit-rate-as-a-metric","title":"Measuring the cache instead of believing in it","minutes":8}}}