{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"caching-the-prefix","title":"Caching a stable prefix","url":"https://addaly.com/learn/context-engineering/caching-the-prefix","position":42,"of":87,"minutes":8,"answer":"Caching matches from the first token forward, so anything variable near the top throws the whole discount away.","keyIdea":"Caching matches from the first token forward, so anything variable near the top throws the whole discount away."},"body":{"format":"markdown","text":"## What is being cached\n\nWhen a model reads your prompt, it computes internal key and value tensors for every token: the KV cache. That work is the prefill, and it dominates the cost and latency of a long prompt.\n\nPrompt caching stores those tensors for a prefix of your input and reuses them on the next call that starts with the same tokens. A hit skips the arithmetic entirely, which is why the discount is large rather than marginal.\n\n## The numbers, roughly\n\nExact multipliers change every few months; the shape does not. Check your provider's current pricing page rather than a course.\n\n- **Anthropic:** explicit, via `cache_control` markers. A cache write costs about 1.25x the base input rate; a read costs about 0.1x: a 90% discount. Default lifetime five minutes, refreshed on each hit, with a longer option at a higher write price.\n- **OpenAI:** automatic, no markers. Cached input tokens are discounted substantially (historically 50%, more on some models). No write premium, which makes it strictly free money, and correspondingly less controllable.\n- **Google:** both an explicit cache with per-token-hour storage pricing, and implicit caching on recent models.\n- **DeepSeek and several open-weight hosts:** cache hits priced around a tenth of misses.\n- **Self-hosted (vLLM, SGLang, TensorRT-LLM):** automatic prefix caching is on or one flag away. There is no billing, only throughput, and the throughput gain is the same mechanism.\n\n## Do the arithmetic once\n\nA support assistant: 20,000 tokens of stable system prompt, tool schemas and policy documents, then a 500-token question. 100,000 calls a month. Base rate $3 per million input tokens.\n\nWithout caching: 20,500 × 100,000 = 2.05 billion tokens = **$6,150**.\n\nWith caching at a 95% hit rate, writes at 1.25x and reads at 0.1x:\n\n- writes: 5,000 × 20,000 = 100M at $3.75/M = $375\n- reads: 95,000 × 20,000 = 1.9B at $0.30/M = $570\n- variable tail: 100,000 × 500 at $3/M = $150\n\nTotal **$1,095**. About 5.6x. Push the stable share higher (a 100k-token document queried repeatedly) and 10x is ordinary. The latency win comes free: time to first token on a hit typically drops by more than half on long prompts.\n\n## The rule that decides whether you get any of it\n\n**The cache is prefix-exact.** Matching starts at token one and stops at the first difference. Everything after the difference is recomputed.\n\nSo one changed byte near the top costs you the entire benefit. The usual culprits:\n\n```python\n# every one of these is a cache miss on every single call\nSYSTEM = f\"Current time: {datetime.now()}\\n...\"      # changes per second\nSYSTEM = f\"Request ID: {uuid4()}\\n...\"               # changes per request\nSYSTEM = f\"You are helping {user.name}.\\n...\"        # changes per user\ntools  = sorted_by(random.shuffle(tool_list))        # changes per call\njson.dumps(schema)                                   # key order not guaranteed\n```\n\nThat last one bites quietly. If you serialise tool schemas from a dict without `sort_keys=True`, the byte order can vary across processes and your cache hit rate becomes a function of which worker handled the request.\n\n## Order by stability\n\n```\n1. system prompt              never changes\n2. tool definitions           changes on deploy\n3. static few-shot examples   changes on deploy\n4. large shared documents     changes on ingest\n--------------------------------- cache breakpoint\n5. conversation history       grows, append-only\n6. current date, user context per call\n7. retrieved chunks           per call\n8. the user's question        per call\n```\n\nHistory sits at position 5 because it is append-only: turn 12 leaves turns 1 through 11 byte-identical, so a growing conversation keeps hitting the cache for everything before the newest turn. Prepend anything to history (a running summary, a re-sorted list) and you break that.\n\n## Things that will surprise you\n\n- **Minimum length.** Many providers will not cache below about 1,024 tokens. A short prompt gets nothing.\n- **Short lifetimes.** Five minutes is common. Low-traffic endpoints pay the write premium repeatedly and save nothing; you can end up slightly worse off. If a route gets one call every ten minutes, do not cache it, or use a longer TTL tier and check the sums.\n- **Scope.** Caches are per API key or organisation. They are not shared between customers, and a cache hit does not reveal anything across accounts.\n- **Images and files count.** A stable image in the prefix caches like text.\n\n## Verify, do not assume\n\nEvery major API returns cache token counts in the response usage. Log them, and put the hit rate on a dashboard next to spend.\n\n```python\nu = response.usage\nlog.info(\"cache_read=%s cache_write=%s uncached=%s\",\n         u.cache_read_input_tokens, u.cache_creation_input_tokens, u.input_tokens)\n```\n\nA hit rate that reads 3% when you expected 95% is the single most common silent cost bug in production LLM systems, and it takes one log line to find."},"check":{"question":"A team enables prompt caching on a 25,000-token prefix. Spend does not move, and the usage logs show cache reads near zero. The prefix begins with a line reading `Session: 7f3a-...` followed by the unchanged system prompt, tools and policy documents. Why is nothing cached?","options":[{"text":"The session line is short, so it falls below the provider's minimum cacheable length and caching is skipped for the whole request.","correct":false,"tempting":"Minimum lengths are real and this prefix is long enough overall, which makes the threshold feel like the culprit, but the minimum applies to the cached span rather than to the first line."},{"text":"Cache entries are keyed on the full prompt including the variable tail, so any per-call content anywhere prevents a hit.","correct":false,"tempting":"It correctly notices that variable content breaks caching, but the matching is prefix-based, so content after the breakpoint is harmless."},{"text":"Matching runs from the first token forward, so a per-session identifier at position zero means nothing after it can ever match.","correct":true},{"text":"Caching only applies to the system role, and the policy documents are being sent as user messages.","correct":false,"tempting":"Roles do feel like natural cache units, so restricting the discount to one of them sounds plausible, but the mechanism operates on the token sequence rather than on message roles."}]},"prev":{"slug":"prefill-decode-and-where-the-time-goes","title":"Where the milliseconds go","url":"https://addaly.com/learn/context-engineering/prefill-decode-and-where-the-time-goes"},"next":{"slug":"designing-a-stable-prefix","title":"Ordering the request so the cache can hit","url":"https://addaly.com/learn/context-engineering/designing-a-stable-prefix"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Caching a stable prefix', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/caching-the-prefix (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Caching a stable prefix","courseTitle":"Context Engineering","module":{"index":5,"slug":"cost-cache-and-latency","title":"Cost, caching and latency"},"answer":null,"keyIdea":"Caching matches from the first token forward, so anything variable near the top throws the whole discount away.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-is-being-cached","runs":[{"text":"What is being cached"}]},{"t":"p","runs":[{"text":"When a model reads your prompt, it computes internal key and value tensors for every token: the KV cache. That work is the prefill, and it dominates the cost and latency of a long prompt."}]},{"t":"p","runs":[{"text":"Prompt caching stores those tensors for a prefix of your input and reuses them on the next call that starts with the same tokens. A hit skips the arithmetic entirely, which is why the discount is large rather than marginal."}]},{"t":"h2","id":"the-numbers-roughly","runs":[{"text":"The numbers, roughly"}]},{"t":"p","runs":[{"text":"Exact multipliers change every few months; the shape does not. Check your provider’s current pricing page rather than a course."}]},{"t":"ul","items":[[{"text":"Anthropic:","b":true},{"text":" explicit, via "},{"text":"cache_control","code":true},{"text":" markers. A cache write costs about 1.25x the base input rate; a read costs about 0.1x: a 90% discount. Default lifetime five minutes, refreshed on each hit, with a longer option at a higher write price."}],[{"text":"OpenAI:","b":true},{"text":" automatic, no markers. Cached input tokens are discounted substantially (historically 50%, more on some models). No write premium, which makes it strictly free money, and correspondingly less controllable."}],[{"text":"Google:","b":true},{"text":" both an explicit cache with per-token-hour storage pricing, and implicit caching on recent models."}],[{"text":"DeepSeek and several open-weight hosts:","b":true},{"text":" cache hits priced around a tenth of misses."}],[{"text":"Self-hosted (vLLM, SGLang, TensorRT-LLM):","b":true},{"text":" automatic prefix caching is on or one flag away. There is no billing, only throughput, and the throughput gain is the same mechanism."}]]},{"t":"h2","id":"do-the-arithmetic-once","runs":[{"text":"Do the arithmetic once"}]},{"t":"p","runs":[{"text":"A support assistant: 20,000 tokens of stable system prompt, tool schemas and policy documents, then a 500-token question. 100,000 calls a month. Base rate $3 per million input tokens."}]},{"t":"p","runs":[{"text":"Without caching: 20,500 × 100,000 = 2.05 billion tokens = "},{"text":"$6,150","b":true},{"text":"."}]},{"t":"p","runs":[{"text":"With caching at a 95% hit rate, writes at 1.25x and reads at 0.1x:"}]},{"t":"ul","items":[[{"text":"writes: 5,000 × 20,000 = 100M at $3.75/M = $375"}],[{"text":"reads: 95,000 × 20,000 = 1.9B at $0.30/M = $570"}],[{"text":"variable tail: 100,000 × 500 at $3/M = $150"}]]},{"t":"p","runs":[{"text":"Total "},{"text":"$1,095","b":true},{"text":". About 5.6x. Push the stable share higher (a 100k-token document queried repeatedly) and 10x is ordinary. The latency win comes free: time to first token on a hit typically drops by more than half on long prompts."}]},{"t":"h2","id":"the-rule-that-decides-whether-you-get-any-of-it","runs":[{"text":"The rule that decides whether you get any of it"}]},{"t":"p","runs":[{"text":"The cache is prefix-exact.","b":true},{"text":" Matching starts at token one and stops at the first difference. Everything after the difference is recomputed."}]},{"t":"p","runs":[{"text":"So one changed byte near the top costs you the entire benefit. The usual culprits:"}]},{"t":"code","lang":"python","label":"Python","text":"# every one of these is a cache miss on every single call\nSYSTEM = f\"Current time: {datetime.now()}\\n...\"      # changes per second\nSYSTEM = f\"Request ID: {uuid4()}\\n...\"               # changes per request\nSYSTEM = f\"You are helping {user.name}.\\n...\"        # changes per user\ntools  = sorted_by(random.shuffle(tool_list))        # changes per call\njson.dumps(schema)                                   # key order not guaranteed"},{"t":"p","runs":[{"text":"That last one bites quietly. If you serialise tool schemas from a dict without "},{"text":"sort_keys=True","code":true},{"text":", the byte order can vary across processes and your cache hit rate becomes a function of which worker handled the request."}]},{"t":"h2","id":"order-by-stability","runs":[{"text":"Order by stability"}]},{"t":"code","lang":"","label":"Text","text":"1. system prompt              never changes\n2. tool definitions           changes on deploy\n3. static few-shot examples   changes on deploy\n4. large shared documents     changes on ingest\n--------------------------------- cache breakpoint\n5. conversation history       grows, append-only\n6. current date, user context per call\n7. retrieved chunks           per call\n8. the user's question        per call"},{"t":"p","runs":[{"text":"History sits at position 5 because it is append-only: turn 12 leaves turns 1 through 11 byte-identical, so a growing conversation keeps hitting the cache for everything before the newest turn. Prepend anything to history (a running summary, a re-sorted list) and you break that."}]},{"t":"h2","id":"things-that-will-surprise-you","runs":[{"text":"Things that will surprise you"}]},{"t":"ul","items":[[{"text":"Minimum length.","b":true},{"text":" Many providers will not cache below about 1,024 tokens. A short prompt gets nothing."}],[{"text":"Short lifetimes.","b":true},{"text":" Five minutes is common. Low-traffic endpoints pay the write premium repeatedly and save nothing; you can end up slightly worse off. If a route gets one call every ten minutes, do not cache it, or use a longer TTL tier and check the sums."}],[{"text":"Scope.","b":true},{"text":" Caches are per API key or organisation. They are not shared between customers, and a cache hit does not reveal anything across accounts."}],[{"text":"Images and files count.","b":true},{"text":" A stable image in the prefix caches like text."}]]},{"t":"h2","id":"verify-do-not-assume","runs":[{"text":"Verify, do not assume"}]},{"t":"p","runs":[{"text":"Every major API returns cache token counts in the response usage. Log them, and put the hit rate on a dashboard next to spend."}]},{"t":"code","lang":"python","label":"Python","text":"u = response.usage\nlog.info(\"cache_read=%s cache_write=%s uncached=%s\",\n         u.cache_read_input_tokens, u.cache_creation_input_tokens, u.input_tokens)"},{"t":"p","runs":[{"text":"A hit rate that reads 3% when you expected 95% is the single most common silent cost bug in production LLM systems, and it takes one log line to find."}]}],"check":{"question":[{"text":"A team enables prompt caching on a 25,000-token prefix. Spend does not move, and the usage logs show cache reads near zero. The prefix begins with a line reading "},{"text":"Session: 7f3a-...","code":true},{"text":" followed by the unchanged system prompt, tools and policy documents. Why is nothing cached?"}],"options":[{"text":[{"text":"The session line is short, so it falls below the provider’s minimum cacheable length and caching is skipped for the whole request."}],"correct":false,"tempting":[{"text":"Minimum lengths are real and this prefix is long enough overall, which makes the threshold feel like the culprit, but the minimum applies to the cached span rather than to the first line."}]},{"text":[{"text":"Cache entries are keyed on the full prompt including the variable tail, so any per-call content anywhere prevents a hit."}],"correct":false,"tempting":[{"text":"It correctly notices that variable content breaks caching, but the matching is prefix-based, so content after the breakpoint is harmless."}]},{"text":[{"text":"Matching runs from the first token forward, so a per-session identifier at position zero means nothing after it can ever match."}],"correct":true,"tempting":null},{"text":[{"text":"Caching only applies to the system role, and the policy documents are being sent as user messages."}],"correct":false,"tempting":[{"text":"Roles do feel like natural cache units, so restricting the discount to one of them sounds plausible, but the mechanism operates on the token sequence rather than on message roles."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"prefill-decode-and-where-the-time-goes","title":"Where the milliseconds go","minutes":9},"next":{"slug":"designing-a-stable-prefix","title":"Ordering the request so the cache can hit","minutes":9}}}