{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"prompt-caching","title":"Prompt caching, and the ordering rule that pays for itself","url":"https://addaly.com/learn/building-with-ai/prompt-caching","position":15,"of":80,"minutes":8,"answer":"Prompt caching matches an exact byte prefix from position zero, so every static part must come before every variable part, and one timestamp near the top destroys the entire discount.","keyIdea":"Prompt caching matches an exact byte prefix from position zero, so every static part must come before every variable part, and one timestamp near the top destroys the entire discount."},"body":{"format":"markdown","text":"## The waste that caching removes\n\nA support assistant sends the same 1,800 tokens on every single call: system prompt, tool definitions, eight examples, a policy document. Only the last 200 tokens (the customer's message) differ. Across a million calls you have paid for 1.8 billion identical tokens.\n\nPrompt caching removes most of that bill. The mechanism is worth understanding because it dictates how you must write the prompt.\n\nWhen a model processes your input it computes a key and value vector for every token (the **KV cache**) and this prefill is most of the cost and latency of a request with a long prompt. Because attention is causal, the KV entries for the first 1,800 tokens depend only on those tokens. If the next request begins with exactly the same 1,800 tokens, that work can be reused verbatim.\n\n## The rule that follows\n\n**The match is on an exact prefix, byte for byte, from position zero.** One changed character anywhere in the prefix invalidates everything after it.\n\nSo the ordering of your prompt is now an engineering constraint, not a style choice:\n\n1. System prompt and tool definitions: identical on every call\n2. Few-shot examples: identical\n3. Large static documents: identical per document\n4. Conversation history: grows, but only at the end\n5. The current user turn: always different\n\nGet this backwards and you get nothing. A prompt that opens with `Today is 2026-09-06, 14:32:11.` has a new prefix every second and will never hit the cache. Put the timestamp after the static block, or round it to the day, or pass it as part of the user turn.\n\nThe same applies to a user's name, a session id, or a randomised greeting anywhere near the top. This is the single most common reason a team enables caching and sees no saving.\n\n## The numbers\n\nRates vary by provider, but the shape is consistent:\n\n- **Cache write**: around 1.25× the normal input price, paid once when the prefix is first stored\n- **Cache read**: around 0.1× the normal input price, a 90 per cent discount\n- **Time to live**: typically five minutes, refreshed on each hit; longer tiers exist at a higher write price\n\nWork an example. 1,800 static tokens plus 200 variable, at $3 per million input.\n\n- Uncached: 2,000 × $3/M = $0.006 per call. A million calls: **$6,000**.\n- Cached, with a good hit rate: 1,800 × $0.30/M + 200 × $3/M = $0.00114. A million calls: **$1,140**, plus a negligible number of writes.\n\nThat is an 80 per cent reduction on a feature nobody rewrote. Latency falls too, because prefill is skipped: commonly a 30 to 60 per cent cut in time to first token on prompts of this size.\n\n## Where it silently fails\n\n**Traffic too sparse.** A five-minute TTL means fewer than one request every five minutes gives you writes and no reads, which is *more* expensive than not caching. Caching pays at sustained volume, not on a prototype.\n\n**Minimum length.** Providers set a floor (often around 1,024 tokens) below which nothing is cached. A 400-token prompt gets no benefit no matter how static it is.\n\n**A moving history.** In a chat, appending turns keeps the prefix stable and the cache warm. But summarising or trimming old turns rewrites the middle of the prompt and invalidates everything downstream. If you trim, trim rarely and in large chunks, so you pay the re-write once rather than every turn.\n\n**Tool definitions that reorder.** If your tool list is built from a dictionary or a set, its serialisation order may change between processes. Sort it. A cache miss caused by non-deterministic JSON key order is a miserable bug to find.\n\n## Verify it, do not assume it\n\nThe response tells you the truth. Log `cache_read_input_tokens` and `cache_creation_input_tokens` on every call and put the ratio on a dashboard.\n\n> Figure: The same prompt, three orderings, three bills. Bars (dollars per million calls): Caching off 6000; Caching on, timestamp at the top 7350; Caching on, static block first 1140. 1,800 static tokens and 200 variable ones, at $3 per million input, $0.30 cached read and $3.75 cached write. A timestamp at the top gives every call a new prefix, so every call is a cache write and caching costs more than not caching. The saving is in the ordering, not the feature flag.\n\n```python\nu = r.usage\nhit = u.cache_read_input_tokens / max(1, u.cache_read_input_tokens\n                                       + u.cache_creation_input_tokens\n                                       + u.input_tokens)\nmetrics.gauge(\"llm.cache_hit_ratio\", hit, tags=[f\"feature:{feature}\"])\n```\n\nA hit ratio that was 0.85 last week and is 0.02 today means somebody added a variable to the top of the prompt. Without the metric, you find out from the invoice a month later.\n\nOne last thing worth saying plainly: caching changes nothing about the output. Same tokens in, same distribution out. It is a pure cost and latency optimisation, which makes it unusual: most of the decisions in this course are trade-offs, and this one is close to free if you order the prompt correctly on the first day."},"check":{"question":"A team enables prompt caching on a chat feature with a 2,000-token static preamble and sees no cost reduction at all. Which explanation fits the mechanism?","options":[{"text":"The conversation history grows, and growing prompts cannot be cached.","correct":false,"tempting":"Growth at the end is exactly the case caching handles well, because the prefix stays byte-identical."},{"text":"The prompt includes the current timestamp or user name near the top, so the prefix differs on every call and no cached entry is ever reused.","correct":true},{"text":"Caching only reduces output token cost, which is not where this feature spends.","correct":false,"tempting":"Reverses the mechanism: caching applies to input prefill, and output tokens are never cached."},{"text":"The provider caches only tool definitions, not free text.","correct":false,"tempting":"Invents a restriction; the cache covers whatever prefix you mark, and tool definitions are simply a common thing to put there."}]},"prev":{"slug":"thinking-budgets","title":"Reasoning models, thinking budgets, and when they are a waste of money","url":"https://addaly.com/learn/building-with-ai/thinking-budgets"},"next":{"slug":"grounding-and-citations","title":"Making the model quote before it answers","url":"https://addaly.com/learn/building-with-ai/grounding-and-citations"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Prompt caching, and the ordering rule that pays for itself', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/prompt-caching (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Prompt caching, and the ordering rule that pays for itself","courseTitle":"Building with AI","module":{"index":2,"slug":"prompting-as-engineering","title":"Prompting as engineering"},"answer":null,"keyIdea":"Prompt caching matches an exact byte prefix from position zero, so every static part must come before every variable part, and one timestamp near the top destroys the entire discount.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"dollars per million calls","title":"The same prompt, three orderings, three bills","series":[{"label":"Caching off","value":6000},{"label":"Caching on, timestamp at the top","value":7350},{"label":"Caching on, static block first","value":1140,"highlight":true}],"caption":"1,800 static tokens and 200 variable ones, at $3 per million input, $0.30 cached read and $3.75 cached write. A timestamp at the top gives every call a new prefix, so every call is a cache write and caching costs more than not caching. The saving is in the ordering, not the feature flag."},"alt":"Figure: The same prompt, three orderings, three bills. Bars (dollars per million calls): Caching off 6000; Caching on, timestamp at the top 7350; Caching on, static block first 1140. 1,800 static tokens and 200 variable ones, at $3 per million input, $0.30 cached read and $3.75 cached write. A timestamp at the top gives every call a new prefix, so every call is a cache write and caching costs more than not caching. The saving is in the ordering, not the feature flag."}},"blocks":[{"t":"h2","id":"the-waste-that-caching-removes","runs":[{"text":"The waste that caching removes"}]},{"t":"p","runs":[{"text":"A support assistant sends the same 1,800 tokens on every single call: system prompt, tool definitions, eight examples, a policy document. Only the last 200 tokens (the customer’s message) differ. Across a million calls you have paid for 1.8 billion identical tokens."}]},{"t":"p","runs":[{"text":"Prompt caching removes most of that bill. The mechanism is worth understanding because it dictates how you must write the prompt."}]},{"t":"p","runs":[{"text":"When a model processes your input it computes a key and value vector for every token (the "},{"text":"KV cache","b":true},{"text":") and this prefill is most of the cost and latency of a request with a long prompt. Because attention is causal, the KV entries for the first 1,800 tokens depend only on those tokens. If the next request begins with exactly the same 1,800 tokens, that work can be reused verbatim."}]},{"t":"h2","id":"the-rule-that-follows","runs":[{"text":"The rule that follows"}]},{"t":"p","runs":[{"text":"The match is on an exact prefix, byte for byte, from position zero.","b":true},{"text":" One changed character anywhere in the prefix invalidates everything after it."}]},{"t":"p","runs":[{"text":"So the ordering of your prompt is now an engineering constraint, not a style choice:"}]},{"t":"ol","start":1,"items":[[{"text":"System prompt and tool definitions: identical on every call"}],[{"text":"Few-shot examples: identical"}],[{"text":"Large static documents: identical per document"}],[{"text":"Conversation history: grows, but only at the end"}],[{"text":"The current user turn: always different"}]]},{"t":"p","runs":[{"text":"Get this backwards and you get nothing. A prompt that opens with "},{"text":"Today is 2026-09-06, 14:32:11.","code":true},{"text":" has a new prefix every second and will never hit the cache. Put the timestamp after the static block, or round it to the day, or pass it as part of the user turn."}]},{"t":"p","runs":[{"text":"The same applies to a user’s name, a session id, or a randomised greeting anywhere near the top. This is the single most common reason a team enables caching and sees no saving."}]},{"t":"h2","id":"the-numbers","runs":[{"text":"The numbers"}]},{"t":"p","runs":[{"text":"Rates vary by provider, but the shape is consistent:"}]},{"t":"ul","items":[[{"text":"Cache write","b":true},{"text":": around 1.25× the normal input price, paid once when the prefix is first stored"}],[{"text":"Cache read","b":true},{"text":": around 0.1× the normal input price, a 90 per cent discount"}],[{"text":"Time to live","b":true},{"text":": typically five minutes, refreshed on each hit; longer tiers exist at a higher write price"}]]},{"t":"p","runs":[{"text":"Work an example. 1,800 static tokens plus 200 variable, at $3 per million input."}]},{"t":"ul","items":[[{"text":"Uncached: 2,000 × $3/M = $0.006 per call. A million calls: "},{"text":"$6,000","b":true},{"text":"."}],[{"text":"Cached, with a good hit rate: 1,800 × $0.30/M + 200 × $3/M = $0.00114. A million calls: "},{"text":"$1,140","b":true},{"text":", plus a negligible number of writes."}]]},{"t":"p","runs":[{"text":"That is an 80 per cent reduction on a feature nobody rewrote. Latency falls too, because prefill is skipped: commonly a 30 to 60 per cent cut in time to first token on prompts of this size."}]},{"t":"h2","id":"where-it-silently-fails","runs":[{"text":"Where it silently fails"}]},{"t":"p","runs":[{"text":"Traffic too sparse.","b":true},{"text":" A five-minute TTL means fewer than one request every five minutes gives you writes and no reads, which is "},{"text":"more","i":true},{"text":" expensive than not caching. Caching pays at sustained volume, not on a prototype."}]},{"t":"p","runs":[{"text":"Minimum length.","b":true},{"text":" Providers set a floor (often around 1,024 tokens) below which nothing is cached. A 400-token prompt gets no benefit no matter how static it is."}]},{"t":"p","runs":[{"text":"A moving history.","b":true},{"text":" In a chat, appending turns keeps the prefix stable and the cache warm. But summarising or trimming old turns rewrites the middle of the prompt and invalidates everything downstream. If you trim, trim rarely and in large chunks, so you pay the re-write once rather than every turn."}]},{"t":"p","runs":[{"text":"Tool definitions that reorder.","b":true},{"text":" If your tool list is built from a dictionary or a set, its serialisation order may change between processes. Sort it. A cache miss caused by non-deterministic JSON key order is a miserable bug to find."}]},{"t":"h2","id":"verify-it-do-not-assume-it","runs":[{"text":"Verify it, do not assume it"}]},{"t":"p","runs":[{"text":"The response tells you the truth. Log "},{"text":"cache_read_input_tokens","code":true},{"text":" and "},{"text":"cache_creation_input_tokens","code":true},{"text":" on every call and put the ratio on a dashboard."}]},{"t":"code","lang":"python","label":"Python","text":"u = r.usage\nhit = u.cache_read_input_tokens / max(1, u.cache_read_input_tokens\n                                       + u.cache_creation_input_tokens\n                                       + u.input_tokens)\nmetrics.gauge(\"llm.cache_hit_ratio\", hit, tags=[f\"feature:{feature}\"])"},{"t":"p","runs":[{"text":"A hit ratio that was 0.85 last week and is 0.02 today means somebody added a variable to the top of the prompt. Without the metric, you find out from the invoice a month later."}]},{"t":"p","runs":[{"text":"One last thing worth saying plainly: caching changes nothing about the output. Same tokens in, same distribution out. It is a pure cost and latency optimisation, which makes it unusual: most of the decisions in this course are trade-offs, and this one is close to free if you order the prompt correctly on the first day."}]}],"check":{"question":[{"text":"A team enables prompt caching on a chat feature with a 2,000-token static preamble and sees no cost reduction at all. Which explanation fits the mechanism?"}],"options":[{"text":[{"text":"The conversation history grows, and growing prompts cannot be cached."}],"correct":false,"tempting":[{"text":"Growth at the end is exactly the case caching handles well, because the prefix stays byte-identical."}]},{"text":[{"text":"The prompt includes the current timestamp or user name near the top, so the prefix differs on every call and no cached entry is ever reused."}],"correct":true,"tempting":null},{"text":[{"text":"Caching only reduces output token cost, which is not where this feature spends."}],"correct":false,"tempting":[{"text":"Reverses the mechanism: caching applies to input prefill, and output tokens are never cached."}]},{"text":[{"text":"The provider caches only tool definitions, not free text."}],"correct":false,"tempting":[{"text":"Invents a restriction; the cache covers whatever prefix you mark, and tool definitions are simply a common thing to put there."}]}],"order":[3,0,1,2],"why":null},"prev":{"slug":"thinking-budgets","title":"Reasoning models, thinking budgets, and when they are a waste of money","minutes":9},"next":{"slug":"grounding-and-citations","title":"Making the model quote before it answers","minutes":9}}}