{"course":{"slug":"building-with-ai","title":"Building with AI","url":"https://addaly.com/learn/building-with-ai","level":"intermediate","track":"build"},"lesson":{"slug":"caching-at-three-levels","title":"Caching at three levels: exact, prefix and semantic","url":"https://addaly.com/learn/building-with-ai/caching-at-three-levels","position":73,"of":80,"minutes":9,"answer":"Three caches do three different jobs (an exact-match response cache for repeated requests, the provider's prefix cache for repeated context.","keyIdea":"Three caches do three different jobs (an exact-match response cache for repeated requests, the provider's prefix cache for repeated context, and a semantic cache that serves similar questions and is the one that can return a wrong answer) and every cache must be keyed on prompt version, model and corpus version, and must never serve one user's content to another."},"body":{"format":"markdown","text":"## Three caches, three questions\n\n\"Have we answered exactly this before?\" \"Have we sent exactly this context before?\" \"Have we answered something like this before?\" Each is a cache, each saves a different cost, and each fails differently. Teams that say \"we added caching\" usually mean one of them and are surprised by the other two.\n\n## Exact-match response cache\n\nKey: a hash of the rendered prompt (system prompt, context, conversation and message) plus the model identifier and the parameters. Value: the response. A hit costs nothing and returns instantly.\n\nHit rates depend entirely on the product. A first-turn question box on a documentation site sees a fair share of identical queries (\"how do I reset my password\") and can hit ten to thirty per cent. A conversational assistant, where every turn carries a different history, hits almost never after the first turn. Measure before assuming; a cache that hits two per cent of the time is still worth having for the denial-of-wallet reason in the safety module, since the repeated-request attack hits it every time.\n\nTwo rules. First, never serve a response generated for one user to another if the prompt contained anything about the first user: the key must include everything that shaped the answer, and a personalised prefix makes cross-user hits impossible by construction, which is the safe outcome. Second, invalidate on every change to the prompt version, the model, or the corpus, by putting all three in the key. A cache keyed on the user's message alone serves last month's policy for as long as it lives.\n\n## Prefix cache\n\nThe provider's cache, from the prompting module. The provider keeps the processed state of a prompt prefix for a short window and charges a fraction of the input price when a later request begins with the same tokens. It saves prefill cost and prefill time, and it does not save output cost at all.\n\nWhat it needs from you: an identical prefix, byte for byte, at the front of the request. The stable material first (system prompt, tool definitions, the document if there is one) and the variable material last. A timestamp in the system prompt, a user's name placed before the instructions, retrieved chunks inserted above the system prompt: each breaks the match and turns every request into a miss. Providers report cached token counts in the usage field; the traces lesson's cost roll-up should show the hit rate per feature, and a feature whose cached-token count is near zero has a prefix that is changing when it should not.\n\nThe window is minutes on most providers, extendable at a price on some. Traffic that arrives steadily keeps it warm. A prefix used once an hour is re-paid every hour.\n\n## Semantic cache\n\nThe third cache embeds the incoming question, looks for a stored question within some similarity of it, and if one is close enough returns its stored answer. The hit rate can be high (paraphrases of common questions collapse onto one entry), and it is the cache that can be wrong.\n\nThe mechanism of the failure is the embeddings lesson's blind spots. \"How do I cancel my annual plan?\" and \"How do I cancel my monthly plan?\" embed very close; a threshold that serves the first answer for the second question has confidently given a user the wrong policy, with no error anywhere. Negations, numbers and names are exactly what a semantic cache blurs, and they are exactly what changes the answer.\n\nWhere it is safe: a fixed set of frequently asked questions with answers that do not depend on the user, a threshold set high on your own data with the calibration check from the embeddings lesson, and a review of hits sampled weekly. Where it is not: anything personalised, anything with an identifier in it, anything where two similar questions have different answers. Many teams find the exact cache plus the prefix cache gives most of the saving with none of the risk, and leave the semantic one out.\n\n## Invalidation\n\nEvery cache in this lesson becomes wrong when the prompt changes, the model changes or the corpus changes. Put a version for each in the key: `prompt_v7`, the dated model identifier, and a corpus version incremented on re-index. A cache that is emptied by a version bump is one you never have to purge by hand, and one you never forget to.\n\n## What to expect\n\nA rough shape for a conversational assistant with retrieval: exact cache low single digits; prefix cache most of the input tokens on turns after the first, if the ordering is right; semantic cache omitted. For a question box over documentation: exact cache in the tens of per cent; prefix cache high; semantic cache tempting and to be measured carefully. Whatever the shape, the cost model from the start of this block has a cell for each rate, and the traces fill it in."},"check":{"question":"A semantic cache serves the answer to \"how do I cancel my annual plan?\" when a user asks \"how do I cancel my monthly plan?\", and the policies differ. What is the mechanism?","options":[{"text":"The cache key omitted the prompt version, so a stale answer from an earlier policy was served.","correct":false,"tempting":"A version omission serves an outdated answer to the same question; here the question itself was different."},{"text":"The questions differ by one content word, and embeddings place them close enough that the threshold treated them as the same question.","correct":true},{"text":"The exact-match cache hashed the question after lowercasing and stripping punctuation, which made the two strings collide in the key space.","correct":false,"tempting":"Normalisation would not make \"annual\" and \"monthly\" collide; only similarity-based matching does."},{"text":"The prefix cache returned the earlier response because both requests began with the same system prompt.","correct":false,"tempting":"A prefix cache reuses processed input state, never a stored response; it cannot return an answer."}]},"prev":{"slug":"where-the-milliseconds-go","title":"Where the milliseconds go","url":"https://addaly.com/learn/building-with-ai/where-the-milliseconds-go"},"next":{"slug":"the-batch-window","title":"The batch window: half price for anything that can wait","url":"https://addaly.com/learn/building-with-ai/the-batch-window"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Caching at three levels: exact, prefix and semantic', in Building with AI. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/building-with-ai/caching-at-three-levels (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Caching at three levels: exact, prefix and semantic","courseTitle":"Building with AI","module":{"index":8,"slug":"cost-latency-and-running-it","title":"Cost, latency and running it in production"},"answer":null,"keyIdea":"Three caches do three different jobs (an exact-match response cache for repeated requests, the provider’s prefix cache for repeated context, and a semantic cache that serves similar questions and is the one that can return a wrong answer) and every cache must be keyed on prompt version, model and corpus version, and must never serve one user’s content to another.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-caches-three-questions","runs":[{"text":"Three caches, three questions"}]},{"t":"p","runs":[{"text":"“Have we answered exactly this before?” “Have we sent exactly this context before?” “Have we answered something like this before?” Each is a cache, each saves a different cost, and each fails differently. Teams that say “we added caching” usually mean one of them and are surprised by the other two."}]},{"t":"h2","id":"exact-match-response-cache","runs":[{"text":"Exact-match response cache"}]},{"t":"p","runs":[{"text":"Key: a hash of the rendered prompt (system prompt, context, conversation and message) plus the model identifier and the parameters. Value: the response. A hit costs nothing and returns instantly."}]},{"t":"p","runs":[{"text":"Hit rates depend entirely on the product. A first-turn question box on a documentation site sees a fair share of identical queries (“how do I reset my password”) and can hit ten to thirty per cent. A conversational assistant, where every turn carries a different history, hits almost never after the first turn. Measure before assuming; a cache that hits two per cent of the time is still worth having for the denial-of-wallet reason in the safety module, since the repeated-request attack hits it every time."}]},{"t":"p","runs":[{"text":"Two rules. First, never serve a response generated for one user to another if the prompt contained anything about the first user: the key must include everything that shaped the answer, and a personalised prefix makes cross-user hits impossible by construction, which is the safe outcome. Second, invalidate on every change to the prompt version, the model, or the corpus, by putting all three in the key. A cache keyed on the user’s message alone serves last month’s policy for as long as it lives."}]},{"t":"h2","id":"prefix-cache","runs":[{"text":"Prefix cache"}]},{"t":"p","runs":[{"text":"The provider’s cache, from the prompting module. The provider keeps the processed state of a prompt prefix for a short window and charges a fraction of the input price when a later request begins with the same tokens. It saves prefill cost and prefill time, and it does not save output cost at all."}]},{"t":"p","runs":[{"text":"What it needs from you: an identical prefix, byte for byte, at the front of the request. The stable material first (system prompt, tool definitions, the document if there is one) and the variable material last. A timestamp in the system prompt, a user’s name placed before the instructions, retrieved chunks inserted above the system prompt: each breaks the match and turns every request into a miss. Providers report cached token counts in the usage field; the traces lesson’s cost roll-up should show the hit rate per feature, and a feature whose cached-token count is near zero has a prefix that is changing when it should not."}]},{"t":"p","runs":[{"text":"The window is minutes on most providers, extendable at a price on some. Traffic that arrives steadily keeps it warm. A prefix used once an hour is re-paid every hour."}]},{"t":"h2","id":"semantic-cache","runs":[{"text":"Semantic cache"}]},{"t":"p","runs":[{"text":"The third cache embeds the incoming question, looks for a stored question within some similarity of it, and if one is close enough returns its stored answer. The hit rate can be high (paraphrases of common questions collapse onto one entry), and it is the cache that can be wrong."}]},{"t":"p","runs":[{"text":"The mechanism of the failure is the embeddings lesson’s blind spots. “How do I cancel my annual plan?” and “How do I cancel my monthly plan?” embed very close; a threshold that serves the first answer for the second question has confidently given a user the wrong policy, with no error anywhere. Negations, numbers and names are exactly what a semantic cache blurs, and they are exactly what changes the answer."}]},{"t":"p","runs":[{"text":"Where it is safe: a fixed set of frequently asked questions with answers that do not depend on the user, a threshold set high on your own data with the calibration check from the embeddings lesson, and a review of hits sampled weekly. Where it is not: anything personalised, anything with an identifier in it, anything where two similar questions have different answers. Many teams find the exact cache plus the prefix cache gives most of the saving with none of the risk, and leave the semantic one out."}]},{"t":"h2","id":"invalidation","runs":[{"text":"Invalidation"}]},{"t":"p","runs":[{"text":"Every cache in this lesson becomes wrong when the prompt changes, the model changes or the corpus changes. Put a version for each in the key: "},{"text":"prompt_v7","code":true},{"text":", the dated model identifier, and a corpus version incremented on re-index. A cache that is emptied by a version bump is one you never have to purge by hand, and one you never forget to."}]},{"t":"h2","id":"what-to-expect","runs":[{"text":"What to expect"}]},{"t":"p","runs":[{"text":"A rough shape for a conversational assistant with retrieval: exact cache low single digits; prefix cache most of the input tokens on turns after the first, if the ordering is right; semantic cache omitted. For a question box over documentation: exact cache in the tens of per cent; prefix cache high; semantic cache tempting and to be measured carefully. Whatever the shape, the cost model from the start of this block has a cell for each rate, and the traces fill it in."}]}],"check":{"question":[{"text":"A semantic cache serves the answer to “how do I cancel my annual plan?” when a user asks “how do I cancel my monthly plan?”, and the policies differ. What is the mechanism?"}],"options":[{"text":[{"text":"The cache key omitted the prompt version, so a stale answer from an earlier policy was served."}],"correct":false,"tempting":[{"text":"A version omission serves an outdated answer to the same question; here the question itself was different."}]},{"text":[{"text":"The questions differ by one content word, and embeddings place them close enough that the threshold treated them as the same question."}],"correct":true,"tempting":null},{"text":[{"text":"The exact-match cache hashed the question after lowercasing and stripping punctuation, which made the two strings collide in the key space."}],"correct":false,"tempting":[{"text":"Normalisation would not make “annual” and “monthly” collide; only similarity-based matching does."}]},{"text":[{"text":"The prefix cache returned the earlier response because both requests began with the same system prompt."}],"correct":false,"tempting":[{"text":"A prefix cache reuses processed input state, never a stored response; it cannot return an answer."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"where-the-milliseconds-go","title":"Where the milliseconds go","minutes":9},"next":{"slug":"the-batch-window","title":"The batch window: half price for anything that can wait","minutes":8}}}