{"course":{"slug":"context-engineering","title":"Context Engineering","url":"https://addaly.com/learn/context-engineering","level":"advanced","track":"build"},"lesson":{"slug":"two-more-caches","title":"The other two caches, and the dangerous one","url":"https://addaly.com/learn/context-engineering/two-more-caches","position":45,"of":87,"minutes":9,"answer":"Cache embeddings by text and model id, cache responses by an exact hash of the assembled request, and treat semantic caching as a wrong-answer generator until you have measured its false-hit rate.","keyIdea":"Cache embeddings by text and model id, cache responses by an exact hash of the assembled request, and treat semantic caching as a wrong-answer generator until you have measured its false-hit rate, while the underrated safe win is caching the retrieval result and still generating fresh."},"body":{"format":"markdown","text":"## Three caches, decreasing safety\n\nPrefix caching is one of three you can run, and they differ sharply in how badly they fail.\n\n**The embedding cache** is entirely safe. Keyed on the normalised query text, it returns a vector you computed before. Embeddings are deterministic for a given model version, so a hit is exactly the value you would have computed.\n\n```python\nkey = sha256(f\"{model_id}:{normalise(text)}\".encode()).hexdigest()\n```\n\nInclude the model id in the key. Forgetting it means an embedding-model upgrade silently serves vectors from the old space, which produces retrieval that is subtly and unfixably wrong: one of the nastier bugs available, because everything still runs.\n\nHit rates are high in practice: head queries repeat, and so does every chunk you re-embed after an unchanged document is reprocessed. This cache pays for itself immediately and cannot be wrong.\n\n**The exact-response cache** is nearly as safe. Key on a hash of the fully assembled request (every token, plus the model id and sampling parameters) and store the response. A hit means the input was byte-identical, so returning the stored answer is defensible.\n\n```python\nkey = sha256((model + str(temp) + rendered_request).encode()).hexdigest()\n```\n\nIts weakness is the hit rate. With retrieval and history in the request, byte-identical inputs are rare. It earns its place on narrow, high-volume surfaces (a documentation search box, a classification endpoint, an autocomplete) where the same input recurs. Give it a TTL, because your corpus changes even when the question does not.\n\n**The semantic cache** is where teams get hurt.\n\n## Why semantic caching is different\n\nThe idea: embed the incoming question, find a previous question within some similarity threshold, return that answer. It promises the hit rate the exact cache lacks.\n\nIt does so by asserting that two different inputs deserve the same output, which is a claim about meaning that a cosine similarity is not qualified to make. The failures are exactly the ones from the relevance-is-not-similarity lesson, now attached to a stored answer:\n\n- \"What is the status of order 88213?\" and \"What is the status of order 88214?\" are extremely similar and have different answers.\n- \"Can I refund this?\" asked by two customers on different plans is the same string with different correct answers.\n- \"Is the API down?\" has an answer with a shelf life of minutes.\n- A negated question sits close to its affirmative twin.\n\nThe damage is worse than an ordinary wrong answer, because it is confidently wrong, reproducible, and served to many users from one bad entry.\n\n## If you must run one, the rules\n\n1. **Never cache anything user-specific.** If the answer depends on who is asking, their account, their permissions or their data, it is not cacheable across users. Full stop.\n2. **Scope the key.** Tenant, language, and the permission set, all in the key. A cross-tenant semantic hit is a data breach, not a bug.\n3. **Exclude entity-bearing queries.** Anything containing a number, an id, a date or a proper noun goes straight to the model. A cheap regex removes most of the dangerous cases.\n4. **Threshold high, and calibrate it on judged pairs.** A similarity that seems close is not close enough; you are asking for near-equivalence, not relevance.\n5. **Set a short TTL** and invalidate on corpus changes.\n6. **Measure the false-hit rate before shipping.** Take a thousand recent query pairs the cache would have matched, and have a human, or a checked judge, say how many deserved the same answer. If it is below 95%, you have built a wrong-answer generator with a good hit rate.\n\nMany teams that run this measurement decide not to ship. That is a successful outcome for an afternoon's work.\n\n## The retrieval cache, which is the underrated one\n\nBetween the extremes sits a cache almost nobody builds and most should: **cache the retrieval result, not the answer.**\n\nKey on the normalised, rewritten query plus the filter set. Store the ranked passage ids. On a hit you skip embedding, search and reranking (often 300 milliseconds and the majority of your pre-model latency) and still run the model on the real question with the real history.\n\nIt is far safer than semantic caching because the model still generates fresh, and the personalisation lives in history and instructions rather than in the cached part. The staleness risk is bounded and handled with a short TTL plus invalidation on index updates.\n\nFor a system where head queries dominate, this is usually the best latency-per-unit-of-risk trade available, and it is a dozen lines of code."},"check":{"question":"A semantic cache returns the delivery date of order 88213 when asked about order 88214. Which design rule would have prevented this?","options":[{"text":"Include the model id in the cache key so upgrades do not serve stale entries.","correct":false,"tempting":"Essential for embedding caches, but both queries here would hit the same model and the same wrong entry."},{"text":"Route queries containing identifiers straight to the model instead of the cache.","correct":true},{"text":"Shorten the time-to-live so the cached answer expires sooner.","correct":false,"tempting":"Limits how long the wrong answer persists without preventing it; a fresh entry for 88213 is equally wrong for 88214."},{"text":"Scope the cache key by tenant, so answers cannot cross customer boundaries.","correct":false,"tempting":"Necessary for isolation, but both orders can belong to the same tenant and the collision still occurs."}]},"prev":{"slug":"cache-hit-rate-as-a-metric","title":"Measuring the cache instead of believing in it","url":"https://addaly.com/learn/context-engineering/cache-hit-rate-as-a-metric"},"next":{"slug":"batching-and-the-slow-lane","title":"Work that can wait","url":"https://addaly.com/learn/context-engineering/batching-and-the-slow-lane"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The other two caches, and the dangerous one', in Context Engineering. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/context-engineering/two-more-caches (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The other two caches, and the dangerous one","courseTitle":"Context Engineering","module":{"index":5,"slug":"cost-cache-and-latency","title":"Cost, caching and latency"},"answer":null,"keyIdea":"Cache embeddings by text and model id, cache responses by an exact hash of the assembled request, and treat semantic caching as a wrong-answer generator until you have measured its false-hit rate, while the underrated safe win is caching the retrieval result and still generating fresh.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"three-caches-decreasing-safety","runs":[{"text":"Three caches, decreasing safety"}]},{"t":"p","runs":[{"text":"Prefix caching is one of three you can run, and they differ sharply in how badly they fail."}]},{"t":"p","runs":[{"text":"The embedding cache","b":true},{"text":" is entirely safe. Keyed on the normalised query text, it returns a vector you computed before. Embeddings are deterministic for a given model version, so a hit is exactly the value you would have computed."}]},{"t":"code","lang":"python","label":"Python","text":"key = sha256(f\"{model_id}:{normalise(text)}\".encode()).hexdigest()"},{"t":"p","runs":[{"text":"Include the model id in the key. Forgetting it means an embedding-model upgrade silently serves vectors from the old space, which produces retrieval that is subtly and unfixably wrong: one of the nastier bugs available, because everything still runs."}]},{"t":"p","runs":[{"text":"Hit rates are high in practice: head queries repeat, and so does every chunk you re-embed after an unchanged document is reprocessed. This cache pays for itself immediately and cannot be wrong."}]},{"t":"p","runs":[{"text":"The exact-response cache","b":true},{"text":" is nearly as safe. Key on a hash of the fully assembled request (every token, plus the model id and sampling parameters) and store the response. A hit means the input was byte-identical, so returning the stored answer is defensible."}]},{"t":"code","lang":"python","label":"Python","text":"key = sha256((model + str(temp) + rendered_request).encode()).hexdigest()"},{"t":"p","runs":[{"text":"Its weakness is the hit rate. With retrieval and history in the request, byte-identical inputs are rare. It earns its place on narrow, high-volume surfaces (a documentation search box, a classification endpoint, an autocomplete) where the same input recurs. Give it a TTL, because your corpus changes even when the question does not."}]},{"t":"p","runs":[{"text":"The semantic cache","b":true},{"text":" is where teams get hurt."}]},{"t":"h2","id":"why-semantic-caching-is-different","runs":[{"text":"Why semantic caching is different"}]},{"t":"p","runs":[{"text":"The idea: embed the incoming question, find a previous question within some similarity threshold, return that answer. It promises the hit rate the exact cache lacks."}]},{"t":"p","runs":[{"text":"It does so by asserting that two different inputs deserve the same output, which is a claim about meaning that a cosine similarity is not qualified to make. The failures are exactly the ones from the relevance-is-not-similarity lesson, now attached to a stored answer:"}]},{"t":"ul","items":[[{"text":"“What is the status of order 88213?” and “What is the status of order 88214?” are extremely similar and have different answers."}],[{"text":"“Can I refund this?” asked by two customers on different plans is the same string with different correct answers."}],[{"text":"“Is the API down?” has an answer with a shelf life of minutes."}],[{"text":"A negated question sits close to its affirmative twin."}]]},{"t":"p","runs":[{"text":"The damage is worse than an ordinary wrong answer, because it is confidently wrong, reproducible, and served to many users from one bad entry."}]},{"t":"h2","id":"if-you-must-run-one-the-rules","runs":[{"text":"If you must run one, the rules"}]},{"t":"ol","start":1,"items":[[{"text":"Never cache anything user-specific.","b":true},{"text":" If the answer depends on who is asking, their account, their permissions or their data, it is not cacheable across users. Full stop."}],[{"text":"Scope the key.","b":true},{"text":" Tenant, language, and the permission set, all in the key. A cross-tenant semantic hit is a data breach, not a bug."}],[{"text":"Exclude entity-bearing queries.","b":true},{"text":" Anything containing a number, an id, a date or a proper noun goes straight to the model. A cheap regex removes most of the dangerous cases."}],[{"text":"Threshold high, and calibrate it on judged pairs.","b":true},{"text":" A similarity that seems close is not close enough; you are asking for near-equivalence, not relevance."}],[{"text":"Set a short TTL","b":true},{"text":" and invalidate on corpus changes."}],[{"text":"Measure the false-hit rate before shipping.","b":true},{"text":" Take a thousand recent query pairs the cache would have matched, and have a human, or a checked judge, say how many deserved the same answer. If it is below 95%, you have built a wrong-answer generator with a good hit rate."}]]},{"t":"p","runs":[{"text":"Many teams that run this measurement decide not to ship. That is a successful outcome for an afternoon’s work."}]},{"t":"h2","id":"the-retrieval-cache-which-is-the-underrated-one","runs":[{"text":"The retrieval cache, which is the underrated one"}]},{"t":"p","runs":[{"text":"Between the extremes sits a cache almost nobody builds and most should: "},{"text":"cache the retrieval result, not the answer.","b":true}]},{"t":"p","runs":[{"text":"Key on the normalised, rewritten query plus the filter set. Store the ranked passage ids. On a hit you skip embedding, search and reranking (often 300 milliseconds and the majority of your pre-model latency) and still run the model on the real question with the real history."}]},{"t":"p","runs":[{"text":"It is far safer than semantic caching because the model still generates fresh, and the personalisation lives in history and instructions rather than in the cached part. The staleness risk is bounded and handled with a short TTL plus invalidation on index updates."}]},{"t":"p","runs":[{"text":"For a system where head queries dominate, this is usually the best latency-per-unit-of-risk trade available, and it is a dozen lines of code."}]}],"check":{"question":[{"text":"A semantic cache returns the delivery date of order 88213 when asked about order 88214. Which design rule would have prevented this?"}],"options":[{"text":[{"text":"Include the model id in the cache key so upgrades do not serve stale entries."}],"correct":false,"tempting":[{"text":"Essential for embedding caches, but both queries here would hit the same model and the same wrong entry."}]},{"text":[{"text":"Route queries containing identifiers straight to the model instead of the cache."}],"correct":true,"tempting":null},{"text":[{"text":"Shorten the time-to-live so the cached answer expires sooner."}],"correct":false,"tempting":[{"text":"Limits how long the wrong answer persists without preventing it; a fresh entry for 88213 is equally wrong for 88214."}]},{"text":[{"text":"Scope the cache key by tenant, so answers cannot cross customer boundaries."}],"correct":false,"tempting":[{"text":"Necessary for isolation, but both orders can belong to the same tenant and the collision still occurs."}]}],"order":[0,2,1,3],"why":null},"prev":{"slug":"cache-hit-rate-as-a-metric","title":"Measuring the cache instead of believing in it","minutes":8},"next":{"slug":"batching-and-the-slow-lane","title":"Work that can wait","minutes":8}}}