Caching at three levels: exact, prefix and semantic

Three caches, three questions

“Have we answered exactly this before?” “Have we sent exactly this context before?” “Have we answered something like this before?” Each is a cache, each saves a different cost, and each fails differently. Teams that say “we added caching” usually mean one of them and are surprised by the other two.

Exact-match response cache

Key: a hash of the rendered prompt (system prompt, context, conversation and message) plus the model identifier and the parameters. Value: the response. A hit costs nothing and returns instantly.

Hit rates depend entirely on the product. A first-turn question box on a documentation site sees a fair share of identical queries (“how do I reset my password”) and can hit ten to thirty per cent. A conversational assistant, where every turn carries a different history, hits almost never after the first turn. Measure before assuming; a cache that hits two per cent of the time is still worth having for the denial-of-wallet reason in the safety module, since the repeated-request attack hits it every time.

Two rules. First, never serve a response generated for one user to another if the prompt contained anything about the first user: the key must include everything that shaped the answer, and a personalised prefix makes cross-user hits impossible by construction, which is the safe outcome. Second, invalidate on every change to the prompt version, the model, or the corpus, by putting all three in the key. A cache keyed on the user’s message alone serves last month’s policy for as long as it lives.

Prefix cache

The provider’s cache, from the prompting module. The provider keeps the processed state of a prompt prefix for a short window and charges a fraction of the input price when a later request begins with the same tokens. It saves prefill cost and prefill time, and it does not save output cost at all.

What it needs from you: an identical prefix, byte for byte, at the front of the request. The stable material first (system prompt, tool definitions, the document if there is one) and the variable material last. A timestamp in the system prompt, a user’s name placed before the instructions, retrieved chunks inserted above the system prompt: each breaks the match and turns every request into a miss. Providers report cached token counts in the usage field; the traces lesson’s cost roll-up should show the hit rate per feature, and a feature whose cached-token count is near zero has a prefix that is changing when it should not.

The window is minutes on most providers, extendable at a price on some. Traffic that arrives steadily keeps it warm. A prefix used once an hour is re-paid every hour.

Semantic cache

The third cache embeds the incoming question, looks for a stored question within some similarity of it, and if one is close enough returns its stored answer. The hit rate can be high (paraphrases of common questions collapse onto one entry), and it is the cache that can be wrong.

The mechanism of the failure is the embeddings lesson’s blind spots. “How do I cancel my annual plan?” and “How do I cancel my monthly plan?” embed very close; a threshold that serves the first answer for the second question has confidently given a user the wrong policy, with no error anywhere. Negations, numbers and names are exactly what a semantic cache blurs, and they are exactly what changes the answer.

Where it is safe: a fixed set of frequently asked questions with answers that do not depend on the user, a threshold set high on your own data with the calibration check from the embeddings lesson, and a review of hits sampled weekly. Where it is not: anything personalised, anything with an identifier in it, anything where two similar questions have different answers. Many teams find the exact cache plus the prefix cache gives most of the saving with none of the risk, and leave the semantic one out.

Invalidation

Every cache in this lesson becomes wrong when the prompt changes, the model changes or the corpus changes. Put a version for each in the key: prompt_v7, the dated model identifier, and a corpus version incremented on re-index. A cache that is emptied by a version bump is one you never have to purge by hand, and one you never forget to.

What to expect

A rough shape for a conversational assistant with retrieval: exact cache low single digits; prefix cache most of the input tokens on turns after the first, if the ordering is right; semantic cache omitted. For a question box over documentation: exact cache in the tens of per cent; prefix cache high; semantic cache tempting and to be measured carefully. Whatever the shape, the cost model from the start of this block has a cell for each rate, and the traces fill it in.