{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"prefix-caching-properly","title":"Prefix caching, and how to design prompts for it","url":"https://addaly.com/learn/running-models-yourself/prefix-caching-properly","position":40,"of":82,"minutes":9,"answer":"Prefix caching reuses the KV cache for an exact leading match, so fixed content goes first and anything variable (a timestamp, a session id) goes last, or the cache never hits at all.","keyIdea":"Prefix caching reuses the KV cache for an exact leading match, so fixed content goes first and anything variable (a timestamp, a session id) goes last, or the cache never hits at all."},"body":{"format":"markdown","text":"## The work you are repeating\n\nEvery request in a typical application starts with the same 1,500 tokens: a system prompt, formatting rules, few-shot examples, tool definitions. Without caching, every one of those requests recomputes the identical prefill. On a server handling ten requests a second, that is fifteen thousand tokens per second of work whose answer never changes.\n\nPrefix caching keeps the KV cache for a prefix that has already been processed and reuses it. The saving is not marginal. A request with a 1,500-token shared preamble and a 100-token question drops from 1,600 tokens of prefill to 100: a 94% reduction in the phase that dominates time to first token.\n\n## How it is implemented\n\nThe cache is stored in fixed-size blocks, typically sixteen tokens each. Each block is hashed together with the hash of every block before it, so a block's identity encodes the whole sequence that led to it. When a new request arrives, the server hashes its prompt block by block and looks each up; matching blocks are reused, and computation begins at the first block that misses.\n\nTwo properties follow directly. The match must be an exact prefix from the very first token: a shared middle is worth nothing. And it is automatic: nothing has to be declared, because identical prefixes hash identically.\n\n> Figure: One line at the top of a preamble, and no cache hits. A two by two grid, rows Stable prefix and Timestamp, columns First request and Every one after: Stable prefix and First request: 1,600 tokens; Stable prefix and Every one after: 100 tokens; Timestamp and First request: 1,600 tokens; Timestamp and Every one after: 1,600 tokens. Blocks are hashed together with every block before them, so a match must be an exact prefix from the very first token. A timestamp, a session id or a user's name in the preamble invalidates all of it on every request, and nothing in the logs announces that it happened.\n\nSGLang's RadixAttention organises these blocks as a prefix tree, which makes partial sharing between many branching conversations natural and cheap. vLLM enables the same idea with `--enable-prefix-caching`, on by default in recent versions. llama.cpp keeps a conversation on the same slot so its cache survives across turns.\n\n## Designing prompts to hit the cache\n\nThis is where a small amount of care pays repeatedly.\n\n**Put the fixed part first and the variable part last.** Everything up to the first difference is cacheable; everything after it is not. So system prompt, then tool definitions, then examples, then retrieved documents, then the user's question.\n\n**Take dynamic values out of the preamble.** A timestamp, a session id, or a user's name at the top of the system prompt invalidates the entire cache for every request. This single mistake is the most common cause of a prefix cache achieving almost no hits, and it is invisible unless you check.\n\n**Keep the order of retrieved documents stable.** If three documents are concatenated in a different order for two similar queries, the prefix diverges at the first one.\n\n**Prefer appending to rewriting.** A conversation that appends turns keeps its whole history cached; one that summarises and rebuilds the prompt each turn throws the cache away every time.\n\n## Measure the hit rate\n\nDo not assume it is working. vLLM exposes prefix cache hit statistics through its metrics endpoint, and the effect is visible directly in time to first token:\n\n```bash\n# same long prompt twice, timed\ntime curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \\\n  -d @long_prompt.json > /dev/null\ntime curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \\\n  -d @long_prompt.json > /dev/null\n```\n\nIf the second call is not markedly faster, caching is off, the prompt is not identical, or the blocks were evicted between calls. All three are worth knowing about.\n\n## The costs and the limits\n\n**Cache blocks occupy memory that concurrent sequences also want.** On a card with limited VRAM, a large prefix cache reduces how many users you can serve at once. Engines evict least-recently-used blocks, so a busy server with diverse prefixes may keep almost nothing useful.\n\n**Sampling parameters do not affect it,** because the cache holds the prompt's key and value vectors, which do not depend on how tokens are chosen. Different temperatures share a cache happily.\n\n**A LoRA adapter or a different model does not share a cache** with another, because the vectors are computed by different weights.\n\n**There is a genuine multi-tenant concern.** Because a hit is much faster than a miss, timing differences can in principle reveal whether some other user has recently sent a particular prefix. On a shared server with untrusted users this is a real side channel, and the mitigations are to disable cross-user sharing or to scope caches per tenant. On a single-team server it does not matter, but it should be a deliberate decision rather than an unexamined default.\n\n## Where it matters most\n\nAgent loops, where a long tool definition block is sent on every step. Classification pipelines with a fixed instruction and a rotating input. Retrieval systems with a stable system prompt. Chat, where each turn extends the previous prefix exactly.\n\nFor those workloads, enabling prefix caching and reordering the prompt is frequently a larger win than changing engines, and it takes twenty minutes."},"check":{"question":"An agent system with a 2,000-token tool-definition preamble enables prefix caching and sees almost no improvement in time to first token. The preamble is identical in every request except that it opens with the current date and time. What is happening?","options":[{"text":"Cache blocks are being evicted between requests because the agent's traffic is too diverse for the available cache memory.","correct":false,"tempting":"A real failure mode, but it would not be explained by the one detail given, and eviction usually still leaves some hits."},{"text":"Tool definitions are sent in a separate field that most engines exclude from the cacheable prefix.","correct":false,"tempting":"Invents an exclusion; tool definitions are rendered into the prompt string by the template like everything else."},{"text":"The timestamp changes the first block, so every subsequent block hash differs and the entire preamble misses.","correct":true},{"text":"Caching requires identical sampling parameters, and the agent varies temperature between planning and execution steps.","correct":false,"tempting":"Attributes a dependency the cache does not have; keys and values are computed before any sampling decision."}]},"prev":{"slug":"continuous-batching","title":"Continuous batching, and the queue underneath it","url":"https://addaly.com/learn/running-models-yourself/continuous-batching"},"next":{"slug":"speculative-decoding","title":"Speculative decoding: several tokens per weight read","url":"https://addaly.com/learn/running-models-yourself/speculative-decoding"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Prefix caching, and how to design prompts for it', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/prefix-caching-properly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Prefix caching, and how to design prompts for it","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Prefix caching reuses the KV cache for an exact leading match, so fixed content goes first and anything variable (a timestamp, a session id) goes last, or the cache never hits at all.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"cols":["First request","Every one after"],"rows":["Stable prefix","Timestamp"],"type":"matrix","cells":[["1,600 tokens","100 tokens"],["1,600 tokens","1,600 tokens"]],"title":"One line at the top of a preamble, and no cache hits","caption":"Blocks are hashed together with every block before them, so a match must be an exact prefix from the very first token. A timestamp, a session id or a user’s name in the preamble invalidates all of it on every request, and nothing in the logs announces that it happened.","cellLabels":[["Full prefill","Cache hit, 94% less"],["Full prefill","The cache never hits"]]},"alt":"Figure: One line at the top of a preamble, and no cache hits. A two by two grid, rows Stable prefix and Timestamp, columns First request and Every one after: Stable prefix and First request: 1,600 tokens; Stable prefix and Every one after: 100 tokens; Timestamp and First request: 1,600 tokens; Timestamp and Every one after: 1,600 tokens. Blocks are hashed together with every block before them, so a match must be an exact prefix from the very first token. A timestamp, a session id or a user’s name in the preamble invalidates all of it on every request, and nothing in the logs announces that it happened."}},"blocks":[{"t":"h2","id":"the-work-you-are-repeating","runs":[{"text":"The work you are repeating"}]},{"t":"p","runs":[{"text":"Every request in a typical application starts with the same 1,500 tokens: a system prompt, formatting rules, few-shot examples, tool definitions. Without caching, every one of those requests recomputes the identical prefill. On a server handling ten requests a second, that is fifteen thousand tokens per second of work whose answer never changes."}]},{"t":"p","runs":[{"text":"Prefix caching keeps the KV cache for a prefix that has already been processed and reuses it. The saving is not marginal. A request with a 1,500-token shared preamble and a 100-token question drops from 1,600 tokens of prefill to 100: a 94% reduction in the phase that dominates time to first token."}]},{"t":"h2","id":"how-it-is-implemented","runs":[{"text":"How it is implemented"}]},{"t":"p","runs":[{"text":"The cache is stored in fixed-size blocks, typically sixteen tokens each. Each block is hashed together with the hash of every block before it, so a block’s identity encodes the whole sequence that led to it. When a new request arrives, the server hashes its prompt block by block and looks each up; matching blocks are reused, and computation begins at the first block that misses."}]},{"t":"p","runs":[{"text":"Two properties follow directly. The match must be an exact prefix from the very first token: a shared middle is worth nothing. And it is automatic: nothing has to be declared, because identical prefixes hash identically."}]},{"t":"p","runs":[{"text":"SGLang’s RadixAttention organises these blocks as a prefix tree, which makes partial sharing between many branching conversations natural and cheap. vLLM enables the same idea with "},{"text":"--enable-prefix-caching","code":true},{"text":", on by default in recent versions. llama.cpp keeps a conversation on the same slot so its cache survives across turns."}]},{"t":"h2","id":"designing-prompts-to-hit-the-cache","runs":[{"text":"Designing prompts to hit the cache"}]},{"t":"p","runs":[{"text":"This is where a small amount of care pays repeatedly."}]},{"t":"p","runs":[{"text":"Put the fixed part first and the variable part last.","b":true},{"text":" Everything up to the first difference is cacheable; everything after it is not. So system prompt, then tool definitions, then examples, then retrieved documents, then the user’s question."}]},{"t":"p","runs":[{"text":"Take dynamic values out of the preamble.","b":true},{"text":" A timestamp, a session id, or a user’s name at the top of the system prompt invalidates the entire cache for every request. This single mistake is the most common cause of a prefix cache achieving almost no hits, and it is invisible unless you check."}]},{"t":"p","runs":[{"text":"Keep the order of retrieved documents stable.","b":true},{"text":" If three documents are concatenated in a different order for two similar queries, the prefix diverges at the first one."}]},{"t":"p","runs":[{"text":"Prefer appending to rewriting.","b":true},{"text":" A conversation that appends turns keeps its whole history cached; one that summarises and rebuilds the prompt each turn throws the cache away every time."}]},{"t":"h2","id":"measure-the-hit-rate","runs":[{"text":"Measure the hit rate"}]},{"t":"p","runs":[{"text":"Do not assume it is working. vLLM exposes prefix cache hit statistics through its metrics endpoint, and the effect is visible directly in time to first token:"}]},{"t":"code","lang":"bash","label":"Bash","text":"# same long prompt twice, timed\ntime curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \\\n  -d @long_prompt.json > /dev/null\ntime curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \\\n  -d @long_prompt.json > /dev/null"},{"t":"p","runs":[{"text":"If the second call is not markedly faster, caching is off, the prompt is not identical, or the blocks were evicted between calls. All three are worth knowing about."}]},{"t":"h2","id":"the-costs-and-the-limits","runs":[{"text":"The costs and the limits"}]},{"t":"p","runs":[{"text":"Cache blocks occupy memory that concurrent sequences also want.","b":true},{"text":" On a card with limited VRAM, a large prefix cache reduces how many users you can serve at once. Engines evict least-recently-used blocks, so a busy server with diverse prefixes may keep almost nothing useful."}]},{"t":"p","runs":[{"text":"Sampling parameters do not affect it,","b":true},{"text":" because the cache holds the prompt’s key and value vectors, which do not depend on how tokens are chosen. Different temperatures share a cache happily."}]},{"t":"p","runs":[{"text":"A LoRA adapter or a different model does not share a cache","b":true},{"text":" with another, because the vectors are computed by different weights."}]},{"t":"p","runs":[{"text":"There is a genuine multi-tenant concern.","b":true},{"text":" Because a hit is much faster than a miss, timing differences can in principle reveal whether some other user has recently sent a particular prefix. On a shared server with untrusted users this is a real side channel, and the mitigations are to disable cross-user sharing or to scope caches per tenant. On a single-team server it does not matter, but it should be a deliberate decision rather than an unexamined default."}]},{"t":"h2","id":"where-it-matters-most","runs":[{"text":"Where it matters most"}]},{"t":"p","runs":[{"text":"Agent loops, where a long tool definition block is sent on every step. Classification pipelines with a fixed instruction and a rotating input. Retrieval systems with a stable system prompt. Chat, where each turn extends the previous prefix exactly."}]},{"t":"p","runs":[{"text":"For those workloads, enabling prefix caching and reordering the prompt is frequently a larger win than changing engines, and it takes twenty minutes."}]}],"check":{"question":[{"text":"An agent system with a 2,000-token tool-definition preamble enables prefix caching and sees almost no improvement in time to first token. The preamble is identical in every request except that it opens with the current date and time. What is happening?"}],"options":[{"text":[{"text":"Cache blocks are being evicted between requests because the agent’s traffic is too diverse for the available cache memory."}],"correct":false,"tempting":[{"text":"A real failure mode, but it would not be explained by the one detail given, and eviction usually still leaves some hits."}]},{"text":[{"text":"Tool definitions are sent in a separate field that most engines exclude from the cacheable prefix."}],"correct":false,"tempting":[{"text":"Invents an exclusion; tool definitions are rendered into the prompt string by the template like everything else."}]},{"text":[{"text":"The timestamp changes the first block, so every subsequent block hash differs and the entire preamble misses."}],"correct":true,"tempting":null},{"text":[{"text":"Caching requires identical sampling parameters, and the agent varies temperature between planning and execution steps."}],"correct":false,"tempting":[{"text":"Attributes a dependency the cache does not have; keys and values are computed before any sampling decision."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"continuous-batching","title":"Continuous batching, and the queue underneath it","minutes":9},"next":{"slug":"speculative-decoding","title":"Speculative decoding: several tokens per weight read","minutes":9}}}