Prefix caching, and how to design prompts for it

The work you are repeating

Every request in a typical application starts with the same 1,500 tokens: a system prompt, formatting rules, few-shot examples, tool definitions. Without caching, every one of those requests recomputes the identical prefill. On a server handling ten requests a second, that is fifteen thousand tokens per second of work whose answer never changes.

Prefix caching keeps the KV cache for a prefix that has already been processed and reuses it. The saving is not marginal. A request with a 1,500-token shared preamble and a 100-token question drops from 1,600 tokens of prefill to 100: a 94% reduction in the phase that dominates time to first token.

How it is implemented

The cache is stored in fixed-size blocks, typically sixteen tokens each. Each block is hashed together with the hash of every block before it, so a block’s identity encodes the whole sequence that led to it. When a new request arrives, the server hashes its prompt block by block and looks each up; matching blocks are reused, and computation begins at the first block that misses.

Two properties follow directly. The match must be an exact prefix from the very first token: a shared middle is worth nothing. And it is automatic: nothing has to be declared, because identical prefixes hash identically.

SGLang’s RadixAttention organises these blocks as a prefix tree, which makes partial sharing between many branching conversations natural and cheap. vLLM enables the same idea with --enable-prefix-caching, on by default in recent versions. llama.cpp keeps a conversation on the same slot so its cache survives across turns.

Designing prompts to hit the cache

This is where a small amount of care pays repeatedly.

Put the fixed part first and the variable part last. Everything up to the first difference is cacheable; everything after it is not. So system prompt, then tool definitions, then examples, then retrieved documents, then the user’s question.

Take dynamic values out of the preamble. A timestamp, a session id, or a user’s name at the top of the system prompt invalidates the entire cache for every request. This single mistake is the most common cause of a prefix cache achieving almost no hits, and it is invisible unless you check.

Keep the order of retrieved documents stable. If three documents are concatenated in a different order for two similar queries, the prefix diverges at the first one.

Prefer appending to rewriting. A conversation that appends turns keeps its whole history cached; one that summarises and rebuilds the prompt each turn throws the cache away every time.

Measure the hit rate

Do not assume it is working. vLLM exposes prefix cache hit statistics through its metrics endpoint, and the effect is visible directly in time to first token:

# same long prompt twice, timed
time curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d @long_prompt.json > /dev/null
time curl -s http://localhost:8000/v1/chat/completions -H 'Content-Type: application/json' \
  -d @long_prompt.json > /dev/null

If the second call is not markedly faster, caching is off, the prompt is not identical, or the blocks were evicted between calls. All three are worth knowing about.

The costs and the limits

Cache blocks occupy memory that concurrent sequences also want. On a card with limited VRAM, a large prefix cache reduces how many users you can serve at once. Engines evict least-recently-used blocks, so a busy server with diverse prefixes may keep almost nothing useful.

Sampling parameters do not affect it, because the cache holds the prompt’s key and value vectors, which do not depend on how tokens are chosen. Different temperatures share a cache happily.

A LoRA adapter or a different model does not share a cache with another, because the vectors are computed by different weights.

There is a genuine multi-tenant concern. Because a hit is much faster than a miss, timing differences can in principle reveal whether some other user has recently sent a particular prefix. On a shared server with untrusted users this is a real side channel, and the mitigations are to disable cross-user sharing or to scope caches per tenant. On a single-team server it does not matter, but it should be a deliberate decision rather than an unexamined default.

Where it matters most

Agent loops, where a long tool definition block is sent on every step. Classification pipelines with a fixed instruction and a rotating input. Retrieval systems with a stable system prompt. Chat, where each turn extends the previous prefix exactly.

For those workloads, enabling prefix caching and reordering the prompt is frequently a larger win than changing engines, and it takes twenty minutes.