The key-value cache, and the engineering built around it

The waste that had to be removed

Generating token 500 means running attention over 499 earlier positions. Their key and value vectors depend only on the tokens themselves, which have not changed. Recomputing them every step would multiply the cost of a 500-token response by a factor of hundreds.

So they are cached. For each layer, for each position, the keys and values are computed once and kept. Each new token computes only its own query, key and value, appends the latter two to the cache, and attends over everything stored.

This is the single most important optimisation in language model serving, and it is why the second token arrives much faster than the first.

The cache is large

The arithmetic, which you can do for any model:

bytes = 2 (key and value)
      × layers
      × kv_heads × head_dim
      × sequence_length
      × bytes_per_number

For our 8B model (32 layers, 8 key-value heads, head dimension 128, bf16) one token costs 2 × 32 × 1,024 × 2 = 131 kilobytes. A 32,000-token conversation therefore costs about 4.2 GB, on top of the 16 GB of weights.

Now the number that matters: the weights are shared across all users, but every user needs their own cache. Ten users with 32,000-token contexts need 42 GB of cache. That is why the memory arithmetic in module one came out the way it did, and why long-context serving is expensive in a way that surprises people who only counted parameters.

Grouped-query attention exists for this reason

Look again at the formula: the cache is proportional to the number of key-value heads.

• Multi-head attention: 32 query heads, 32 key-value heads. Full-size cache.
• Multi-query attention: 32 query heads, 1 key-value head. Cache divided by 32, with a measurable quality cost.
• Grouped-query attention: 32 query heads, 8 key-value heads. Cache divided by 4, with quality close to full multi-head.

Grouped-query attention is now near-universal in open models, and it was adopted for exactly one reason: it shrinks the cache. When you see num_key_value_heads smaller than num_attention_heads in a config, you are looking at a memory decision, not a quality one.

Newer designs go further: compressing the cache into a lower-dimensional latent form and reconstructing keys and values on the fly, trading a little arithmetic for a large memory saving.

Sliding window and hybrid attention

Another route: stop attending to everything. A sliding-window layer attends only to the last few thousand tokens, so its cache stops growing. Stack such layers with occasional full-attention layers and information can still propagate a long way: a token influences its window, which influences the next, and so on up the stack.

The trade is real and should be stated plainly: exact recall of a specific detail from 60,000 tokens back is harder for a windowed layer than a full one. Many current models use a hybrid, with a minority of full-attention layers, precisely to keep some exact long-range recall while bounding the cache.

Paged attention

Early serving systems allocated a contiguous block of memory per request, sized for the maximum possible length. Most requests never reached it, so most of that memory sat empty: reported waste of 60 to 80 per cent.

Paged attention borrows the idea of virtual memory: store the cache in fixed-size blocks, allocated on demand, tracked by a table. Utilisation goes up sharply, more requests fit, throughput rises several-fold. It also makes prefix sharing natural: two requests with the same system prompt can point at the same physical blocks instead of each holding a copy.

vLLM, which introduced this, is free and open source, and it is the reason a self-hosted deployment can be within reach of a small team.

What to take away

Three things you can act on.

Long conversations cost memory, not just tokens. Trimming history helps your provider’s economics and often your latency.

A shared, stable prefix is worth engineering for. Identical opening tokens across requests can be shared or cached; a per-request timestamp at the top of your system prompt destroys that. This is the mechanism behind prompt caching in module four.

Cache size is a first-class model property. When choosing a model to self-host, compute its per-token cache cost alongside its parameter count. Two models of the same size can differ fourfold in how many users they will serve.