The KV cache, in depth

Why the cache exists at all

Attention needs, for every new token, the key and value vectors of every previous token. Recomputing them would make generation quadratic in length and unbearably slow. So they are computed once and kept. That store is the KV cache, and it is the single term that turns a model which fits into a model which does not.

The size, per token:

bytes = 2 x layers x kv_heads x head_dim x bytes_per_element

The leading 2 is K and V. Nothing else in local inference has a formula this simple with consequences this large.

Why two 8B models differ sixfold

The term that varies wildly between architectures is kv_heads.

Multi-head attention (MHA) gives every query head its own key and value head. This is what models before about 2023 did. Llama 2 13B has 40 layers and 40 KV heads with head dim 128, so 800 KiB per token: 3.1 GiB at only 4k context, on top of the weights.

Grouped-query attention (GQA) shares one KV head across several query heads. Llama 3.1 8B has 32 query heads and 8 KV heads: the cache shrinks fourfold, to 128 KiB per token, with a quality cost small enough that essentially every model since has adopted it. GQA is the reason long context became affordable, and it is worth understanding that this was a memory decision, not a quality one.

Multi-query attention (MQA) takes it to one KV head for all queries. Cheapest cache, most quality cost, now uncommon.

Multi-head latent attention (MLA), used by the DeepSeek family, compresses K and V into a shared low-rank latent vector and reconstructs them on the fly. The cache becomes several times smaller again than GQA. It costs extra arithmetic per token, which is a good trade on a GPU and a worse one on a CPU.

So when you read “8B model”, the memory it needs at 32k context can be one gigabyte or six, and only the config file tells you which.

Sliding windows and hybrid attention

Several model families do not keep the whole cache. Sliding-window attention limits each layer to the last N tokens (4,096 in Mistral’s original design) so the cache stops growing once the window fills. Gemma and several recent models interleave: most layers use a small local window, one in every four or five uses full attention. The result is a cache that grows very slowly with context.

This is real memory savings and a real capability limit. A model whose local layers see 1,024 tokens cannot use fine detail from 40,000 tokens ago in those layers, however large the advertised window is. Where you see a model claiming a huge context on modest memory, this is usually why, and it is a fair trade rather than a trick, but it is a trade.

Quantising the cache

The cache does not have to be fp16. llama.cpp lets you store it at 8 or 4 bits:

llama-server -m model.gguf -c 32768 --flash-attn \
  --cache-type-k q8_0 --cache-type-v q8_0

At q8_0 the halving is close to free: the difference is hard to find on ordinary tasks. At q4_0 it is not free: recall over long context degrades noticeably, which is the exact capability you extended the context to use, so a 4-bit cache with a 64k window is frequently self-defeating.

Two mechanical details people trip over. Flash attention is generally required for quantised cache kernels; without it the flags are ignored or the run fails. And K tolerates quantisation better than V, so --cache-type-k q4_0 --cache-type-v q8_0 is a reasonable asymmetric setting when you are tight.

vLLM has its own equivalent (--kv-cache-dtype fp8), which on Ada and Hopper cards is nearly lossless because fp8 has an exponent and can represent the outliers that hurt integer quantisation.

How the cache is stored matters as much as its size

A naive server reserves the worst case per request: if max_model_len is 32k, every concurrent request gets 32k of cache reserved whether it uses 200 tokens or 30,000. On a 24 GB card that limits you to a handful of concurrent users regardless of what they actually send.

PagedAttention, vLLM’s central idea, stores the cache in fixed-size blocks the way an operating system stores memory in pages. A request occupies only the blocks it has filled, blocks are allocated as it grows, and there is almost no fragmentation. The same card then holds many times more concurrent sequences. This is not a small optimisation; it is most of the reason a serving engine outperforms a naive loop at concurrency.

The number to carry

For a rough plan: a modern GQA model at around 8B costs roughly 100 to 150 KiB per token of context in fp16, halving at 8-bit. Multiply by your context length, add it to the weights, add 15% overhead, and you have the answer before downloading anything.

And when a model that fit yesterday runs out of memory today, the cache is almost always what changed: a longer document, more concurrent users, or a context setting somebody raised.