{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"the-kv-cache-in-depth","title":"The KV cache, in depth","url":"https://addaly.com/learn/running-models-yourself/the-kv-cache-in-depth","position":12,"of":82,"minutes":10,"answer":"The KV cache is the term that decides whether a model fits, and its size per token is set by the attention architecture: grouped-query.","keyIdea":"The KV cache is the term that decides whether a model fits, and its size per token is set by the attention architecture: grouped-query, sliding-window and latent attention differ by several times at the same parameter count."},"body":{"format":"markdown","text":"## Why the cache exists at all\n\nAttention needs, for every new token, the key and value vectors of every previous token. Recomputing them would make generation quadratic in length and unbearably slow. So they are computed once and kept. That store is the KV cache, and it is the single term that turns a model which fits into a model which does not.\n\nThe size, per token:\n\n```\nbytes = 2 x layers x kv_heads x head_dim x bytes_per_element\n```\n\nThe leading 2 is K and V. Nothing else in local inference has a formula this simple with consequences this large.\n\n## Why two 8B models differ sixfold\n\nThe term that varies wildly between architectures is `kv_heads`.\n\n**Multi-head attention (MHA)** gives every query head its own key and value head. This is what models before about 2023 did. Llama 2 13B has 40 layers and 40 KV heads with head dim 128, so 800 KiB per token: 3.1 GiB at only 4k context, on top of the weights.\n\n> Figure: Two models of similar size, six times the cache. Bars (KiB of KV cache per token of context): Llama 2 13B, multi-head, 40 KV heads 800; Llama 3.1 8B, grouped-query, 8 KV heads 128; That same 8B with an 8-bit cache 64. Only the config file tells you which you have: layers, KV heads and head dimension, doubled for K and V. Grouped-query attention shares one KV head across several query heads, and it was a memory decision rather than a quality one: it is the reason long context became affordable at all.\n\n**Grouped-query attention (GQA)** shares one KV head across several query heads. Llama 3.1 8B has 32 query heads and 8 KV heads: the cache shrinks fourfold, to 128 KiB per token, with a quality cost small enough that essentially every model since has adopted it. GQA is the reason long context became affordable, and it is worth understanding that this was a memory decision, not a quality one.\n\n**Multi-query attention (MQA)** takes it to one KV head for all queries. Cheapest cache, most quality cost, now uncommon.\n\n**Multi-head latent attention (MLA)**, used by the DeepSeek family, compresses K and V into a shared low-rank latent vector and reconstructs them on the fly. The cache becomes several times smaller again than GQA. It costs extra arithmetic per token, which is a good trade on a GPU and a worse one on a CPU.\n\nSo when you read \"8B model\", the memory it needs at 32k context can be one gigabyte or six, and only the config file tells you which.\n\n## Sliding windows and hybrid attention\n\nSeveral model families do not keep the whole cache. **Sliding-window attention** limits each layer to the last N tokens (4,096 in Mistral's original design) so the cache stops growing once the window fills. Gemma and several recent models interleave: most layers use a small local window, one in every four or five uses full attention. The result is a cache that grows very slowly with context.\n\nThis is real memory savings and a real capability limit. A model whose local layers see 1,024 tokens cannot use fine detail from 40,000 tokens ago in those layers, however large the advertised window is. Where you see a model claiming a huge context on modest memory, this is usually why, and it is a fair trade rather than a trick, but it is a trade.\n\n## Quantising the cache\n\nThe cache does not have to be fp16. llama.cpp lets you store it at 8 or 4 bits:\n\n```bash\nllama-server -m model.gguf -c 32768 --flash-attn \\\n  --cache-type-k q8_0 --cache-type-v q8_0\n```\n\nAt `q8_0` the halving is close to free: the difference is hard to find on ordinary tasks. At `q4_0` it is not free: recall over long context degrades noticeably, which is the exact capability you extended the context to use, so a 4-bit cache with a 64k window is frequently self-defeating.\n\nTwo mechanical details people trip over. Flash attention is generally required for quantised cache kernels; without it the flags are ignored or the run fails. And K tolerates quantisation better than V, so `--cache-type-k q4_0 --cache-type-v q8_0` is a reasonable asymmetric setting when you are tight.\n\nvLLM has its own equivalent (`--kv-cache-dtype fp8`), which on Ada and Hopper cards is nearly lossless because fp8 has an exponent and can represent the outliers that hurt integer quantisation.\n\n## How the cache is stored matters as much as its size\n\nA naive server reserves the worst case per request: if `max_model_len` is 32k, every concurrent request gets 32k of cache reserved whether it uses 200 tokens or 30,000. On a 24 GB card that limits you to a handful of concurrent users regardless of what they actually send.\n\nPagedAttention, vLLM's central idea, stores the cache in fixed-size blocks the way an operating system stores memory in pages. A request occupies only the blocks it has filled, blocks are allocated as it grows, and there is almost no fragmentation. The same card then holds many times more concurrent sequences. This is not a small optimisation; it is most of the reason a serving engine outperforms a naive loop at concurrency.\n\n## The number to carry\n\nFor a rough plan: a modern GQA model at around 8B costs roughly 100 to 150 KiB per token of context in fp16, halving at 8-bit. Multiply by your context length, add it to the weights, add 15% overhead, and you have the answer before downloading anything.\n\nAnd when a model that fit yesterday runs out of memory today, the cache is almost always what changed: a longer document, more concurrent users, or a context setting somebody raised."},"check":{"question":"A team runs an 8B GQA model at 8k context on a 12 GB card with room to spare. They raise the context to 64k and hit out-of-memory errors. Someone proposes dropping the weights from Q4_K_M to Q3_K_M. Why is that a poor first move?","options":[{"text":"Q3 quants are not supported alongside long context in most engines, so the change would fail before it saved anything.","correct":false,"tempting":"Invents an incompatibility; bit rate and context length are independent settings in every mainstream engine."},{"text":"It saves about a gigabyte of weights while the cache grew by roughly seven, and it pays real quality: quantising the cache to 8-bit halves the term that actually grew.","correct":true},{"text":"Weight quantisation does not reduce memory during inference because the weights are dequantised into fp16 buffers before each matrix multiply.","correct":false,"tempting":"Describes a per-block operation as if it were a full-model expansion; dequantisation happens in small tiles and the resident file stays small."},{"text":"The problem is fragmentation rather than capacity, so the fix is a serving engine with paged cache blocks and not any change to precision.","correct":false,"tempting":"Reaches for a real mechanism that matters at concurrency, not for one long single-stream context where the cache genuinely is that large."}]},"prev":{"slug":"small-hardware-reality","title":"What actually runs on 8GB","url":"https://addaly.com/learn/running-models-yourself/small-hardware-reality"},"next":{"slug":"prefill-and-decode","title":"Prefill and decode are two different machines","url":"https://addaly.com/learn/running-models-yourself/prefill-and-decode"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The KV cache, in depth', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/the-kv-cache-in-depth (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The KV cache, in depth","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"The KV cache is the term that decides whether a model fits, and its size per token is set by the attention architecture: grouped-query, sliding-window and latent attention differ by several times at the same parameter count.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"KiB of KV cache per token of context","title":"Two models of similar size, six times the cache","series":[{"label":"Llama 2 13B, multi-head, 40 KV heads","value":800},{"label":"Llama 3.1 8B, grouped-query, 8 KV heads","value":128,"highlight":true},{"label":"That same 8B with an 8-bit cache","value":64}],"caption":"Only the config file tells you which you have: layers, KV heads and head dimension, doubled for K and V. Grouped-query attention shares one KV head across several query heads, and it was a memory decision rather than a quality one: it is the reason long context became affordable at all."},"alt":"Figure: Two models of similar size, six times the cache. Bars (KiB of KV cache per token of context): Llama 2 13B, multi-head, 40 KV heads 800; Llama 3.1 8B, grouped-query, 8 KV heads 128; That same 8B with an 8-bit cache 64. Only the config file tells you which you have: layers, KV heads and head dimension, doubled for K and V. Grouped-query attention shares one KV head across several query heads, and it was a memory decision rather than a quality one: it is the reason long context became affordable at all."}},"blocks":[{"t":"h2","id":"why-the-cache-exists-at-all","runs":[{"text":"Why the cache exists at all"}]},{"t":"p","runs":[{"text":"Attention needs, for every new token, the key and value vectors of every previous token. Recomputing them would make generation quadratic in length and unbearably slow. So they are computed once and kept. That store is the KV cache, and it is the single term that turns a model which fits into a model which does not."}]},{"t":"p","runs":[{"text":"The size, per token:"}]},{"t":"code","lang":"","label":"Text","text":"bytes = 2 x layers x kv_heads x head_dim x bytes_per_element"},{"t":"p","runs":[{"text":"The leading 2 is K and V. Nothing else in local inference has a formula this simple with consequences this large."}]},{"t":"h2","id":"why-two-8b-models-differ-sixfold","runs":[{"text":"Why two 8B models differ sixfold"}]},{"t":"p","runs":[{"text":"The term that varies wildly between architectures is "},{"text":"kv_heads","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"Multi-head attention (MHA)","b":true},{"text":" gives every query head its own key and value head. This is what models before about 2023 did. Llama 2 13B has 40 layers and 40 KV heads with head dim 128, so 800 KiB per token: 3.1 GiB at only 4k context, on top of the weights."}]},{"t":"p","runs":[{"text":"Grouped-query attention (GQA)","b":true},{"text":" shares one KV head across several query heads. Llama 3.1 8B has 32 query heads and 8 KV heads: the cache shrinks fourfold, to 128 KiB per token, with a quality cost small enough that essentially every model since has adopted it. GQA is the reason long context became affordable, and it is worth understanding that this was a memory decision, not a quality one."}]},{"t":"p","runs":[{"text":"Multi-query attention (MQA)","b":true},{"text":" takes it to one KV head for all queries. Cheapest cache, most quality cost, now uncommon."}]},{"t":"p","runs":[{"text":"Multi-head latent attention (MLA)","b":true},{"text":", used by the DeepSeek family, compresses K and V into a shared low-rank latent vector and reconstructs them on the fly. The cache becomes several times smaller again than GQA. It costs extra arithmetic per token, which is a good trade on a GPU and a worse one on a CPU."}]},{"t":"p","runs":[{"text":"So when you read “8B model”, the memory it needs at 32k context can be one gigabyte or six, and only the config file tells you which."}]},{"t":"h2","id":"sliding-windows-and-hybrid-attention","runs":[{"text":"Sliding windows and hybrid attention"}]},{"t":"p","runs":[{"text":"Several model families do not keep the whole cache. "},{"text":"Sliding-window attention","b":true},{"text":" limits each layer to the last N tokens (4,096 in Mistral’s original design) so the cache stops growing once the window fills. Gemma and several recent models interleave: most layers use a small local window, one in every four or five uses full attention. The result is a cache that grows very slowly with context."}]},{"t":"p","runs":[{"text":"This is real memory savings and a real capability limit. A model whose local layers see 1,024 tokens cannot use fine detail from 40,000 tokens ago in those layers, however large the advertised window is. Where you see a model claiming a huge context on modest memory, this is usually why, and it is a fair trade rather than a trick, but it is a trade."}]},{"t":"h2","id":"quantising-the-cache","runs":[{"text":"Quantising the cache"}]},{"t":"p","runs":[{"text":"The cache does not have to be fp16. llama.cpp lets you store it at 8 or 4 bits:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -c 32768 --flash-attn \\\n  --cache-type-k q8_0 --cache-type-v q8_0"},{"t":"p","runs":[{"text":"At "},{"text":"q8_0","code":true},{"text":" the halving is close to free: the difference is hard to find on ordinary tasks. At "},{"text":"q4_0","code":true},{"text":" it is not free: recall over long context degrades noticeably, which is the exact capability you extended the context to use, so a 4-bit cache with a 64k window is frequently self-defeating."}]},{"t":"p","runs":[{"text":"Two mechanical details people trip over. Flash attention is generally required for quantised cache kernels; without it the flags are ignored or the run fails. And K tolerates quantisation better than V, so "},{"text":"--cache-type-k q4_0 --cache-type-v q8_0","code":true},{"text":" is a reasonable asymmetric setting when you are tight."}]},{"t":"p","runs":[{"text":"vLLM has its own equivalent ("},{"text":"--kv-cache-dtype fp8","code":true},{"text":"), which on Ada and Hopper cards is nearly lossless because fp8 has an exponent and can represent the outliers that hurt integer quantisation."}]},{"t":"h2","id":"how-the-cache-is-stored-matters-as-much-as-its-size","runs":[{"text":"How the cache is stored matters as much as its size"}]},{"t":"p","runs":[{"text":"A naive server reserves the worst case per request: if "},{"text":"max_model_len","code":true},{"text":" is 32k, every concurrent request gets 32k of cache reserved whether it uses 200 tokens or 30,000. On a 24 GB card that limits you to a handful of concurrent users regardless of what they actually send."}]},{"t":"p","runs":[{"text":"PagedAttention, vLLM’s central idea, stores the cache in fixed-size blocks the way an operating system stores memory in pages. A request occupies only the blocks it has filled, blocks are allocated as it grows, and there is almost no fragmentation. The same card then holds many times more concurrent sequences. This is not a small optimisation; it is most of the reason a serving engine outperforms a naive loop at concurrency."}]},{"t":"h2","id":"the-number-to-carry","runs":[{"text":"The number to carry"}]},{"t":"p","runs":[{"text":"For a rough plan: a modern GQA model at around 8B costs roughly 100 to 150 KiB per token of context in fp16, halving at 8-bit. Multiply by your context length, add it to the weights, add 15% overhead, and you have the answer before downloading anything."}]},{"t":"p","runs":[{"text":"And when a model that fit yesterday runs out of memory today, the cache is almost always what changed: a longer document, more concurrent users, or a context setting somebody raised."}]}],"check":{"question":[{"text":"A team runs an 8B GQA model at 8k context on a 12 GB card with room to spare. They raise the context to 64k and hit out-of-memory errors. Someone proposes dropping the weights from Q4_K_M to Q3_K_M. Why is that a poor first move?"}],"options":[{"text":[{"text":"Q3 quants are not supported alongside long context in most engines, so the change would fail before it saved anything."}],"correct":false,"tempting":[{"text":"Invents an incompatibility; bit rate and context length are independent settings in every mainstream engine."}]},{"text":[{"text":"It saves about a gigabyte of weights while the cache grew by roughly seven, and it pays real quality: quantising the cache to 8-bit halves the term that actually grew."}],"correct":true,"tempting":null},{"text":[{"text":"Weight quantisation does not reduce memory during inference because the weights are dequantised into fp16 buffers before each matrix multiply."}],"correct":false,"tempting":[{"text":"Describes a per-block operation as if it were a full-model expansion; dequantisation happens in small tiles and the resident file stays small."}]},{"text":[{"text":"The problem is fragmentation rather than capacity, so the fix is a serving engine with paged cache blocks and not any change to precision."}],"correct":false,"tempting":[{"text":"Reaches for a real mechanism that matters at concurrency, not for one long single-stream context where the cache genuinely is that large."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"small-hardware-reality","title":"What actually runs on 8GB","minutes":8},"next":{"slug":"prefill-and-decode","title":"Prefill and decode are two different machines","minutes":9}}}