{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"attention-and-the-kv-cache","title":"Attention variants, and the cache that eats your memory","url":"https://addaly.com/learn/open-models/attention-and-the-kv-cache","position":25,"of":71,"minutes":10,"answer":"KV cache memory is two times layers times key-value heads times head dimension per token, it grows with every concurrent user, and grouped-query attention.","keyIdea":"KV cache memory is two times layers times key-value heads times head dimension per token, it grows with every concurrent user, and grouped-query attention, cache quantisation and a lower context limit are the three levers that change it."},"body":{"format":"markdown","text":"## Why there is a cache at all\n\nGenerating token 500 requires attending to the keys and values of tokens 1 to 499. Recomputing them every step would be quadratic and absurd, so they are computed once and kept. That store is the KV cache, and on any serious deployment it is the thing that runs out, not the weights.\n\nThe size is exact and worth memorising:\n\n```\nbytes per token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_element\n```\n\nThe leading 2 is for keys and values. For Llama-3.1-8B (32 layers, 8 key-value heads, head dimension 128, fp16) that is 2 × 32 × 8 × 128 × 2 = **131,072 bytes per token**, or 128 KB.\n\n- 8,000 tokens of context: about 1.0 GB\n- 32,000 tokens: about 4.2 GB\n- 128,000 tokens: about 16.8 GB, which is as much as the weights\n\nAnd that is for **one conversation**. Ten concurrent users at 8k context each need 10 GB of cache on top of the model. This is why a server that runs beautifully in a demo falls over with twelve people on it.\n\n> Figure: KV cache for one conversation, Llama-3.1-8B. Line chart of Gigabytes of cache against Context length, tokens: fp16 cache: 128 KB per token (8000, 1) (16000, 2.1) (32000, 4.2) (64000, 8.4) (128000, 16.8); 8-bit cache (8000, 0.5) (16000, 1.05) (32000, 2.1) (64000, 4.2) (128000, 8.4). The weights are 16 GB. At 128k of context the cache alone matches them, and this is one conversation: ten concurrent users at 8k each want 10 GB on top of the model. Quantising the cache halves it, at a cost that shows up on long-context retrieval before it shows up on chat.\n\n## Three attention layouts\n\n**Multi-head attention** gives every query head its own key and value heads. Maximum quality, maximum cache. Llama 2's 7B has 32 query heads and 32 key-value heads, so its cache is 512 KB per token: four times the Llama 3 model above, at a similar parameter count.\n\n**Multi-query attention** gives all query heads a single shared key-value head. Smallest possible cache, some quality cost.\n\n**Grouped-query attention** is the compromise everything now uses: several query heads share one key-value head. The ratio `num_attention_heads / num_key_value_heads` is the saving. At 32 and 8, the cache is a quarter of what full multi-head would need, with a quality difference small enough that the whole field adopted it within a year.\n\nThis is why `num_key_value_heads` is the field to look at in a config when you care about serving. Two models of identical parameter count can differ by four times in how many users one GPU will hold.\n\n**Sliding window attention** attacks the problem differently: each token attends only to the last N tokens. Mistral's early models used a 4,096-token window; Gemma alternates local sliding-window layers with occasional global layers, so most layers keep a small fixed cache while a few see everything. The cache stops growing past the window, which turns a linear memory cost into a constant one. The trade is that information from far back reaches the present only indirectly, through several layers of local attention.\n\n## What you can actually do about it\n\n**Quantise the cache.** Most serving stacks can store keys and values in 8-bit rather than 16-bit, halving the memory for a small quality cost that is usually invisible on ordinary tasks:\n\n```bash\nllama-server -m model.gguf -c 32768 --cache-type-k q8_0 --cache-type-v q8_0\nvllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8 --max-model-len 32768\n```\n\nTest it before you trust it. Cache quantisation hurts more on long-context retrieval than on short chat, because subtle key distinctions are what let the model find a specific fact far back.\n\n**Cap the context you actually allow.** Serving a 128k model with `--max-model-len 8192` is not cowardice; it is choosing to serve sixteen times as many users. Most production traffic never needs more.\n\n**Let the serving layer manage it.** vLLM's paged attention allocates the cache in fixed blocks like operating-system pages instead of reserving a contiguous maximum per request. That removes most of the waste from over-allocating for requests that turn out short, and is a large part of why vLLM serves several times more concurrent users than a naive implementation on the same hardware.\n\n**Reuse the shared prefix.** If every request starts with the same 2,000-token system prompt, prefix caching computes it once and shares it across requests. On a chat product this is often the single biggest saving available, and it is a flag rather than a project.\n\n## The arithmetic to do before you buy\n\nTake your GPU memory. Subtract the weights at your chosen precision. Subtract about 1 to 2 GB of runtime overhead. Divide what remains by (bytes per token × your context limit). That is your concurrency.\n\nFor a 24 GB card running the 8B at Q4 with 8k context: 24 − 5 − 2 = 17 GB, divided by 1.0 GB per conversation, is about 16 concurrent conversations. If that number is smaller than your traffic, the fix is a shorter context limit or cache quantisation long before it is a bigger model."},"check":{"question":"Two 7B models have the same layer count and head dimension. Model A has 32 key-value heads, model B has 8. On the same 24 GB GPU at the same context length, what differs?","options":[{"text":"Model A is four times faster to generate with, since more key-value heads mean more parallel attention work.","correct":false,"tempting":"More key-value heads add memory traffic rather than throughput; if anything the larger cache slows decoding."},{"text":"Nothing meaningful, since both models have the same parameter count and therefore the same memory footprint.","correct":false,"tempting":"Counts only the weights; the cache is a separate allocation that scales with key-value heads and grows per user."},{"text":"Model B serves roughly four times as many concurrent conversations, because its KV cache per token is a quarter the size.","correct":true},{"text":"Model A supports a four times longer context, because more key-value heads give it more positional capacity.","correct":false,"tempting":"Reverses the relationship: a larger cache per token means fewer tokens fit in the same memory, and context capacity is unrelated to head count."}]},"prev":{"slug":"the-chat-template-contract","title":"The chat template is a contract","url":"https://addaly.com/learn/open-models/the-chat-template-contract"},"next":{"slug":"context-length-in-practice","title":"What a 128k context window really means","url":"https://addaly.com/learn/open-models/context-length-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Attention variants, and the cache that eats your memory', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/attention-and-the-kv-cache (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Attention variants, and the cache that eats your memory","courseTitle":"The Open Model Ecosystem","module":{"index":3,"slug":"inside-the-artefact","title":"Inside the files"},"answer":null,"keyIdea":"KV cache memory is two times layers times key-value heads times head dimension per token, it grows with every concurrent user, and grouped-query attention, cache quantisation and a lower context limit are the three levers that change it.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"line","title":"KV cache for one conversation, Llama-3.1-8B","series":[{"label":"fp16 cache: 128 KB per token","points":[[8000,1],[16000,2.1],[32000,4.2],[64000,8.4],[128000,16.8]]},{"label":"8-bit cache","points":[[8000,0.5],[16000,1.05],[32000,2.1],[64000,4.2],[128000,8.4]]}],"xLabel":"Context length, tokens","yLabel":"Gigabytes of cache","caption":"The weights are 16 GB. At 128k of context the cache alone matches them, and this is one conversation: ten concurrent users at 8k each want 10 GB on top of the model. Quantising the cache halves it, at a cost that shows up on long-context retrieval before it shows up on chat."},"alt":"Figure: KV cache for one conversation, Llama-3.1-8B. Line chart of Gigabytes of cache against Context length, tokens: fp16 cache: 128 KB per token (8000, 1) (16000, 2.1) (32000, 4.2) (64000, 8.4) (128000, 16.8); 8-bit cache (8000, 0.5) (16000, 1.05) (32000, 2.1) (64000, 4.2) (128000, 8.4). The weights are 16 GB. At 128k of context the cache alone matches them, and this is one conversation: ten concurrent users at 8k each want 10 GB on top of the model. Quantising the cache halves it, at a cost that shows up on long-context retrieval before it shows up on chat."}},"blocks":[{"t":"h2","id":"why-there-is-a-cache-at-all","runs":[{"text":"Why there is a cache at all"}]},{"t":"p","runs":[{"text":"Generating token 500 requires attending to the keys and values of tokens 1 to 499. Recomputing them every step would be quadratic and absurd, so they are computed once and kept. That store is the KV cache, and on any serious deployment it is the thing that runs out, not the weights."}]},{"t":"p","runs":[{"text":"The size is exact and worth memorising:"}]},{"t":"code","lang":"","label":"Text","text":"bytes per token = 2 * num_layers * num_kv_heads * head_dim * bytes_per_element"},{"t":"p","runs":[{"text":"The leading 2 is for keys and values. For Llama-3.1-8B (32 layers, 8 key-value heads, head dimension 128, fp16) that is 2 × 32 × 8 × 128 × 2 = "},{"text":"131,072 bytes per token","b":true},{"text":", or 128 KB."}]},{"t":"ul","items":[[{"text":"8,000 tokens of context: about 1.0 GB"}],[{"text":"32,000 tokens: about 4.2 GB"}],[{"text":"128,000 tokens: about 16.8 GB, which is as much as the weights"}]]},{"t":"p","runs":[{"text":"And that is for "},{"text":"one conversation","b":true},{"text":". Ten concurrent users at 8k context each need 10 GB of cache on top of the model. This is why a server that runs beautifully in a demo falls over with twelve people on it."}]},{"t":"h2","id":"three-attention-layouts","runs":[{"text":"Three attention layouts"}]},{"t":"p","runs":[{"text":"Multi-head attention","b":true},{"text":" gives every query head its own key and value heads. Maximum quality, maximum cache. Llama 2’s 7B has 32 query heads and 32 key-value heads, so its cache is 512 KB per token: four times the Llama 3 model above, at a similar parameter count."}]},{"t":"p","runs":[{"text":"Multi-query attention","b":true},{"text":" gives all query heads a single shared key-value head. Smallest possible cache, some quality cost."}]},{"t":"p","runs":[{"text":"Grouped-query attention","b":true},{"text":" is the compromise everything now uses: several query heads share one key-value head. The ratio "},{"text":"num_attention_heads / num_key_value_heads","code":true},{"text":" is the saving. At 32 and 8, the cache is a quarter of what full multi-head would need, with a quality difference small enough that the whole field adopted it within a year."}]},{"t":"p","runs":[{"text":"This is why "},{"text":"num_key_value_heads","code":true},{"text":" is the field to look at in a config when you care about serving. Two models of identical parameter count can differ by four times in how many users one GPU will hold."}]},{"t":"p","runs":[{"text":"Sliding window attention","b":true},{"text":" attacks the problem differently: each token attends only to the last N tokens. Mistral’s early models used a 4,096-token window; Gemma alternates local sliding-window layers with occasional global layers, so most layers keep a small fixed cache while a few see everything. The cache stops growing past the window, which turns a linear memory cost into a constant one. The trade is that information from far back reaches the present only indirectly, through several layers of local attention."}]},{"t":"h2","id":"what-you-can-actually-do-about-it","runs":[{"text":"What you can actually do about it"}]},{"t":"p","runs":[{"text":"Quantise the cache.","b":true},{"text":" Most serving stacks can store keys and values in 8-bit rather than 16-bit, halving the memory for a small quality cost that is usually invisible on ordinary tasks:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -c 32768 --cache-type-k q8_0 --cache-type-v q8_0\nvllm serve Qwen/Qwen3-8B --kv-cache-dtype fp8 --max-model-len 32768"},{"t":"p","runs":[{"text":"Test it before you trust it. Cache quantisation hurts more on long-context retrieval than on short chat, because subtle key distinctions are what let the model find a specific fact far back."}]},{"t":"p","runs":[{"text":"Cap the context you actually allow.","b":true},{"text":" Serving a 128k model with "},{"text":"--max-model-len 8192","code":true},{"text":" is not cowardice; it is choosing to serve sixteen times as many users. Most production traffic never needs more."}]},{"t":"p","runs":[{"text":"Let the serving layer manage it.","b":true},{"text":" vLLM’s paged attention allocates the cache in fixed blocks like operating-system pages instead of reserving a contiguous maximum per request. That removes most of the waste from over-allocating for requests that turn out short, and is a large part of why vLLM serves several times more concurrent users than a naive implementation on the same hardware."}]},{"t":"p","runs":[{"text":"Reuse the shared prefix.","b":true},{"text":" If every request starts with the same 2,000-token system prompt, prefix caching computes it once and shares it across requests. On a chat product this is often the single biggest saving available, and it is a flag rather than a project."}]},{"t":"h2","id":"the-arithmetic-to-do-before-you-buy","runs":[{"text":"The arithmetic to do before you buy"}]},{"t":"p","runs":[{"text":"Take your GPU memory. Subtract the weights at your chosen precision. Subtract about 1 to 2 GB of runtime overhead. Divide what remains by (bytes per token × your context limit). That is your concurrency."}]},{"t":"p","runs":[{"text":"For a 24 GB card running the 8B at Q4 with 8k context: 24 − 5 − 2 = 17 GB, divided by 1.0 GB per conversation, is about 16 concurrent conversations. If that number is smaller than your traffic, the fix is a shorter context limit or cache quantisation long before it is a bigger model."}]}],"check":{"question":[{"text":"Two 7B models have the same layer count and head dimension. Model A has 32 key-value heads, model B has 8. On the same 24 GB GPU at the same context length, what differs?"}],"options":[{"text":[{"text":"Model A is four times faster to generate with, since more key-value heads mean more parallel attention work."}],"correct":false,"tempting":[{"text":"More key-value heads add memory traffic rather than throughput; if anything the larger cache slows decoding."}]},{"text":[{"text":"Nothing meaningful, since both models have the same parameter count and therefore the same memory footprint."}],"correct":false,"tempting":[{"text":"Counts only the weights; the cache is a separate allocation that scales with key-value heads and grows per user."}]},{"text":[{"text":"Model B serves roughly four times as many concurrent conversations, because its KV cache per token is a quarter the size."}],"correct":true,"tempting":null},{"text":[{"text":"Model A supports a four times longer context, because more key-value heads give it more positional capacity."}],"correct":false,"tempting":[{"text":"Reverses the relationship: a larger cache per token means fewer tokens fit in the same memory, and context capacity is unrelated to head count."}]}],"order":[1,0,3,2],"why":null},"prev":{"slug":"the-chat-template-contract","title":"The chat template is a contract","minutes":9},"next":{"slug":"context-length-in-practice","title":"What a 128k context window really means","minutes":9}}}