{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"kv-cache-and-attention-variants","title":"The key-value cache, and the engineering built around it","url":"https://addaly.com/learn/how-llms-work/kv-cache-and-attention-variants","position":19,"of":78,"minutes":9,"answer":"Generation is fast only because every earlier token's keys and values are cached, and that cache (not the weights) is what usually limits how many users a GPU can serve.","keyIdea":"Generation is fast only because every earlier token's keys and values are cached, and that cache (not the weights) is what usually limits how many users a GPU can serve."},"body":{"format":"markdown","text":"## The waste that had to be removed\n\nGenerating token 500 means running attention over 499 earlier positions. Their key and value vectors depend only on the tokens themselves, which have not changed. Recomputing them every step would multiply the cost of a 500-token response by a factor of hundreds.\n\nSo they are cached. For each layer, for each position, the keys and values are computed once and kept. Each new token computes only its own query, key and value, appends the latter two to the cache, and attends over everything stored.\n\nThis is the single most important optimisation in language model serving, and it is why the second token arrives much faster than the first.\n\n## The cache is large\n\nThe arithmetic, which you can do for any model:\n\n```text\nbytes = 2 (key and value)\n      × layers\n      × kv_heads × head_dim\n      × sequence_length\n      × bytes_per_number\n```\n\nFor our 8B model (32 layers, 8 key-value heads, head dimension 128, bf16) one token costs 2 × 32 × 1,024 × 2 = **131 kilobytes**. A 32,000-token conversation therefore costs about **4.2 GB**, on top of the 16 GB of weights.\n\nNow the number that matters: the weights are shared across all users, but **every user needs their own cache**. Ten users with 32,000-token contexts need 42 GB of cache. That is why the memory arithmetic in module one came out the way it did, and why long-context serving is expensive in a way that surprises people who only counted parameters.\n\n## Grouped-query attention exists for this reason\n\nLook again at the formula: the cache is proportional to the number of *key-value* heads.\n\n- **Multi-head attention**: 32 query heads, 32 key-value heads. Full-size cache.\n- **Multi-query attention**: 32 query heads, 1 key-value head. Cache divided by 32, with a measurable quality cost.\n- **Grouped-query attention**: 32 query heads, 8 key-value heads. Cache divided by 4, with quality close to full multi-head.\n\nGrouped-query attention is now near-universal in open models, and it was adopted for exactly one reason: it shrinks the cache. When you see `num_key_value_heads` smaller than `num_attention_heads` in a config, you are looking at a memory decision, not a quality one.\n\n> Figure: One 32,000-token conversation, same eight-billion model. Bars (gigabytes of memory): Cache: multi-head, 32 kv heads 16.8; Cache: grouped-query, 8 kv heads 4.2; Cache: multi-query, 1 kv head 0.52; Model weights, shared by all users 16. Per token the cache costs 2 × layers × kv_heads × head_dim × bytes, so it is proportional to the key-value head count and nothing else. The weights load once and serve everybody; the cache is per user and grows with every token. Grouped-query attention exists for this single reason.\n\nNewer designs go further: compressing the cache into a lower-dimensional latent form and reconstructing keys and values on the fly, trading a little arithmetic for a large memory saving.\n\n## Sliding window and hybrid attention\n\nAnother route: stop attending to everything. A sliding-window layer attends only to the last few thousand tokens, so its cache stops growing. Stack such layers with occasional full-attention layers and information can still propagate a long way: a token influences its window, which influences the next, and so on up the stack.\n\nThe trade is real and should be stated plainly: exact recall of a specific detail from 60,000 tokens back is harder for a windowed layer than a full one. Many current models use a hybrid, with a minority of full-attention layers, precisely to keep some exact long-range recall while bounding the cache.\n\n## Paged attention\n\nEarly serving systems allocated a contiguous block of memory per request, sized for the maximum possible length. Most requests never reached it, so most of that memory sat empty: reported waste of 60 to 80 per cent.\n\nPaged attention borrows the idea of virtual memory: store the cache in fixed-size blocks, allocated on demand, tracked by a table. Utilisation goes up sharply, more requests fit, throughput rises several-fold. It also makes prefix sharing natural: two requests with the same system prompt can point at the same physical blocks instead of each holding a copy.\n\nvLLM, which introduced this, is free and open source, and it is the reason a self-hosted deployment can be within reach of a small team.\n\n## What to take away\n\nThree things you can act on.\n\n**Long conversations cost memory, not just tokens.** Trimming history helps your provider's economics and often your latency.\n\n**A shared, stable prefix is worth engineering for.** Identical opening tokens across requests can be shared or cached; a per-request timestamp at the top of your system prompt destroys that. This is the mechanism behind prompt caching in module four.\n\n**Cache size is a first-class model property.** When choosing a model to self-host, compute its per-token cache cost alongside its parameter count. Two models of the same size can differ fourfold in how many users they will serve."},"check":{"question":"A team adds a line at the very top of their system prompt reading \"Current time: 14:32:07\" and finds their per-request cost rises and latency worsens across the board. Why?","options":[{"text":"The extra tokens push the prompt past a pricing tier boundary.","correct":false,"tempting":"Tiered pricing does exist in some products, and a sudden cost jump is exactly what a tier boundary looks like."},{"text":"Timestamps cause the model to reason about time, which lengthens responses.","correct":false,"tempting":"Content really does influence response length, so attributing extra output to a semantic cue is a fair inference."},{"text":"A changing first token makes every request's prefix unique, so no cached or shared prefix state can be reused and the full prompt must be processed every time.","correct":true},{"text":"Frequent cache invalidation forces the server to reload the model weights.","correct":false,"tempting":"Weight reloading is catastrophic when it happens, which makes it a memorable explanation for sudden slowness."}]},"prev":{"slug":"mixture-of-experts","title":"Mixture of experts: big model, small bill","url":"https://addaly.com/learn/how-llms-work/mixture-of-experts"},"next":{"slug":"next-token-prediction","title":"Next-token prediction, and where the reasoning comes from","url":"https://addaly.com/learn/how-llms-work/next-token-prediction"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The key-value cache, and the engineering built around it', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/kv-cache-and-attention-variants (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The key-value cache, and the engineering built around it","courseTitle":"How a Language Model Actually Works","module":{"index":2,"slug":"inside-the-block","title":"Inside the block"},"answer":null,"keyIdea":"Generation is fast only because every earlier token’s keys and values are cached, and that cache (not the weights) is what usually limits how many users a GPU can serve.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"gigabytes of memory","title":"One 32,000-token conversation, same eight-billion model","series":[{"label":"Cache: multi-head, 32 kv heads","value":16.8},{"label":"Cache: grouped-query, 8 kv heads","value":4.2,"highlight":true},{"label":"Cache: multi-query, 1 kv head","value":0.52},{"label":"Model weights, shared by all users","value":16}],"caption":"Per token the cache costs 2 × layers × kv_heads × head_dim × bytes, so it is proportional to the key-value head count and nothing else. The weights load once and serve everybody; the cache is per user and grows with every token. Grouped-query attention exists for this single reason."},"alt":"Figure: One 32,000-token conversation, same eight-billion model. Bars (gigabytes of memory): Cache: multi-head, 32 kv heads 16.8; Cache: grouped-query, 8 kv heads 4.2; Cache: multi-query, 1 kv head 0.52; Model weights, shared by all users 16. Per token the cache costs 2 × layers × kv_heads × head_dim × bytes, so it is proportional to the key-value head count and nothing else. The weights load once and serve everybody; the cache is per user and grows with every token. Grouped-query attention exists for this single reason."}},"blocks":[{"t":"h2","id":"the-waste-that-had-to-be-removed","runs":[{"text":"The waste that had to be removed"}]},{"t":"p","runs":[{"text":"Generating token 500 means running attention over 499 earlier positions. Their key and value vectors depend only on the tokens themselves, which have not changed. Recomputing them every step would multiply the cost of a 500-token response by a factor of hundreds."}]},{"t":"p","runs":[{"text":"So they are cached. For each layer, for each position, the keys and values are computed once and kept. Each new token computes only its own query, key and value, appends the latter two to the cache, and attends over everything stored."}]},{"t":"p","runs":[{"text":"This is the single most important optimisation in language model serving, and it is why the second token arrives much faster than the first."}]},{"t":"h2","id":"the-cache-is-large","runs":[{"text":"The cache is large"}]},{"t":"p","runs":[{"text":"The arithmetic, which you can do for any model:"}]},{"t":"code","lang":"text","label":"Text","text":"bytes = 2 (key and value)\n      × layers\n      × kv_heads × head_dim\n      × sequence_length\n      × bytes_per_number"},{"t":"p","runs":[{"text":"For our 8B model (32 layers, 8 key-value heads, head dimension 128, bf16) one token costs 2 × 32 × 1,024 × 2 = "},{"text":"131 kilobytes","b":true},{"text":". A 32,000-token conversation therefore costs about "},{"text":"4.2 GB","b":true},{"text":", on top of the 16 GB of weights."}]},{"t":"p","runs":[{"text":"Now the number that matters: the weights are shared across all users, but "},{"text":"every user needs their own cache","b":true},{"text":". Ten users with 32,000-token contexts need 42 GB of cache. That is why the memory arithmetic in module one came out the way it did, and why long-context serving is expensive in a way that surprises people who only counted parameters."}]},{"t":"h2","id":"grouped-query-attention-exists-for-this-reason","runs":[{"text":"Grouped-query attention exists for this reason"}]},{"t":"p","runs":[{"text":"Look again at the formula: the cache is proportional to the number of "},{"text":"key-value","i":true},{"text":" heads."}]},{"t":"ul","items":[[{"text":"Multi-head attention","b":true},{"text":": 32 query heads, 32 key-value heads. Full-size cache."}],[{"text":"Multi-query attention","b":true},{"text":": 32 query heads, 1 key-value head. Cache divided by 32, with a measurable quality cost."}],[{"text":"Grouped-query attention","b":true},{"text":": 32 query heads, 8 key-value heads. Cache divided by 4, with quality close to full multi-head."}]]},{"t":"p","runs":[{"text":"Grouped-query attention is now near-universal in open models, and it was adopted for exactly one reason: it shrinks the cache. When you see "},{"text":"num_key_value_heads","code":true},{"text":" smaller than "},{"text":"num_attention_heads","code":true},{"text":" in a config, you are looking at a memory decision, not a quality one."}]},{"t":"p","runs":[{"text":"Newer designs go further: compressing the cache into a lower-dimensional latent form and reconstructing keys and values on the fly, trading a little arithmetic for a large memory saving."}]},{"t":"h2","id":"sliding-window-and-hybrid-attention","runs":[{"text":"Sliding window and hybrid attention"}]},{"t":"p","runs":[{"text":"Another route: stop attending to everything. A sliding-window layer attends only to the last few thousand tokens, so its cache stops growing. Stack such layers with occasional full-attention layers and information can still propagate a long way: a token influences its window, which influences the next, and so on up the stack."}]},{"t":"p","runs":[{"text":"The trade is real and should be stated plainly: exact recall of a specific detail from 60,000 tokens back is harder for a windowed layer than a full one. Many current models use a hybrid, with a minority of full-attention layers, precisely to keep some exact long-range recall while bounding the cache."}]},{"t":"h2","id":"paged-attention","runs":[{"text":"Paged attention"}]},{"t":"p","runs":[{"text":"Early serving systems allocated a contiguous block of memory per request, sized for the maximum possible length. Most requests never reached it, so most of that memory sat empty: reported waste of 60 to 80 per cent."}]},{"t":"p","runs":[{"text":"Paged attention borrows the idea of virtual memory: store the cache in fixed-size blocks, allocated on demand, tracked by a table. Utilisation goes up sharply, more requests fit, throughput rises several-fold. It also makes prefix sharing natural: two requests with the same system prompt can point at the same physical blocks instead of each holding a copy."}]},{"t":"p","runs":[{"text":"vLLM, which introduced this, is free and open source, and it is the reason a self-hosted deployment can be within reach of a small team."}]},{"t":"h2","id":"what-to-take-away","runs":[{"text":"What to take away"}]},{"t":"p","runs":[{"text":"Three things you can act on."}]},{"t":"p","runs":[{"text":"Long conversations cost memory, not just tokens.","b":true},{"text":" Trimming history helps your provider’s economics and often your latency."}]},{"t":"p","runs":[{"text":"A shared, stable prefix is worth engineering for.","b":true},{"text":" Identical opening tokens across requests can be shared or cached; a per-request timestamp at the top of your system prompt destroys that. This is the mechanism behind prompt caching in module four."}]},{"t":"p","runs":[{"text":"Cache size is a first-class model property.","b":true},{"text":" When choosing a model to self-host, compute its per-token cache cost alongside its parameter count. Two models of the same size can differ fourfold in how many users they will serve."}]}],"check":{"question":[{"text":"A team adds a line at the very top of their system prompt reading “Current time: 14:32:07” and finds their per-request cost rises and latency worsens across the board. Why?"}],"options":[{"text":[{"text":"The extra tokens push the prompt past a pricing tier boundary."}],"correct":false,"tempting":[{"text":"Tiered pricing does exist in some products, and a sudden cost jump is exactly what a tier boundary looks like."}]},{"text":[{"text":"Timestamps cause the model to reason about time, which lengthens responses."}],"correct":false,"tempting":[{"text":"Content really does influence response length, so attributing extra output to a semantic cue is a fair inference."}]},{"text":[{"text":"A changing first token makes every request’s prefix unique, so no cached or shared prefix state can be reused and the full prompt must be processed every time."}],"correct":true,"tempting":null},{"text":[{"text":"Frequent cache invalidation forces the server to reload the model weights."}],"correct":false,"tempting":[{"text":"Weight reloading is catastrophic when it happens, which makes it a memorable explanation for sudden slowness."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"mixture-of-experts","title":"Mixture of experts: big model, small bill","minutes":9},"next":{"slug":"next-token-prediction","title":"Next-token prediction, and where the reasoning comes from","minutes":8}}}