{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"does-it-fit","title":"Does it fit: the arithmetic","url":"https://addaly.com/learn/running-models-yourself/does-it-fit","position":10,"of":82,"minutes":9,"answer":"Weights are fixed; the KV cache grows with context and layers, and that is what usually breaks the budget.","keyIdea":"Weights are fixed; the KV cache grows with context and layers, and that is what usually breaks the budget."},"body":{"format":"markdown","text":"Stop looking up tables of \"models that fit in 8GB\". Learn the two sums and you can answer for any model, any quant, any context length.\n\nTotal memory = weights + KV cache + overhead.\n\n## Weights\n\n```\nbytes = parameters x bytes_per_weight\n```\n\nBytes per weight comes from the format. Useful figures, in bits per weight (divide by 8):\n\n| format | bits/weight |\n|---|---|\n| fp16 / bf16 | 16 |\n| Q8_0 | 8.5 |\n| Q6_K | 6.6 |\n| Q5_K_M | 5.7 |\n| Q4_K_M | 4.8 |\n| Q3_K_M | 3.9 |\n| IQ2_XS | 2.4 |\n\nThe 4-bit formats are above 4 bits because each block of weights carries a scale. Lesson 3 explains why.\n\nSo Llama 3.1 8B at Q4_K_M: 8.03e9 x 4.8 / 8 = 4.8 GB. The published file is 4.9 GB. Close enough to plan with.\n\nA 70B at Q4_K_M: 70e9 x 0.6 = 42 GB. Two 24 GB cards, or a 64 GB Mac, or it does not happen.\n\n> Figure: An 8 GB card, an 8B model, and where the gigabytes go. Layers, top to bottom: Weights: 4.9 GB (8.03 billion parameters at 4.8 bits each, and fixed whatever you change next); KV cache at 8k: 1.0 GB (128 KiB a token: 2 for K and V, by 32 layers, by 8 KV heads, by 128, by 2 bytes); Overhead: 0.8 GB (CUDA context, compute buffers, activations, the framework. Add fifteen per cent); Total: 6.7 GB (Fits, with headroom. The same model at 32k context comes to 9.7 GB and does not). Weights are the number people quote and the cache is the term that breaks the budget. Raise the window to 32k and the cache quadruples to 4.0 GB while the weights do not move at all, which is why the first remedy is almost never a smaller quant.\n\n## KV cache\n\nEvery token you have already processed leaves behind a key and a value vector in every layer. This is what \"context costs memory\" means.\n\n```\nbytes_per_token = 2 x n_layers x n_kv_heads x head_dim x bytes_per_element\n```\n\nThe leading 2 is K and V. All four numbers are in the model's `config.json` on Hugging Face: look for `num_hidden_layers`, `num_key_value_heads`, and `hidden_size / num_attention_heads` for head dim.\n\nLlama 3.1 8B: 32 layers, 8 KV heads, head dim 128, fp16 cache.\n\n```\n2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token\n```\n\nAt 8,192 tokens: 1.0 GiB. At 32,768: 4.0 GiB. At 128k: 16 GiB: four times the weights.\n\nNow contrast Llama 2 13B, which predates grouped-query attention and has 40 KV heads across 40 layers:\n\n```\n2 x 40 x 40 x 128 x 2 = 819,200 bytes = 800 KiB per token\n```\n\nThat is 3.1 GiB at only 4k context. Grouped-query attention, which shares one KV head across several query heads, is the reason long context became affordable. Two models of similar size can differ sixfold here.\n\n## Overhead\n\nCUDA context, compute buffers, activations, the framework itself. Budget 0.6 to 1.5 GB, or add 15%. It is not nothing and it is what turns \"it should just fit\" into an out-of-memory error.\n\n## A worked decision\n\nAn 8 GB card. Llama 3.1 8B Q4_K_M, 8k context:\n\n- weights 4.9 GB\n- KV 1.0 GB\n- overhead 0.8 GB\n- **total 6.7 GB**: fits, with headroom.\n\nSame model at 32k context: KV becomes 4.0 GB, total 9.7 GB. Does not fit. Your options, in order of what they cost you:\n\n1. **Quantise the KV cache to 8-bit.** Halves it to 2.0 GB, total 7.7 GB. Fits. Quality cost is small at q8_0; q4_0 cache is noticeably worse for long recall.\n2. **Drop to 16k context.** Total 7.7 GB too, with no quality cost at all, if you do not need 32k.\n3. **Drop weights to Q3_K_M** (3.9 GB). Saves 1 GB and costs more quality than either of the above. Last resort.\n\nIn llama.cpp those first two are:\n\n```bash\nllama-server -m model.gguf -c 32768 --flash-attn \\\n  --cache-type-k q8_0 --cache-type-v q8_0\n```\n\n## Partial offload\n\nIf it does not fit, llama.cpp will run some layers on the GPU and the rest on the CPU (`-ngl 24` for 24 layers). This works, and it is slow roughly in proportion to how much stayed on the CPU. Ten percent of layers on CPU is not a ten percent slowdown; it is often a fifty percent one, because every token waits for those layers.\n\nDo the sum before you download 40 GB."},"check":{"question":"An 8B model at Q4 uses 6.7 GB of an 8 GB card at 8k context. Raising context to 32k makes it fail to allocate. Which fix most directly addresses what grew, while keeping the model and the 32k context?","options":[{"text":"Store the KV cache in 8-bit instead of 16-bit, roughly halving the 4 GB cache.","correct":true},{"text":"Enable FlashAttention, which removes the need to materialise the KV cache.","correct":false,"tempting":"FlashAttention avoids materialising the large attention matrix, which is activation memory, not the KV cache."},{"text":"Drop the weights from Q4_K_M to Q3_K_M to free room for the longer context.","correct":false,"tempting":"Shrinks the part that did not change; the roughly 3 GB increase was entirely cache, and 1 GB of weight savings does not cover it."},{"text":"Shorten the system prompt so fewer tokens sit in the cache.","correct":false,"tempting":"Assumes the cache is sized by tokens currently in use, but servers reserve the full context window at load time."}]},"prev":{"slug":"local-is-not-automatically-private","title":"Local is not automatically private","url":"https://addaly.com/learn/running-models-yourself/local-is-not-automatically-private"},"next":{"slug":"small-hardware-reality","title":"What actually runs on 8GB","url":"https://addaly.com/learn/running-models-yourself/small-hardware-reality"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Does it fit: the arithmetic', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/does-it-fit (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Does it fit: the arithmetic","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"Weights are fixed; the KV cache grows with context and layers, and that is what usually breaks the budget.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 2.1","figure":{"type":"stack","title":"An 8 GB card, an 8B model, and where the gigabytes go","layers":[{"label":"Weights: 4.9 GB","detail":"8.03 billion parameters at 4.8 bits each, and fixed whatever you change next"},{"label":"KV cache at 8k: 1.0 GB","detail":"128 KiB a token: 2 for K and V, by 32 layers, by 8 KV heads, by 128, by 2 bytes"},{"label":"Overhead: 0.8 GB","detail":"CUDA context, compute buffers, activations, the framework. Add fifteen per cent"},{"label":"Total: 6.7 GB","detail":"Fits, with headroom. The same model at 32k context comes to 9.7 GB and does not"}],"caption":"Weights are the number people quote and the cache is the term that breaks the budget. Raise the window to 32k and the cache quadruples to 4.0 GB while the weights do not move at all, which is why the first remedy is almost never a smaller quant."},"alt":"Figure: An 8 GB card, an 8B model, and where the gigabytes go. Layers, top to bottom: Weights: 4.9 GB (8.03 billion parameters at 4.8 bits each, and fixed whatever you change next); KV cache at 8k: 1.0 GB (128 KiB a token: 2 for K and V, by 32 layers, by 8 KV heads, by 128, by 2 bytes); Overhead: 0.8 GB (CUDA context, compute buffers, activations, the framework. Add fifteen per cent); Total: 6.7 GB (Fits, with headroom. The same model at 32k context comes to 9.7 GB and does not). Weights are the number people quote and the cache is the term that breaks the budget. Raise the window to 32k and the cache quadruples to 4.0 GB while the weights do not move at all, which is why the first remedy is almost never a smaller quant."}},"blocks":[{"t":"p","runs":[{"text":"Stop looking up tables of “models that fit in 8GB”. Learn the two sums and you can answer for any model, any quant, any context length."}],"lede":true},{"t":"p","runs":[{"text":"Total memory = weights + KV cache + overhead."}]},{"t":"h2","id":"weights","runs":[{"text":"Weights"}]},{"t":"code","lang":"","label":"Text","text":"bytes = parameters x bytes_per_weight"},{"t":"p","runs":[{"text":"Bytes per weight comes from the format. Useful figures, in bits per weight (divide by 8):"}]},{"t":"table","head":[[{"text":"format"}],[{"text":"bits/weight"}]],"align":["left","left"],"rows":[[[{"text":"fp16 / bf16"}],[{"text":"16"}]],[[{"text":"Q8_0"}],[{"text":"8.5"}]],[[{"text":"Q6_K"}],[{"text":"6.6"}]],[[{"text":"Q5_K_M"}],[{"text":"5.7"}]],[[{"text":"Q4_K_M"}],[{"text":"4.8"}]],[[{"text":"Q3_K_M"}],[{"text":"3.9"}]],[[{"text":"IQ2_XS"}],[{"text":"2.4"}]]]},{"t":"p","runs":[{"text":"The 4-bit formats are above 4 bits because each block of weights carries a scale. Lesson 3 explains why."}]},{"t":"p","runs":[{"text":"So Llama 3.1 8B at Q4_K_M: 8.03e9 x 4.8 / 8 = 4.8 GB. The published file is 4.9 GB. Close enough to plan with."}]},{"t":"p","runs":[{"text":"A 70B at Q4_K_M: 70e9 x 0.6 = 42 GB. Two 24 GB cards, or a 64 GB Mac, or it does not happen."}]},{"t":"h2","id":"kv-cache","runs":[{"text":"KV cache"}]},{"t":"p","runs":[{"text":"Every token you have already processed leaves behind a key and a value vector in every layer. This is what “context costs memory” means."}]},{"t":"code","lang":"","label":"Text","text":"bytes_per_token = 2 x n_layers x n_kv_heads x head_dim x bytes_per_element"},{"t":"p","runs":[{"text":"The leading 2 is K and V. All four numbers are in the model’s "},{"text":"config.json","code":true},{"text":" on Hugging Face: look for "},{"text":"num_hidden_layers","code":true},{"text":", "},{"text":"num_key_value_heads","code":true},{"text":", and "},{"text":"hidden_size / num_attention_heads","code":true},{"text":" for head dim."}]},{"t":"p","runs":[{"text":"Llama 3.1 8B: 32 layers, 8 KV heads, head dim 128, fp16 cache."}]},{"t":"code","lang":"","label":"Text","text":"2 x 32 x 8 x 128 x 2 = 131,072 bytes = 128 KiB per token"},{"t":"p","runs":[{"text":"At 8,192 tokens: 1.0 GiB. At 32,768: 4.0 GiB. At 128k: 16 GiB: four times the weights."}]},{"t":"p","runs":[{"text":"Now contrast Llama 2 13B, which predates grouped-query attention and has 40 KV heads across 40 layers:"}]},{"t":"code","lang":"","label":"Text","text":"2 x 40 x 40 x 128 x 2 = 819,200 bytes = 800 KiB per token"},{"t":"p","runs":[{"text":"That is 3.1 GiB at only 4k context. Grouped-query attention, which shares one KV head across several query heads, is the reason long context became affordable. Two models of similar size can differ sixfold here."}]},{"t":"h2","id":"overhead","runs":[{"text":"Overhead"}]},{"t":"p","runs":[{"text":"CUDA context, compute buffers, activations, the framework itself. Budget 0.6 to 1.5 GB, or add 15%. It is not nothing and it is what turns “it should just fit” into an out-of-memory error."}]},{"t":"h2","id":"a-worked-decision","runs":[{"text":"A worked decision"}]},{"t":"p","runs":[{"text":"An 8 GB card. Llama 3.1 8B Q4_K_M, 8k context:"}]},{"t":"ul","items":[[{"text":"weights 4.9 GB"}],[{"text":"KV 1.0 GB"}],[{"text":"overhead 0.8 GB"}],[{"text":"total 6.7 GB","b":true},{"text":": fits, with headroom."}]]},{"t":"p","runs":[{"text":"Same model at 32k context: KV becomes 4.0 GB, total 9.7 GB. Does not fit. Your options, in order of what they cost you:"}]},{"t":"ol","start":1,"items":[[{"text":"Quantise the KV cache to 8-bit.","b":true},{"text":" Halves it to 2.0 GB, total 7.7 GB. Fits. Quality cost is small at q8_0; q4_0 cache is noticeably worse for long recall."}],[{"text":"Drop to 16k context.","b":true},{"text":" Total 7.7 GB too, with no quality cost at all, if you do not need 32k."}],[{"text":"Drop weights to Q3_K_M","b":true},{"text":" (3.9 GB). Saves 1 GB and costs more quality than either of the above. Last resort."}]]},{"t":"p","runs":[{"text":"In llama.cpp those first two are:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -c 32768 --flash-attn \\\n  --cache-type-k q8_0 --cache-type-v q8_0"},{"t":"h2","id":"partial-offload","runs":[{"text":"Partial offload"}]},{"t":"p","runs":[{"text":"If it does not fit, llama.cpp will run some layers on the GPU and the rest on the CPU ("},{"text":"-ngl 24","code":true},{"text":" for 24 layers). This works, and it is slow roughly in proportion to how much stayed on the CPU. Ten percent of layers on CPU is not a ten percent slowdown; it is often a fifty percent one, because every token waits for those layers."}]},{"t":"p","runs":[{"text":"Do the sum before you download 40 GB."}]}],"check":{"question":[{"text":"An 8B model at Q4 uses 6.7 GB of an 8 GB card at 8k context. Raising context to 32k makes it fail to allocate. Which fix most directly addresses what grew, while keeping the model and the 32k context?"}],"options":[{"text":[{"text":"Store the KV cache in 8-bit instead of 16-bit, roughly halving the 4 GB cache."}],"correct":true,"tempting":null},{"text":[{"text":"Enable FlashAttention, which removes the need to materialise the KV cache."}],"correct":false,"tempting":[{"text":"FlashAttention avoids materialising the large attention matrix, which is activation memory, not the KV cache."}]},{"text":[{"text":"Drop the weights from Q4_K_M to Q3_K_M to free room for the longer context."}],"correct":false,"tempting":[{"text":"Shrinks the part that did not change; the roughly 3 GB increase was entirely cache, and 1 GB of weight savings does not cover it."}]},{"text":[{"text":"Shorten the system prompt so fewer tokens sit in the cache."}],"correct":false,"tempting":[{"text":"Assumes the cache is sized by tokens currently in use, but servers reserve the full context window at load time."}]}],"order":[2,0,3,1],"why":null},"prev":{"slug":"local-is-not-automatically-private","title":"Local is not automatically private","minutes":8},"next":{"slug":"small-hardware-reality","title":"What actually runs on 8GB","minutes":8}}}