{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"when-it-does-not-fit","title":"When it does not fit: the ordered remedies","url":"https://addaly.com/learn/running-models-yourself/when-it-does-not-fit","position":18,"of":82,"minutes":9,"answer":"Work out whether weights or cache is the oversized term, then walk the ladder from free changes (context length, 8-bit cache, evicting other processes) down to quantisation and offload, which cost real quality.","keyIdea":"Work out whether weights or cache is the oversized term, then walk the ladder from free changes (context length, 8-bit cache, evicting other processes) down to quantisation and offload, which cost real quality."},"body":{"format":"markdown","text":"## Fix the right term\n\nOut of memory is not one problem. Work out which term is too large before changing anything, because the remedies for weights and for cache are different and applying the wrong one costs quality for nothing.\n\nThe sum, once more: weights + KV cache + overhead. Compute all three. If the cache is a quarter of the total, halving it saves an eighth; if it is two thirds, halving it saves a third. That arithmetic decides the order of everything below.\n\n## The ladder, cheapest quality cost first\n\n**1. Reduce the context window.** Free. Zero quality cost if you do not need the length. Most people set 32k out of optimism and send 3,000 tokens. Set it to what you actually use plus headroom. In Ollama that is `num_ctx`; in llama.cpp `-c`; in vLLM `--max-model-len`.\n\n**2. Quantise the KV cache to 8-bit.** Halves the cache term for a difference most tasks cannot detect. Requires flash attention in llama.cpp. This is the best value change available and it is routinely skipped.\n\n**3. Close the other things using the memory.** A browser with hardware acceleration, a desktop compositor, another model still resident. `nvidia-smi` will name them. On Ollama, `OLLAMA_KEEP_ALIVE=0` unloads a model as soon as it goes idle rather than holding it for five minutes.\n\n**4. Drop one quantisation tier.** Q5_K_M to Q4_K_M is a real but small cost and buys about 15% of the weight size. Q6 to Q5 is smaller still. Stop before Q3 unless nothing else works.\n\n**5. Use a smaller model in the same family.** Often better than crushing a bigger one. A 4B at Q5 frequently beats an 8B at Q2 on anything requiring precision, and it is twice as fast.\n\n**6. Use a task-specialised model.** A code-specific 7B in place of a general 14B; an embedding model in place of an LLM for retrieval; a small classifier in place of a general model for labelling. This is a redesign rather than a setting, and it is often the largest win available.\n\n**7. Consider a mixture-of-experts model.** If you are short of GPU memory and rich in system RAM, this changes the problem rather than trading against it.\n\n**8. Partial offload.** Real, and expensive in the way the harmonic mean makes it. Accept it when the CPU share is small, or when the model is MoE and you can offload expert tensors specifically.\n\n**9. Quantise the cache to 4-bit.** Deliberately below offload, because it damages exactly the long-context recall you are usually extending context to get.\n\n**10. Buy memory.** Honest, last, and sometimes correct. System RAM is cheap and helps MoE and CPU inference; VRAM is expensive and is the only thing that helps a dense model on a GPU.\n\n## Fitting into named sizes\n\nSome concrete landing points, all at Q4_K_M with 8k context and roughly 1 GB of overhead:\n\n- **4 GB VRAM.** A 3B fits with room. A 4B is tight. An 8B does not fit and partial offload is painful.\n- **8 GB.** An 8B at 8k fits with about a gigabyte spare. At 32k it does not, unless the cache is quantised.\n- **12 GB.** A 14B fits at 8k. An 8B fits with long context and a quantised cache.\n- **16 GB.** A 14B with generous context, or a 22B tightly.\n- **24 GB.** A 32B at 8k, with little room to spare. A 14B with very long context comfortably.\n- **2 x 24 GB or 64 GB unified.** A 70B at Q4 with moderate context.\n\nRecompute these for your model rather than trusting them; a model with a fat KV cache moves every line.\n\n## The failure modes to recognise\n\n**CUDA out of memory during a long conversation, not at startup.** The cache grew. Cap the context and reserve for it up front rather than discovering the limit at token 6,000.\n\n**Fits, then fails when a second user arrives.** Each concurrent sequence needs its own cache. Either cap concurrency or use an engine with paged cache blocks.\n\n**Fits on Linux, fails on Windows with the same card.** The desktop compositor holds several hundred megabytes of VRAM. Real, and it is why the same configuration behaves differently across operating systems.\n\n**Works, then stops working after an update.** Engines change default context lengths and default cache types between versions. Pin your flags explicitly rather than relying on defaults, and the problem disappears permanently."},"check":{"question":"A 14B at Q4 (about 8.5 GB) with a 32k context (about 4 GB of fp16 cache) will not start on a 12 GB card. Which single change is most likely to make it fit at the lowest quality cost?","options":[{"text":"Quantise the KV cache to 8-bit, which removes about 2 GB and is close to undetectable on ordinary tasks.","correct":true},{"text":"Drop the weights to Q3_K_M, saving roughly 1.6 GB.","correct":false,"tempting":"Attacks the term that is not the surprise and pays a real, measurable quality cost for less saving than the cache offers for free."},{"text":"Enable partial offload of four layers to the CPU, which frees over a gigabyte.","correct":false,"tempting":"Trades a large speed penalty (the harmonic-mean effect) for memory that a cache setting could have released with no speed cost at all."},{"text":"Switch to a mixture-of-experts model of similar total size, so fewer parameters are active per token.","correct":false,"tempting":"Confuses the active-parameter benefit, which is about speed, with the memory constraint, where total parameters are what must be resident."}]},"prev":{"slug":"measuring-your-machine","title":"Measuring what you actually have","url":"https://addaly.com/learn/running-models-yourself/measuring-your-machine"},"next":{"slug":"quantisation-properly","title":"Quantisation, properly","url":"https://addaly.com/learn/running-models-yourself/quantisation-properly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'When it does not fit: the ordered remedies', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/when-it-does-not-fit (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"When it does not fit: the ordered remedies","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"Work out whether weights or cache is the oversized term, then walk the ladder from free changes (context length, 8-bit cache, evicting other processes) down to quantisation and offload, which cost real quality.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"fix-the-right-term","runs":[{"text":"Fix the right term"}]},{"t":"p","runs":[{"text":"Out of memory is not one problem. Work out which term is too large before changing anything, because the remedies for weights and for cache are different and applying the wrong one costs quality for nothing."}]},{"t":"p","runs":[{"text":"The sum, once more: weights + KV cache + overhead. Compute all three. If the cache is a quarter of the total, halving it saves an eighth; if it is two thirds, halving it saves a third. That arithmetic decides the order of everything below."}]},{"t":"h2","id":"the-ladder-cheapest-quality-cost-first","runs":[{"text":"The ladder, cheapest quality cost first"}]},{"t":"p","runs":[{"text":"1. Reduce the context window.","b":true},{"text":" Free. Zero quality cost if you do not need the length. Most people set 32k out of optimism and send 3,000 tokens. Set it to what you actually use plus headroom. In Ollama that is "},{"text":"num_ctx","code":true},{"text":"; in llama.cpp "},{"text":"-c","code":true},{"text":"; in vLLM "},{"text":"--max-model-len","code":true},{"text":"."}]},{"t":"p","runs":[{"text":"2. Quantise the KV cache to 8-bit.","b":true},{"text":" Halves the cache term for a difference most tasks cannot detect. Requires flash attention in llama.cpp. This is the best value change available and it is routinely skipped."}]},{"t":"p","runs":[{"text":"3. Close the other things using the memory.","b":true},{"text":" A browser with hardware acceleration, a desktop compositor, another model still resident. "},{"text":"nvidia-smi","code":true},{"text":" will name them. On Ollama, "},{"text":"OLLAMA_KEEP_ALIVE=0","code":true},{"text":" unloads a model as soon as it goes idle rather than holding it for five minutes."}]},{"t":"p","runs":[{"text":"4. Drop one quantisation tier.","b":true},{"text":" Q5_K_M to Q4_K_M is a real but small cost and buys about 15% of the weight size. Q6 to Q5 is smaller still. Stop before Q3 unless nothing else works."}]},{"t":"p","runs":[{"text":"5. Use a smaller model in the same family.","b":true},{"text":" Often better than crushing a bigger one. A 4B at Q5 frequently beats an 8B at Q2 on anything requiring precision, and it is twice as fast."}]},{"t":"p","runs":[{"text":"6. Use a task-specialised model.","b":true},{"text":" A code-specific 7B in place of a general 14B; an embedding model in place of an LLM for retrieval; a small classifier in place of a general model for labelling. This is a redesign rather than a setting, and it is often the largest win available."}]},{"t":"p","runs":[{"text":"7. Consider a mixture-of-experts model.","b":true},{"text":" If you are short of GPU memory and rich in system RAM, this changes the problem rather than trading against it."}]},{"t":"p","runs":[{"text":"8. Partial offload.","b":true},{"text":" Real, and expensive in the way the harmonic mean makes it. Accept it when the CPU share is small, or when the model is MoE and you can offload expert tensors specifically."}]},{"t":"p","runs":[{"text":"9. Quantise the cache to 4-bit.","b":true},{"text":" Deliberately below offload, because it damages exactly the long-context recall you are usually extending context to get."}]},{"t":"p","runs":[{"text":"10. Buy memory.","b":true},{"text":" Honest, last, and sometimes correct. System RAM is cheap and helps MoE and CPU inference; VRAM is expensive and is the only thing that helps a dense model on a GPU."}]},{"t":"h2","id":"fitting-into-named-sizes","runs":[{"text":"Fitting into named sizes"}]},{"t":"p","runs":[{"text":"Some concrete landing points, all at Q4_K_M with 8k context and roughly 1 GB of overhead:"}]},{"t":"ul","items":[[{"text":"4 GB VRAM.","b":true},{"text":" A 3B fits with room. A 4B is tight. An 8B does not fit and partial offload is painful."}],[{"text":"8 GB.","b":true},{"text":" An 8B at 8k fits with about a gigabyte spare. At 32k it does not, unless the cache is quantised."}],[{"text":"12 GB.","b":true},{"text":" A 14B fits at 8k. An 8B fits with long context and a quantised cache."}],[{"text":"16 GB.","b":true},{"text":" A 14B with generous context, or a 22B tightly."}],[{"text":"24 GB.","b":true},{"text":" A 32B at 8k, with little room to spare. A 14B with very long context comfortably."}],[{"text":"2 x 24 GB or 64 GB unified.","b":true},{"text":" A 70B at Q4 with moderate context."}]]},{"t":"p","runs":[{"text":"Recompute these for your model rather than trusting them; a model with a fat KV cache moves every line."}]},{"t":"h2","id":"the-failure-modes-to-recognise","runs":[{"text":"The failure modes to recognise"}]},{"t":"p","runs":[{"text":"CUDA out of memory during a long conversation, not at startup.","b":true},{"text":" The cache grew. Cap the context and reserve for it up front rather than discovering the limit at token 6,000."}]},{"t":"p","runs":[{"text":"Fits, then fails when a second user arrives.","b":true},{"text":" Each concurrent sequence needs its own cache. Either cap concurrency or use an engine with paged cache blocks."}]},{"t":"p","runs":[{"text":"Fits on Linux, fails on Windows with the same card.","b":true},{"text":" The desktop compositor holds several hundred megabytes of VRAM. Real, and it is why the same configuration behaves differently across operating systems."}]},{"t":"p","runs":[{"text":"Works, then stops working after an update.","b":true},{"text":" Engines change default context lengths and default cache types between versions. Pin your flags explicitly rather than relying on defaults, and the problem disappears permanently."}]}],"check":{"question":[{"text":"A 14B at Q4 (about 8.5 GB) with a 32k context (about 4 GB of fp16 cache) will not start on a 12 GB card. Which single change is most likely to make it fit at the lowest quality cost?"}],"options":[{"text":[{"text":"Quantise the KV cache to 8-bit, which removes about 2 GB and is close to undetectable on ordinary tasks."}],"correct":true,"tempting":null},{"text":[{"text":"Drop the weights to Q3_K_M, saving roughly 1.6 GB."}],"correct":false,"tempting":[{"text":"Attacks the term that is not the surprise and pays a real, measurable quality cost for less saving than the cache offers for free."}]},{"text":[{"text":"Enable partial offload of four layers to the CPU, which frees over a gigabyte."}],"correct":false,"tempting":[{"text":"Trades a large speed penalty (the harmonic-mean effect) for memory that a cache setting could have released with no speed cost at all."}]},{"text":[{"text":"Switch to a mixture-of-experts model of similar total size, so fewer parameters are active per token."}],"correct":false,"tempting":[{"text":"Confuses the active-parameter benefit, which is about speed, with the memory constraint, where total parameters are what must be resident."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"measuring-your-machine","title":"Measuring what you actually have","minutes":8},"next":{"slug":"quantisation-properly","title":"Quantisation, properly","minutes":9}}}