{"course":{"slug":"choosing-your-tools","title":"Choosing and Using the Tools","url":"https://addaly.com/learn/choosing-your-tools","level":"beginner","track":"work"},"lesson":{"slug":"sizing-for-your-machine","title":"Working out what your machine will run","url":"https://addaly.com/learn/choosing-your-tools/sizing-for-your-machine","position":41,"of":76,"minutes":9,"answer":"Budget memory for the weights, the KV cache that grows with your context length, and the operating system, and when a local model is too slow.","keyIdea":"Budget memory for the weights, the KV cache that grows with your context length, and the operating system, and when a local model is too slow, cut context and cache precision before you cut model size."},"body":{"format":"markdown","text":"## The arithmetic, including the part people forget\n\nThe weights are the obvious cost. At 4-bit quantisation a model needs roughly **half a byte per parameter**, so an 8-billion-parameter model is about 4.5 GB on disk and a similar amount in memory.\n\nThe part that surprises people is the **KV cache**: the model's working memory for the conversation so far. It grows linearly with how much text is in the context, and on a long document it can rival the weights.\n\n```\n8B model, 4-bit weights                  ~4.5 GB\nKV cache at   4,000 tokens of context    ~0.5 GB\nKV cache at  32,000 tokens of context    ~4.0 GB\nplus the runtime and the operating system ~2.0 GB\n                                          -------\nshort chats:      ~7 GB       long documents: ~10.5 GB\n```\n\nSo a machine that runs an 8B model beautifully for chat may swap to disk and crawl the moment you paste a fifty-page report in. That is the single most common \"it worked yesterday\" complaint, and the fix is not a bigger model: it is a smaller context setting, or 8-bit cache quantisation, which most runners now offer and which roughly halves that column.\n\n## What your hardware actually gives you\n\n**Ordinary laptop, 8 GB RAM.** 3B to 4B models at 4-bit, short contexts. Useful for drafting, rewriting and summarising pasted text.\n\n> Figure: What an 8B model needs, beyond the weights. Bars (GB of memory): Weights at 4-bit 4.5; KV cache, 4,000 tokens of context 0.5; KV cache, 32,000 tokens of context 4; Runtime and operating system 2. Short chats fit in about seven gigabytes; the same model handed a fifty-page report wants about ten and a half. That is the it-worked-yesterday complaint, and the fix is a smaller context setting or 8-bit cache quantisation rather than a smaller model.\n\n**16 GB.** 7B to 9B comfortably, or a 12B at a squeeze. This is where a local model becomes a real daily tool.\n\n**32 GB.** 24B to 32B at 4-bit. Quality here is good for everyday work.\n\n**Apple Silicon.** Unified memory means the GPU can use most of the system RAM, which is why a Mac with 32 or 64 GB runs models that would need an expensive graphics card on a PC. Roughly 70% of total RAM is available to the model by default.\n\n**A discrete GPU.** What matters is VRAM, not the card's name. 8 GB fits a 7B comfortably; 12 GB fits a 14B; 24 GB fits a 32B, or a 70B at low quantisation with reduced context. Anything that does not fit in VRAM spills into system RAM and slows down by an order of magnitude: not a little, an order of magnitude.\n\n## Measuring rather than guessing\n\nEvery runner reports speed. Look for tokens per second after a real prompt, not a one-word one.\n\n```bash\nollama run qwen3:8b --verbose        # prints eval rate at the end\n```\n\nThe interpretation:\n\n- **Under 5 tokens/sec**: painful to read along with. Fine for a job you leave running.\n- **10 to 20**: comfortable conversation, roughly reading pace.\n- **30 and above**: feels instant.\n\nWatch the second number too, **prompt evaluation speed**. That is how long it takes to read your input before answering. On a CPU it can be slow enough that a fifty-page document takes a minute before the first word appears, even though generation afterwards is fine.\n\n## When it is too slow\n\nIn order of what to try, cheapest first:\n\n1. **Reduce the context setting.** Most runners default to something large. If your work is short questions, 4,000 tokens is plenty and it frees a lot of memory.\n2. **Quantise the KV cache to 8-bit.** A setting in llama.cpp, Ollama and LM Studio. Roughly halves cache memory for a negligible quality cost.\n3. **Drop one quantisation level**, Q5 to Q4. Small quality cost, real speed gain.\n4. **Use a smaller model.** A 4B that runs at 25 tokens/sec is more useful than an 8B that runs at 3.\n5. **Offload some layers to the GPU** if you have one. Even a modest card holding half the layers helps considerably.\n6. **Close the browser.** Not a joke: browsers routinely hold several gigabytes, and reclaiming them is often the difference between fitting and swapping.\n\n## The honest ceiling\n\nThere is a size above which your machine cannot go, and no setting changes it. If a model does not fit, it will either refuse to load or run at a speed that makes it unusable. That is not a failure of configuration.\n\nAt that point the choices are a smaller model, a hosted open model at a few cents an hour, or accepting that this particular task is not one your hardware does. All three are reasonable. Buying a graphics card to save a subscription is arithmetic worth doing carefully before, not after: the card costs more than several years of the subscription, and the models it fits will keep changing."},"check":{"question":"A 4-bit 8B model chats smoothly on a 16 GB laptop, then crawls when a fifty-page report is pasted in. What changed?","options":[{"text":"The model exceeded its trained context length and fell back to a slower sliding-window attention mode.","correct":false,"tempting":"Invokes a fallback mechanism where the observed symptom is memory pressure, and exceeding the window normally truncates rather than slows."},{"text":"The quantisation degrades on long inputs, so the model needs to reload weights at higher precision.","correct":false,"tempting":"Invents dynamic precision; quantised weights are fixed on disk and do not change with input length."},{"text":"Long documents are processed twice, once for retrieval and once for generation, doubling the work.","correct":false,"tempting":"Describes a retrieval pipeline that a plain local runner does not perform on pasted text."},{"text":"The KV cache grew with the context, pushing past available memory and forcing the machine to swap.","correct":true}]},"prev":{"slug":"on-your-laptop","title":"Running a model on your own machine","url":"https://addaly.com/learn/choosing-your-tools/on-your-laptop"},"next":{"slug":"reading-a-model-card","title":"Reading a model card before you download","url":"https://addaly.com/learn/choosing-your-tools/reading-a-model-card"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Working out what your machine will run', in Choosing and Using the Tools. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/choosing-your-tools/sizing-for-your-machine (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Working out what your machine will run","courseTitle":"Choosing and Using the Tools","module":{"index":5,"slug":"open-models-and-your-own-hardware","title":"Open models and your own hardware"},"answer":null,"keyIdea":"Budget memory for the weights, the KV cache that grows with your context length, and the operating system, and when a local model is too slow, cut context and cache precision before you cut model size.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"GB of memory","title":"What an 8B model needs, beyond the weights","series":[{"label":"Weights at 4-bit","value":4.5},{"label":"KV cache, 4,000 tokens of context","value":0.5},{"label":"KV cache, 32,000 tokens of context","value":4,"highlight":true},{"label":"Runtime and operating system","value":2}],"caption":"Short chats fit in about seven gigabytes; the same model handed a fifty-page report wants about ten and a half. That is the it-worked-yesterday complaint, and the fix is a smaller context setting or 8-bit cache quantisation rather than a smaller model."},"alt":"Figure: What an 8B model needs, beyond the weights. Bars (GB of memory): Weights at 4-bit 4.5; KV cache, 4,000 tokens of context 0.5; KV cache, 32,000 tokens of context 4; Runtime and operating system 2. Short chats fit in about seven gigabytes; the same model handed a fifty-page report wants about ten and a half. That is the it-worked-yesterday complaint, and the fix is a smaller context setting or 8-bit cache quantisation rather than a smaller model."}},"blocks":[{"t":"h2","id":"the-arithmetic-including-the-part-people-forget","runs":[{"text":"The arithmetic, including the part people forget"}]},{"t":"p","runs":[{"text":"The weights are the obvious cost. At 4-bit quantisation a model needs roughly "},{"text":"half a byte per parameter","b":true},{"text":", so an 8-billion-parameter model is about 4.5 GB on disk and a similar amount in memory."}]},{"t":"p","runs":[{"text":"The part that surprises people is the "},{"text":"KV cache","b":true},{"text":": the model’s working memory for the conversation so far. It grows linearly with how much text is in the context, and on a long document it can rival the weights."}]},{"t":"code","lang":"","label":"Text","text":"8B model, 4-bit weights                  ~4.5 GB\nKV cache at   4,000 tokens of context    ~0.5 GB\nKV cache at  32,000 tokens of context    ~4.0 GB\nplus the runtime and the operating system ~2.0 GB\n                                          -------\nshort chats:      ~7 GB       long documents: ~10.5 GB"},{"t":"p","runs":[{"text":"So a machine that runs an 8B model beautifully for chat may swap to disk and crawl the moment you paste a fifty-page report in. That is the single most common “it worked yesterday” complaint, and the fix is not a bigger model: it is a smaller context setting, or 8-bit cache quantisation, which most runners now offer and which roughly halves that column."}]},{"t":"h2","id":"what-your-hardware-actually-gives-you","runs":[{"text":"What your hardware actually gives you"}]},{"t":"p","runs":[{"text":"Ordinary laptop, 8 GB RAM.","b":true},{"text":" 3B to 4B models at 4-bit, short contexts. Useful for drafting, rewriting and summarising pasted text."}]},{"t":"p","runs":[{"text":"16 GB.","b":true},{"text":" 7B to 9B comfortably, or a 12B at a squeeze. This is where a local model becomes a real daily tool."}]},{"t":"p","runs":[{"text":"32 GB.","b":true},{"text":" 24B to 32B at 4-bit. Quality here is good for everyday work."}]},{"t":"p","runs":[{"text":"Apple Silicon.","b":true},{"text":" Unified memory means the GPU can use most of the system RAM, which is why a Mac with 32 or 64 GB runs models that would need an expensive graphics card on a PC. Roughly 70% of total RAM is available to the model by default."}]},{"t":"p","runs":[{"text":"A discrete GPU.","b":true},{"text":" What matters is VRAM, not the card’s name. 8 GB fits a 7B comfortably; 12 GB fits a 14B; 24 GB fits a 32B, or a 70B at low quantisation with reduced context. Anything that does not fit in VRAM spills into system RAM and slows down by an order of magnitude: not a little, an order of magnitude."}]},{"t":"h2","id":"measuring-rather-than-guessing","runs":[{"text":"Measuring rather than guessing"}]},{"t":"p","runs":[{"text":"Every runner reports speed. Look for tokens per second after a real prompt, not a one-word one."}]},{"t":"code","lang":"bash","label":"Bash","text":"ollama run qwen3:8b --verbose        # prints eval rate at the end"},{"t":"p","runs":[{"text":"The interpretation:"}]},{"t":"ul","items":[[{"text":"Under 5 tokens/sec","b":true},{"text":": painful to read along with. Fine for a job you leave running."}],[{"text":"10 to 20","b":true},{"text":": comfortable conversation, roughly reading pace."}],[{"text":"30 and above","b":true},{"text":": feels instant."}]]},{"t":"p","runs":[{"text":"Watch the second number too, "},{"text":"prompt evaluation speed","b":true},{"text":". That is how long it takes to read your input before answering. On a CPU it can be slow enough that a fifty-page document takes a minute before the first word appears, even though generation afterwards is fine."}]},{"t":"h2","id":"when-it-is-too-slow","runs":[{"text":"When it is too slow"}]},{"t":"p","runs":[{"text":"In order of what to try, cheapest first:"}]},{"t":"ol","start":1,"items":[[{"text":"Reduce the context setting.","b":true},{"text":" Most runners default to something large. If your work is short questions, 4,000 tokens is plenty and it frees a lot of memory."}],[{"text":"Quantise the KV cache to 8-bit.","b":true},{"text":" A setting in llama.cpp, Ollama and LM Studio. Roughly halves cache memory for a negligible quality cost."}],[{"text":"Drop one quantisation level","b":true},{"text":", Q5 to Q4. Small quality cost, real speed gain."}],[{"text":"Use a smaller model.","b":true},{"text":" A 4B that runs at 25 tokens/sec is more useful than an 8B that runs at 3."}],[{"text":"Offload some layers to the GPU","b":true},{"text":" if you have one. Even a modest card holding half the layers helps considerably."}],[{"text":"Close the browser.","b":true},{"text":" Not a joke: browsers routinely hold several gigabytes, and reclaiming them is often the difference between fitting and swapping."}]]},{"t":"h2","id":"the-honest-ceiling","runs":[{"text":"The honest ceiling"}]},{"t":"p","runs":[{"text":"There is a size above which your machine cannot go, and no setting changes it. If a model does not fit, it will either refuse to load or run at a speed that makes it unusable. That is not a failure of configuration."}]},{"t":"p","runs":[{"text":"At that point the choices are a smaller model, a hosted open model at a few cents an hour, or accepting that this particular task is not one your hardware does. All three are reasonable. Buying a graphics card to save a subscription is arithmetic worth doing carefully before, not after: the card costs more than several years of the subscription, and the models it fits will keep changing."}]}],"check":{"question":[{"text":"A 4-bit 8B model chats smoothly on a 16 GB laptop, then crawls when a fifty-page report is pasted in. What changed?"}],"options":[{"text":[{"text":"The model exceeded its trained context length and fell back to a slower sliding-window attention mode."}],"correct":false,"tempting":[{"text":"Invokes a fallback mechanism where the observed symptom is memory pressure, and exceeding the window normally truncates rather than slows."}]},{"text":[{"text":"The quantisation degrades on long inputs, so the model needs to reload weights at higher precision."}],"correct":false,"tempting":[{"text":"Invents dynamic precision; quantised weights are fixed on disk and do not change with input length."}]},{"text":[{"text":"Long documents are processed twice, once for retrieval and once for generation, doubling the work."}],"correct":false,"tempting":[{"text":"Describes a retrieval pipeline that a plain local runner does not perform on pasted text."}]},{"text":[{"text":"The KV cache grew with the context, pushing past available memory and forcing the machine to swap."}],"correct":true,"tempting":null}],"order":[1,2,3,0],"why":null},"prev":{"slug":"on-your-laptop","title":"Running a model on your own machine","minutes":9},"next":{"slug":"reading-a-model-card","title":"Reading a model card before you download","minutes":8}}}