{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"small-hardware-reality","title":"What actually runs on 8GB","url":"https://addaly.com/learn/running-models-yourself/small-hardware-reality","position":11,"of":82,"minutes":8,"answer":"Generation speed is memory bandwidth divided by model size; cores and capacity do not change that ceiling.","keyIdea":"Generation speed is memory bandwidth divided by model size; cores and capacity do not change that ceiling."},"body":{"format":"markdown","text":"Most writing about local models assumes a 4090. Most readers do not have one. Here is what a mid-range laptop, a Mac, and a free Colab session do, with the arithmetic that predicts it.\n\n## The one equation for generation speed\n\nGenerating one token at a time reads every weight in the model, once. So decoding is limited by memory bandwidth, not by compute:\n\n```\ntokens/sec (ceiling) = effective_bandwidth / model_size_in_bytes\n```\n\nEffective bandwidth is roughly 60 to 75% of the theoretical figure. A typical dual-channel DDR4-3200 laptop has 51.2 GB/s theoretical, so about 30 GB/s real.\n\nOn that machine:\n\n| model | size at Q4 | predicted | typical measured |\n|---|---|---|---|\n| 1.7B | 1.1 GB | 27 tok/s | 20 to 25 |\n| 3B | 2.0 GB | 15 tok/s | 11 to 15 |\n| 4B | 2.5 GB | 12 tok/s | 9 to 12 |\n| 8B | 4.9 GB | 6 tok/s | 4 to 7 |\n\nThe prediction is close enough to be useful. Do it before downloading.\n\n**For 8 GB of RAM with no discrete GPU, the honest recommendation is a 3B or 4B model at Q4_K_M with 4k to 8k context.** That is useful: summarising, rewriting, classification, extraction, tagging, simple questions. It is not useful for hard code or long reasoning, and pretending otherwise wastes your evening.\n\n## The part nobody warns you about\n\nProcessing the prompt is a different operation. It handles all tokens at once, so it is compute-bound, and CPUs are bad at it. An 8B model on a laptop CPU might prefill at 20 to 60 tokens/second. A 2,000-token prompt is therefore 30 to 100 seconds of nothing happening before the first word appears.\n\nThat delay is what makes CPU inference feel broken, and it is why RAG systems that stuff 6,000 tokens of retrieved context are miserable on a laptop. Keep prompts short. Reuse cached prefixes where the server supports it.\n\nIntegrated graphics help here more than you would expect and less than you would hope. An iGPU shares the same RAM, so it cannot raise the decoding ceiling: same bandwidth. But it has far more parallel compute than the CPU cores, so prefill often gets 2 to 3x faster. llama.cpp's Vulkan backend works on Intel and AMD integrated graphics:\n\n```bash\nllama-server -m model.gguf -ngl 99 -c 4096   # a Vulkan build offloads to the iGPU\n```\n\n## Macs\n\nApple Silicon is quietly the best value for this, because the CPU and GPU share one pool of high-bandwidth memory. Approximate figures: base M1/M2/M3 around 68 to 100 GB/s, M4 around 120, the Pro chips 200 to 270, the Max chips 400 to 550, Ultra around 800.\n\nA 16 GB M-series machine runs an 8B at Q4 around 15 to 25 tok/s with good prefill. A 32 GB one runs 14B comfortably and 32B at Q4 slowly but usably.\n\nmacOS caps how much RAM the GPU may hold: roughly 65 to 75%. On a 32 GB machine you can raise it:\n\n```bash\nsudo sysctl iogpu.wired_limit_mb=24576\n```\n\nLeave the operating system several gigabytes, and note it resets on reboot.\n\n## Colab's free tier\n\nA T4 with 16 GB of VRAM and about 320 GB/s. That runs an 8B in fp16 at 30 to 40 tok/s, or a 14B at Q4 comfortably. Sessions disconnect, nothing persists, and the terms do not permit running a public service through a tunnel.\n\nUse it for exactly one thing: evaluating whether a model is good enough for your task before you spend money on hardware. That is the test from lesson 1, and Colab is a fine place to run it.\n\n## If you are going to buy\n\nThe cheapest serious entry is a used NVIDIA card with 12 GB and high bandwidth: around 360 GB/s gets you 35 to 45 tok/s on an 8B at Q4, and prefill stops being the problem. Older datacentre cards with lots of memory are cheap for a reason: their bandwidth or their fp16 performance is poor, and driver support is a project in itself."},"check":{"question":"A 3B model at Q4 generates about 14 tok/s on your laptop. You want to roughly double that. Which change is most likely to actually do it?","options":[{"text":"Move to a machine with substantially faster memory, such as dual-channel DDR5-6000 instead of DDR4-3200.","correct":true},{"text":"Switch to a CPU with twice as many cores.","correct":false,"tempting":"Assumes decoding is compute-bound, when at batch size one it is waiting on memory, not on arithmetic."},{"text":"Increase RAM from 8 GB to 16 GB.","correct":false,"tempting":"Confuses capacity, which decides whether the model fits, with bandwidth, which decides how fast weights stream."},{"text":"Offload the model to the integrated GPU, since a GPU is faster than a CPU.","correct":false,"tempting":"An iGPU shares the same memory bus, so it improves prompt processing but leaves the generation ceiling untouched."}]},"prev":{"slug":"does-it-fit","title":"Does it fit: the arithmetic","url":"https://addaly.com/learn/running-models-yourself/does-it-fit"},"next":{"slug":"the-kv-cache-in-depth","title":"The KV cache, in depth","url":"https://addaly.com/learn/running-models-yourself/the-kv-cache-in-depth"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'What actually runs on 8GB', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/small-hardware-reality (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"What actually runs on 8GB","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"Generation speed is memory bandwidth divided by model size; cores and capacity do not change that ceiling.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"p","runs":[{"text":"Most writing about local models assumes a 4090. Most readers do not have one. Here is what a mid-range laptop, a Mac, and a free Colab session do, with the arithmetic that predicts it."}],"lede":true},{"t":"h2","id":"the-one-equation-for-generation-speed","runs":[{"text":"The one equation for generation speed"}]},{"t":"p","runs":[{"text":"Generating one token at a time reads every weight in the model, once. So decoding is limited by memory bandwidth, not by compute:"}]},{"t":"code","lang":"","label":"Text","text":"tokens/sec (ceiling) = effective_bandwidth / model_size_in_bytes"},{"t":"p","runs":[{"text":"Effective bandwidth is roughly 60 to 75% of the theoretical figure. A typical dual-channel DDR4-3200 laptop has 51.2 GB/s theoretical, so about 30 GB/s real."}]},{"t":"p","runs":[{"text":"On that machine:"}]},{"t":"table","head":[[{"text":"model"}],[{"text":"size at Q4"}],[{"text":"predicted"}],[{"text":"typical measured"}]],"align":["left","left","left","left"],"rows":[[[{"text":"1.7B"}],[{"text":"1.1 GB"}],[{"text":"27 tok/s"}],[{"text":"20 to 25"}]],[[{"text":"3B"}],[{"text":"2.0 GB"}],[{"text":"15 tok/s"}],[{"text":"11 to 15"}]],[[{"text":"4B"}],[{"text":"2.5 GB"}],[{"text":"12 tok/s"}],[{"text":"9 to 12"}]],[[{"text":"8B"}],[{"text":"4.9 GB"}],[{"text":"6 tok/s"}],[{"text":"4 to 7"}]]]},{"t":"p","runs":[{"text":"The prediction is close enough to be useful. Do it before downloading."}]},{"t":"p","runs":[{"text":"For 8 GB of RAM with no discrete GPU, the honest recommendation is a 3B or 4B model at Q4_K_M with 4k to 8k context.","b":true},{"text":" That is useful: summarising, rewriting, classification, extraction, tagging, simple questions. It is not useful for hard code or long reasoning, and pretending otherwise wastes your evening."}]},{"t":"h2","id":"the-part-nobody-warns-you-about","runs":[{"text":"The part nobody warns you about"}]},{"t":"p","runs":[{"text":"Processing the prompt is a different operation. It handles all tokens at once, so it is compute-bound, and CPUs are bad at it. An 8B model on a laptop CPU might prefill at 20 to 60 tokens/second. A 2,000-token prompt is therefore 30 to 100 seconds of nothing happening before the first word appears."}]},{"t":"p","runs":[{"text":"That delay is what makes CPU inference feel broken, and it is why RAG systems that stuff 6,000 tokens of retrieved context are miserable on a laptop. Keep prompts short. Reuse cached prefixes where the server supports it."}]},{"t":"p","runs":[{"text":"Integrated graphics help here more than you would expect and less than you would hope. An iGPU shares the same RAM, so it cannot raise the decoding ceiling: same bandwidth. But it has far more parallel compute than the CPU cores, so prefill often gets 2 to 3x faster. llama.cpp’s Vulkan backend works on Intel and AMD integrated graphics:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -ngl 99 -c 4096   # a Vulkan build offloads to the iGPU"},{"t":"h2","id":"macs","runs":[{"text":"Macs"}]},{"t":"p","runs":[{"text":"Apple Silicon is quietly the best value for this, because the CPU and GPU share one pool of high-bandwidth memory. Approximate figures: base M1/M2/M3 around 68 to 100 GB/s, M4 around 120, the Pro chips 200 to 270, the Max chips 400 to 550, Ultra around 800."}]},{"t":"p","runs":[{"text":"A 16 GB M-series machine runs an 8B at Q4 around 15 to 25 tok/s with good prefill. A 32 GB one runs 14B comfortably and 32B at Q4 slowly but usably."}]},{"t":"p","runs":[{"text":"macOS caps how much RAM the GPU may hold: roughly 65 to 75%. On a 32 GB machine you can raise it:"}]},{"t":"code","lang":"bash","label":"Bash","text":"sudo sysctl iogpu.wired_limit_mb=24576"},{"t":"p","runs":[{"text":"Leave the operating system several gigabytes, and note it resets on reboot."}]},{"t":"h2","id":"colab-s-free-tier","runs":[{"text":"Colab’s free tier"}]},{"t":"p","runs":[{"text":"A T4 with 16 GB of VRAM and about 320 GB/s. That runs an 8B in fp16 at 30 to 40 tok/s, or a 14B at Q4 comfortably. Sessions disconnect, nothing persists, and the terms do not permit running a public service through a tunnel."}]},{"t":"p","runs":[{"text":"Use it for exactly one thing: evaluating whether a model is good enough for your task before you spend money on hardware. That is the test from lesson 1, and Colab is a fine place to run it."}]},{"t":"h2","id":"if-you-are-going-to-buy","runs":[{"text":"If you are going to buy"}]},{"t":"p","runs":[{"text":"The cheapest serious entry is a used NVIDIA card with 12 GB and high bandwidth: around 360 GB/s gets you 35 to 45 tok/s on an 8B at Q4, and prefill stops being the problem. Older datacentre cards with lots of memory are cheap for a reason: their bandwidth or their fp16 performance is poor, and driver support is a project in itself."}]}],"check":{"question":[{"text":"A 3B model at Q4 generates about 14 tok/s on your laptop. You want to roughly double that. Which change is most likely to actually do it?"}],"options":[{"text":[{"text":"Move to a machine with substantially faster memory, such as dual-channel DDR5-6000 instead of DDR4-3200."}],"correct":true,"tempting":null},{"text":[{"text":"Switch to a CPU with twice as many cores."}],"correct":false,"tempting":[{"text":"Assumes decoding is compute-bound, when at batch size one it is waiting on memory, not on arithmetic."}]},{"text":[{"text":"Increase RAM from 8 GB to 16 GB."}],"correct":false,"tempting":[{"text":"Confuses capacity, which decides whether the model fits, with bandwidth, which decides how fast weights stream."}]},{"text":[{"text":"Offload the model to the integrated GPU, since a GPU is faster than a CPU."}],"correct":false,"tempting":[{"text":"An iGPU shares the same memory bus, so it improves prompt processing but leaves the generation ceiling untouched."}]}],"order":[3,2,1,0],"why":null},"prev":{"slug":"does-it-fit","title":"Does it fit: the arithmetic","minutes":9},"next":{"slug":"the-kv-cache-in-depth","title":"The KV cache, in depth","minutes":10}}}