Most writing about local models assumes a 4090. Most readers do not have one. Here is what a mid-range laptop, a Mac, and a free Colab session do, with the arithmetic that predicts it.
The one equation for generation speed
Generating one token at a time reads every weight in the model, once. So decoding is limited by memory bandwidth, not by compute:
tokens/sec (ceiling) = effective_bandwidth / model_size_in_bytesEffective bandwidth is roughly 60 to 75% of the theoretical figure. A typical dual-channel DDR4-3200 laptop has 51.2 GB/s theoretical, so about 30 GB/s real.
On that machine:
| model | size at Q4 | predicted | typical measured |
|---|---|---|---|
| 1.7B | 1.1 GB | 27 tok/s | 20 to 25 |
| 3B | 2.0 GB | 15 tok/s | 11 to 15 |
| 4B | 2.5 GB | 12 tok/s | 9 to 12 |
| 8B | 4.9 GB | 6 tok/s | 4 to 7 |
The prediction is close enough to be useful. Do it before downloading.
For 8 GB of RAM with no discrete GPU, the honest recommendation is a 3B or 4B model at Q4_K_M with 4k to 8k context. That is useful: summarising, rewriting, classification, extraction, tagging, simple questions. It is not useful for hard code or long reasoning, and pretending otherwise wastes your evening.
The part nobody warns you about
Processing the prompt is a different operation. It handles all tokens at once, so it is compute-bound, and CPUs are bad at it. An 8B model on a laptop CPU might prefill at 20 to 60 tokens/second. A 2,000-token prompt is therefore 30 to 100 seconds of nothing happening before the first word appears.
That delay is what makes CPU inference feel broken, and it is why RAG systems that stuff 6,000 tokens of retrieved context are miserable on a laptop. Keep prompts short. Reuse cached prefixes where the server supports it.
Integrated graphics help here more than you would expect and less than you would hope. An iGPU shares the same RAM, so it cannot raise the decoding ceiling: same bandwidth. But it has far more parallel compute than the CPU cores, so prefill often gets 2 to 3x faster. llama.cpp’s Vulkan backend works on Intel and AMD integrated graphics:
llama-server -m model.gguf -ngl 99 -c 4096 # a Vulkan build offloads to the iGPUMacs
Apple Silicon is quietly the best value for this, because the CPU and GPU share one pool of high-bandwidth memory. Approximate figures: base M1/M2/M3 around 68 to 100 GB/s, M4 around 120, the Pro chips 200 to 270, the Max chips 400 to 550, Ultra around 800.
A 16 GB M-series machine runs an 8B at Q4 around 15 to 25 tok/s with good prefill. A 32 GB one runs 14B comfortably and 32B at Q4 slowly but usably.
macOS caps how much RAM the GPU may hold: roughly 65 to 75%. On a 32 GB machine you can raise it:
sudo sysctl iogpu.wired_limit_mb=24576Leave the operating system several gigabytes, and note it resets on reboot.
Colab’s free tier
A T4 with 16 GB of VRAM and about 320 GB/s. That runs an 8B in fp16 at 30 to 40 tok/s, or a 14B at Q4 comfortably. Sessions disconnect, nothing persists, and the terms do not permit running a public service through a tunnel.
Use it for exactly one thing: evaluating whether a model is good enough for your task before you spend money on hardware. That is the test from lesson 1, and Colab is a fine place to run it.
If you are going to buy
The cheapest serious entry is a used NVIDIA card with 12 GB and high bandwidth: around 360 GB/s gets you 35 to 45 tok/s on an 8B at Q4, and prefill stops being the problem. Older datacentre cards with lots of memory are cheap for a reason: their bandwidth or their fp16 performance is poor, and driver support is a project in itself.
Every lesson here is open. Your progress is recorded once you pass the Open Model Ecosystem exam.
Check your understanding
A 3B model at Q4 generates about 14 tok/s on your laptop. You want to roughly double that. Which change is most likely to actually do it?
Pick the one you would defend. Your answer stays private.
Stuck on something?Ask Tri, your AI tutor