Most writing about local models assumes a 4090. Most readers do not have one. Here is what a mid-range laptop, a Mac, and a free Colab session genuinely do, with the arithmetic that predicts it.
The one equation for generation speed
Generating one token at a time reads every weight in the model, once. So decoding is limited by memory bandwidth, not by compute:
tokens/sec (ceiling) = effective_bandwidth / model_size_in_bytesEffective bandwidth is roughly 60–75% of the theoretical figure. A typical dual-channel DDR4-3200 laptop has 51.2 GB/s theoretical, so about 30 GB/s real.
On that machine:
| model | size at Q4 | predicted | typical measured | |---|---|---|---| | 1.7B | 1.1 GB | 27 tok/s | 20–25 | | 3B | 2.0 GB | 15 tok/s | 11–15 | | 4B | 2.5 GB | 12 tok/s | 9–12 | | 8B | 4.9 GB | 6 tok/s | 4–7 |
The prediction is close enough to be useful. Do it before downloading.
For 8 GB of RAM with no discrete GPU, the honest recommendation is a 3B or 4B model at Q4_K_M with 4k–8k context. That is genuinely useful: summarising, rewriting, classification, extraction, tagging, simple questions. It is not useful for hard code or long reasoning, and pretending otherwise wastes your evening.
The part nobody warns you about
Processing the prompt is a different operation. It handles all tokens at once, so it is compute-bound, and CPUs are bad at it. An 8B model on a laptop CPU might prefill at 20–60 tokens/second. A 2,000-token prompt is therefore 30–100 seconds of nothing happening before the first word appears.
That delay is what makes CPU inference feel broken, and it is why RAG systems that stuff 6,000 tokens of retrieved context are miserable on a laptop. Keep prompts short. Reuse cached prefixes where the server supports it.
Integrated graphics help here more than you would expect and less than you would hope. An iGPU shares the same RAM, so it cannot raise the decoding ceiling — same bandwidth. But it has far more parallel compute than the CPU cores, so prefill often gets 2–3x faster. llama.cpp's Vulkan backend works on Intel and AMD integrated graphics:
llama-server -m model.gguf -ngl 99 -c 4096 # a Vulkan build offloads to the iGPUMacs
Apple Silicon is quietly the best value for this, because the CPU and GPU share one pool of high-bandwidth memory. Approximate figures: base M1/M2/M3 around 68–100 GB/s, M4 around 120, the Pro chips 200–270, the Max chips 400–550, Ultra around 800.
A 16 GB M-series machine runs an 8B at Q4 around 15–25 tok/s with good prefill. A 32 GB one runs 14B comfortably and 32B at Q4 slowly but usably.
macOS caps how much RAM the GPU may hold — roughly 65–75%. On a 32 GB machine you can raise it:
sudo sysctl iogpu.wired_limit_mb=24576Leave the operating system several gigabytes, and note it resets on reboot.
Colab's free tier
A T4 with 16 GB of VRAM and about 320 GB/s. That runs an 8B in fp16 at 30–40 tok/s, or a 14B at Q4 comfortably. Sessions disconnect, nothing persists, and the terms do not permit running a public service through a tunnel.
Use it for exactly one thing: evaluating whether a model is good enough for your task before you spend money on hardware. That is the test from lesson 1, and Colab is a fine place to run it.
If you are going to buy
The cheapest serious entry is a used NVIDIA card with 12 GB and high bandwidth — around 360 GB/s gets you 35–45 tok/s on an 8B at Q4, and prefill stops being the problem. Older datacentre cards with lots of memory are cheap for a reason: their bandwidth or their fp16 performance is poor, and driver support is a project in itself.
Before you move on