Predicting speed before you download

Generation is limited by memory bandwidth, not by arithmetic

To produce one token, the machine must read every active weight from memory once. A 4.9 GB model means 4.9 GB moved per token. The arithmetic performed on those weights is trivial by comparison: modern processors can do far more multiplication than they can feed themselves data.

So the ceiling is:

tokens per second  =  memory bandwidth  /  bytes of active weights

That one line predicts local model speed better than any benchmark chart, and you can evaluate it before downloading anything.

Worked examples

An ordinary laptop, dual-channel DDR4-3200. Bandwidth around 50 GB/s. An 8B model at Q4 is 4.9 GB. The ceiling is 50 / 4.9 ≈ 10 tokens per second, and in practice you will see 6 to 8. That is about four words per second: readable, not comfortable.

The same laptop, a 3B model at Q4: 2.0 GB. Ceiling 25 tokens per second, real perhaps 15 to 18. This is the reason 3B models feel usable on machines where 8B models feel broken, and it is entirely about the file size.

Apple M-series with unified memory. An M2 Pro has around 200 GB/s, an M2 Max around 400, an M3 Max around 400. At 200 GB/s the 8B at Q4 gives a ceiling of 40 tokens per second, real around 25 to 30. The GPU and the CPU share this memory, which is why Apple laptops punch far above x86 laptops of the same nominal specification.

A consumer GPU. An RTX 3090 or 4090 has around 1,000 GB/s. Ceiling for the 8B at Q4 is roughly 200 tokens per second; real numbers of 100 to 140 are typical.

A datacentre GPU. An H100 has around 3,350 GB/s. The ceiling is enormous, which is why these machines are used for many users at once rather than for one.

Expect 60% to 80% of the theoretical number. The gap is attention over the growing cache, kernel overhead, and the fact that nothing achieves peak bandwidth.

Prefill is a different animal

Reading your prompt is compute-bound, not bandwidth-bound: all the tokens of the prompt go through the weights together, so the weights are read once for the whole batch. The cost is roughly 2 × parameters × tokens floating point operations, and processors have a great deal of arithmetic capacity.

This is why llama-bench reports hundreds or thousands of tokens per second for prompt processing and ten or forty for generation, on the same machine and model. It is not an inconsistency; the two phases are limited by different resources.

Two practical consequences. A long prompt with a short answer is fast. A short prompt with a long answer is slow. And on a CPU, prefill of a very long document can still take a while, because CPU arithmetic capacity is modest too.

Three things this explains at once

Why batching is nearly free. Serving eight users at once reads the weights once for all eight. Total throughput rises almost eight-fold while each individual user sees little slowdown. This is the entire economic basis of hosted inference.

Why mixture-of-experts models are fast. Only the active experts are read per token, so the bandwidth cost matches the active parameter count even though the memory occupied matches the total.

Why quantising speeds things up. Halving the bytes halves the reading, so it roughly doubles the speed. Quantisation is a speed optimisation as much as a memory one, which is often forgotten.

Use it as a purchase test

Before spending money, ask what bandwidth you are buying rather than what compute.

A machine with a large amount of slow memory will run big models slowly; a machine with fast memory will run models that fit quickly. For local inference specifically, memory bandwidth and memory capacity are the two numbers that matter, and raw compute barely enters into it.

The quickest sanity check on any claim about local speed: divide the claimed bandwidth by the model file size. If someone reports 60 tokens per second for a 5 GB model, they are claiming 300 GB/s of bandwidth, and you can ask what machine provides it.