{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"predicting-tokens-per-second","title":"Predicting speed before you download","url":"https://addaly.com/learn/open-models/predicting-tokens-per-second","position":36,"of":71,"minutes":9,"answer":"Decoding reads every active weight from memory once per token, so tokens per second is bounded by memory bandwidth divided by model file size.","keyIdea":"Decoding reads every active weight from memory once per token, so tokens per second is bounded by memory bandwidth divided by model file size, while prefill is compute-bound and therefore one or two orders of magnitude faster."},"body":{"format":"markdown","text":"## Generation is limited by memory bandwidth, not by arithmetic\n\nTo produce one token, the machine must read every active weight from memory once. A 4.9 GB model means 4.9 GB moved per token. The arithmetic performed on those weights is trivial by comparison: modern processors can do far more multiplication than they can feed themselves data.\n\nSo the ceiling is:\n\n```\ntokens per second  =  memory bandwidth  /  bytes of active weights\n```\n\nThat one line predicts local model speed better than any benchmark chart, and you can evaluate it before downloading anything.\n\n> Figure: Ceiling for an 8B at Q4: bandwidth divided by 4.9 GB. Bars (tokens per second, before losses): Laptop, dual-channel DDR4: 50 GB/s 10; Apple M2 Pro: about 200 GB/s 41; Apple M2 Max: about 400 GB/s 82; RTX 4090: about 1,000 GB/s 204. Expect sixty to eighty per cent of these in practice. The same arithmetic explains why a 3B at 2.0 GB feels usable on the machine where the 8B feels broken, and it is the quickest check on anybody's claim about local speed: divide the bandwidth by the file size and ask what machine provides it.\n\n## Worked examples\n\n**An ordinary laptop, dual-channel DDR4-3200.** Bandwidth around 50 GB/s. An 8B model at Q4 is 4.9 GB. The ceiling is 50 / 4.9 ≈ **10 tokens per second**, and in practice you will see 6 to 8. That is about four words per second: readable, not comfortable.\n\n**The same laptop, a 3B model at Q4**: 2.0 GB. Ceiling 25 tokens per second, real perhaps 15 to 18. This is the reason 3B models feel usable on machines where 8B models feel broken, and it is entirely about the file size.\n\n**Apple M-series with unified memory.** An M2 Pro has around 200 GB/s, an M2 Max around 400, an M3 Max around 400. At 200 GB/s the 8B at Q4 gives a ceiling of 40 tokens per second, real around 25 to 30. The GPU and the CPU share this memory, which is why Apple laptops punch far above x86 laptops of the same nominal specification.\n\n**A consumer GPU.** An RTX 3090 or 4090 has around 1,000 GB/s. Ceiling for the 8B at Q4 is roughly 200 tokens per second; real numbers of 100 to 140 are typical.\n\n**A datacentre GPU.** An H100 has around 3,350 GB/s. The ceiling is enormous, which is why these machines are used for many users at once rather than for one.\n\nExpect 60% to 80% of the theoretical number. The gap is attention over the growing cache, kernel overhead, and the fact that nothing achieves peak bandwidth.\n\n## Prefill is a different animal\n\nReading your prompt is compute-bound, not bandwidth-bound: all the tokens of the prompt go through the weights together, so the weights are read once for the whole batch. The cost is roughly 2 × parameters × tokens floating point operations, and processors have a great deal of arithmetic capacity.\n\nThis is why `llama-bench` reports hundreds or thousands of tokens per second for prompt processing and ten or forty for generation, on the same machine and model. It is not an inconsistency; the two phases are limited by different resources.\n\nTwo practical consequences. A long prompt with a short answer is fast. A short prompt with a long answer is slow. And on a CPU, prefill of a very long document can still take a while, because CPU arithmetic capacity is modest too.\n\n## Three things this explains at once\n\n**Why batching is nearly free.** Serving eight users at once reads the weights once for all eight. Total throughput rises almost eight-fold while each individual user sees little slowdown. This is the entire economic basis of hosted inference.\n\n**Why mixture-of-experts models are fast.** Only the active experts are read per token, so the bandwidth cost matches the active parameter count even though the memory occupied matches the total.\n\n**Why quantising speeds things up.** Halving the bytes halves the reading, so it roughly doubles the speed. Quantisation is a speed optimisation as much as a memory one, which is often forgotten.\n\n## Use it as a purchase test\n\nBefore spending money, ask what bandwidth you are buying rather than what compute.\n\nA machine with a large amount of slow memory will run big models slowly; a machine with fast memory will run models that fit quickly. For local inference specifically, memory bandwidth and memory capacity are the two numbers that matter, and raw compute barely enters into it.\n\n> The quickest sanity check on any claim about local speed: divide the claimed bandwidth by the model file size. If someone reports 60 tokens per second for a 5 GB model, they are claiming 300 GB/s of bandwidth, and you can ask what machine provides it."},"check":{"question":"Someone reports 45 tokens per second generating with a 13B model at Q4 (about 8 GB) on a laptop with dual-channel DDR5 at roughly 80 GB/s. What should you conclude?","options":[{"text":"It is plausible, since quantised models bypass the memory bottleneck by computing in lower precision.","correct":false,"tempting":"Lower precision reduces bytes read, which is already accounted for in the 8 GB figure; it does not remove the bandwidth limit."},{"text":"It is plausible if the prompt was long, since long prompts raise generation throughput.","correct":false,"tempting":"Confuses prefill with decode; a long prompt raises prompt-processing throughput and slightly slows decoding."},{"text":"The number implies about 360 GB/s of bandwidth, far above the machine's 80 GB/s, so it is probably prompt processing rather than generation.","correct":true},{"text":"It is plausible because the operating system caches the model, so weights are read from cache rather than memory.","correct":false,"tempting":"The cache the operating system keeps is in the same memory; CPU caches are megabytes and cannot hold an 8 GB model."}]},"prev":{"slug":"ollama-and-the-friendly-path","title":"Ollama and the friendly path","url":"https://addaly.com/learn/open-models/ollama-and-the-friendly-path"},"next":{"slug":"serving-many-users","title":"Serving many users at once","url":"https://addaly.com/learn/open-models/serving-many-users"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Predicting speed before you download', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/predicting-tokens-per-second (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Predicting speed before you download","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Decoding reads every active weight from memory once per token, so tokens per second is bounded by memory bandwidth divided by model file size, while prefill is compute-bound and therefore one or two orders of magnitude faster.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens per second, before losses","title":"Ceiling for an 8B at Q4: bandwidth divided by 4.9 GB","series":[{"label":"Laptop, dual-channel DDR4: 50 GB/s","value":10,"highlight":true},{"label":"Apple M2 Pro: about 200 GB/s","value":41},{"label":"Apple M2 Max: about 400 GB/s","value":82},{"label":"RTX 4090: about 1,000 GB/s","value":204}],"caption":"Expect sixty to eighty per cent of these in practice. The same arithmetic explains why a 3B at 2.0 GB feels usable on the machine where the 8B feels broken, and it is the quickest check on anybody’s claim about local speed: divide the bandwidth by the file size and ask what machine provides it."},"alt":"Figure: Ceiling for an 8B at Q4: bandwidth divided by 4.9 GB. Bars (tokens per second, before losses): Laptop, dual-channel DDR4: 50 GB/s 10; Apple M2 Pro: about 200 GB/s 41; Apple M2 Max: about 400 GB/s 82; RTX 4090: about 1,000 GB/s 204. Expect sixty to eighty per cent of these in practice. The same arithmetic explains why a 3B at 2.0 GB feels usable on the machine where the 8B feels broken, and it is the quickest check on anybody’s claim about local speed: divide the bandwidth by the file size and ask what machine provides it."}},"blocks":[{"t":"h2","id":"generation-is-limited-by-memory-bandwidth-not-by-arithmetic","runs":[{"text":"Generation is limited by memory bandwidth, not by arithmetic"}]},{"t":"p","runs":[{"text":"To produce one token, the machine must read every active weight from memory once. A 4.9 GB model means 4.9 GB moved per token. The arithmetic performed on those weights is trivial by comparison: modern processors can do far more multiplication than they can feed themselves data."}]},{"t":"p","runs":[{"text":"So the ceiling is:"}]},{"t":"code","lang":"","label":"Text","text":"tokens per second  =  memory bandwidth  /  bytes of active weights"},{"t":"p","runs":[{"text":"That one line predicts local model speed better than any benchmark chart, and you can evaluate it before downloading anything."}]},{"t":"h2","id":"worked-examples","runs":[{"text":"Worked examples"}]},{"t":"p","runs":[{"text":"An ordinary laptop, dual-channel DDR4-3200.","b":true},{"text":" Bandwidth around 50 GB/s. An 8B model at Q4 is 4.9 GB. The ceiling is 50 / 4.9 ≈ "},{"text":"10 tokens per second","b":true},{"text":", and in practice you will see 6 to 8. That is about four words per second: readable, not comfortable."}]},{"t":"p","runs":[{"text":"The same laptop, a 3B model at Q4","b":true},{"text":": 2.0 GB. Ceiling 25 tokens per second, real perhaps 15 to 18. This is the reason 3B models feel usable on machines where 8B models feel broken, and it is entirely about the file size."}]},{"t":"p","runs":[{"text":"Apple M-series with unified memory.","b":true},{"text":" An M2 Pro has around 200 GB/s, an M2 Max around 400, an M3 Max around 400. At 200 GB/s the 8B at Q4 gives a ceiling of 40 tokens per second, real around 25 to 30. The GPU and the CPU share this memory, which is why Apple laptops punch far above x86 laptops of the same nominal specification."}]},{"t":"p","runs":[{"text":"A consumer GPU.","b":true},{"text":" An RTX 3090 or 4090 has around 1,000 GB/s. Ceiling for the 8B at Q4 is roughly 200 tokens per second; real numbers of 100 to 140 are typical."}]},{"t":"p","runs":[{"text":"A datacentre GPU.","b":true},{"text":" An H100 has around 3,350 GB/s. The ceiling is enormous, which is why these machines are used for many users at once rather than for one."}]},{"t":"p","runs":[{"text":"Expect 60% to 80% of the theoretical number. The gap is attention over the growing cache, kernel overhead, and the fact that nothing achieves peak bandwidth."}]},{"t":"h2","id":"prefill-is-a-different-animal","runs":[{"text":"Prefill is a different animal"}]},{"t":"p","runs":[{"text":"Reading your prompt is compute-bound, not bandwidth-bound: all the tokens of the prompt go through the weights together, so the weights are read once for the whole batch. The cost is roughly 2 × parameters × tokens floating point operations, and processors have a great deal of arithmetic capacity."}]},{"t":"p","runs":[{"text":"This is why "},{"text":"llama-bench","code":true},{"text":" reports hundreds or thousands of tokens per second for prompt processing and ten or forty for generation, on the same machine and model. It is not an inconsistency; the two phases are limited by different resources."}]},{"t":"p","runs":[{"text":"Two practical consequences. A long prompt with a short answer is fast. A short prompt with a long answer is slow. And on a CPU, prefill of a very long document can still take a while, because CPU arithmetic capacity is modest too."}]},{"t":"h2","id":"three-things-this-explains-at-once","runs":[{"text":"Three things this explains at once"}]},{"t":"p","runs":[{"text":"Why batching is nearly free.","b":true},{"text":" Serving eight users at once reads the weights once for all eight. Total throughput rises almost eight-fold while each individual user sees little slowdown. This is the entire economic basis of hosted inference."}]},{"t":"p","runs":[{"text":"Why mixture-of-experts models are fast.","b":true},{"text":" Only the active experts are read per token, so the bandwidth cost matches the active parameter count even though the memory occupied matches the total."}]},{"t":"p","runs":[{"text":"Why quantising speeds things up.","b":true},{"text":" Halving the bytes halves the reading, so it roughly doubles the speed. Quantisation is a speed optimisation as much as a memory one, which is often forgotten."}]},{"t":"h2","id":"use-it-as-a-purchase-test","runs":[{"text":"Use it as a purchase test"}]},{"t":"p","runs":[{"text":"Before spending money, ask what bandwidth you are buying rather than what compute."}]},{"t":"p","runs":[{"text":"A machine with a large amount of slow memory will run big models slowly; a machine with fast memory will run models that fit quickly. For local inference specifically, memory bandwidth and memory capacity are the two numbers that matter, and raw compute barely enters into it."}]},{"t":"quote","runs":[{"text":"The quickest sanity check on any claim about local speed: divide the claimed bandwidth by the model file size. If someone reports 60 tokens per second for a 5 GB model, they are claiming 300 GB/s of bandwidth, and you can ask what machine provides it."}]}],"check":{"question":[{"text":"Someone reports 45 tokens per second generating with a 13B model at Q4 (about 8 GB) on a laptop with dual-channel DDR5 at roughly 80 GB/s. What should you conclude?"}],"options":[{"text":[{"text":"It is plausible, since quantised models bypass the memory bottleneck by computing in lower precision."}],"correct":false,"tempting":[{"text":"Lower precision reduces bytes read, which is already accounted for in the 8 GB figure; it does not remove the bandwidth limit."}]},{"text":[{"text":"It is plausible if the prompt was long, since long prompts raise generation throughput."}],"correct":false,"tempting":[{"text":"Confuses prefill with decode; a long prompt raises prompt-processing throughput and slightly slows decoding."}]},{"text":[{"text":"The number implies about 360 GB/s of bandwidth, far above the machine’s 80 GB/s, so it is probably prompt processing rather than generation."}],"correct":true,"tempting":null},{"text":[{"text":"It is plausible because the operating system caches the model, so weights are read from cache rather than memory."}],"correct":false,"tempting":[{"text":"The cache the operating system keeps is in the same memory; CPU caches are megabytes and cannot hold an 8 GB model."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"ollama-and-the-friendly-path","title":"Ollama and the friendly path","minutes":8},"next":{"slug":"serving-many-users","title":"Serving many users at once","minutes":9}}}