{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"the-bandwidth-wall","title":"The bandwidth wall: why decode speed is a division","url":"https://addaly.com/learn/how-llms-work/the-bandwidth-wall","position":59,"of":78,"minutes":9,"answer":"Single-stream generation reads every weight once per token, so its speed is memory bandwidth divided by model bytes, which is why quantisation.","keyIdea":"Single-stream generation reads every weight once per token, so its speed is memory bandwidth divided by model bytes, which is why quantisation, mixture-of-experts and batching help and why a faster-compute GPU with the same bandwidth does not."},"body":{"format":"markdown","text":"## One division\n\nHere is the most useful single fact in this module. When a dense model generates one token for one user, it must read every weight once. So the fastest it can possibly go is:\n\n```text\ntokens per second  ≈  memory bandwidth  ÷  bytes of weights\n```\n\nThat is it. Not the number of cores, not the FLOPS on the spec sheet. Bandwidth divided by size.\n\nPut numbers in. A data-centre GPU of the H100 class moves about 3.35 terabytes a second from its memory. An 8-billion-parameter model in bf16 is 16 gigabytes. 3,350 ÷ 16 is about 209 tokens a second, and a real system reaches sixty to eighty per cent of that bound.\n\nA consumer card with a terabyte a second and the same model: about 60 a second. A laptop with unified memory at 200 gigabytes a second, running the same model quantised to four bits (call it 5 gigabytes) has a bound of 40 a second, and delivers 25 to 35. A flagship phone at 60 gigabytes a second, running a 3-billion-parameter model at four bits, has a bound near 30 and delivers half that. A desktop CPU with two channels of DDR5, around 90 gigabytes a second, gets a bound of 18 on the 5-gigabyte model and delivers about ten.\n\n> Figure: Bandwidth divided by bytes, five machines. Bars (tokens per second, single stream, upper bound): Data-centre GPU, 8B at bf16 (3,350 ÷ 16) 209; Consumer GPU, 8B at bf16 (1,000 ÷ 16) 63; Laptop, unified memory, 8B at 4-bit (200 ÷ 5) 40; Flagship phone, 3B at 4-bit (60 ÷ 2) 30; Desktop CPU, DDR5, 8B at 4-bit (90 ÷ 5) 18. Every number is one division, and real systems reach sixty to eighty per cent of it. Note what is absent: not one of these used a FLOPS figure. A card with twice the compute and the same bandwidth generates single-stream tokens at exactly the same rate.\n\nEvery one of those matches what people report, and none of them needed a benchmark to predict.\n\n## Why the compute sits idle\n\nGenerating one token is a matrix-vector multiplication for each weight matrix: a vector of a few thousand numbers against a matrix of millions. Each weight is loaded from memory, used in one multiply-add, and discarded. That is about one floating-point operation per byte moved.\n\nA modern GPU can perform hundreds of operations for every byte it loads. When the work offers one per byte, the arithmetic units wait on memory almost all of the time. The technical phrase is that decode is **memory-bandwidth bound**, and it is why the spec-sheet number that matters for single-stream generation is the memory number.\n\n## Prefill is the other regime\n\nProcessing the prompt is different. All the prompt's tokens go through at once, so each weight loaded is used against thousands of vectors, not one. That is a matrix-matrix multiplication with high arithmetic intensity, and it is bounded by compute instead.\n\nThe 8-billion model costs about 2 × 8 billion operations per token (module five's inference formula). The H100-class card does roughly a thousand trillion bf16 operations a second, so the theoretical prefill rate is around 60,000 tokens a second, and real systems manage 10,000 to 20,000. A 10,000-token prompt therefore takes roughly half a second to a second before the first output token: module three's time-to-first-token, now with a cause.\n\n## The crossover, and why servers batch\n\nDivide compute by bandwidth for the card: a thousand trillion operations a second over 3.35 trillion bytes a second is about 300 operations per byte. Decode offers one. So you need about 300 tokens in flight (300 sequences generating at once) before the card is doing arithmetic as fast as it is reading memory.\n\nThat is the number behind module two's batching lesson. A server holding a few hundred users' sequences reads each weight once per step and uses it a few hundred times. Throughput per card rises by roughly the batch size, until the crossover, after which compute is the limit. Below the crossover, adding users is nearly free; that is the economic fact the whole serving industry is built on, and the next lessons develop it.\n\n## Three consequences you can now predict\n\n**Quantisation speeds decode.** Halve the bytes and you double the bound. A four-bit model is not faster because the arithmetic is simpler; it is faster because there is less to read. The lesson after next explains what the halving costs.\n\n**Mixture-of-experts speeds decode.** Only the active experts' weights are read for a given token, mostly. In a batch, different tokens route to different experts, so a busy server reads more of the model per step than the per-token figure suggests.\n\n**Long contexts slow decode, not just prefill.** The key-value cache is read at every step too. Module two's arithmetic gave 131 kilobytes per token for the 8-billion model; at 32,000 tokens that is 4.2 gigabytes of cache read alongside 16 gigabytes of weights: a quarter more traffic, a quarter slower. At 128,000 tokens the cache exceeds the weights, and the model runs at less than half its short-context speed. People notice this and blame the model.\n\n## Measure it once\n\nRun any model locally, note the tokens per second, and divide by the bound you computed. Fifty to eighty per cent is normal. Far below that means you are not bandwidth-bound: some layers are on the CPU, the weights are crossing a slow bus, or a kernel is mis-set. The ratio tells you whether to tune or to shrug.\n\n## What this is not about\n\nIt is not about FLOPS. A card with twice the compute and the same bandwidth generates single-stream tokens at the same rate as the old one, and teams that have bought the upgrade have been surprised. The H100 to H200 step kept the compute and raised the bandwidth to 4.8 terabytes a second, and it was a real decode speed-up for exactly that reason. It is also why Apple silicon, with its wide unified memory, is competitive per pound for local models despite far less raw compute. When you are shopping for generation speed, you are shopping for bandwidth."},"check":{"question":"A team replaces a GPU with a newer one that has twice the compute and identical memory bandwidth. Single-user token generation speed does not change. Why?","options":[{"text":"The new card's extra compute is not used until the driver and inference engine are updated to target it.","correct":false,"tempting":"A plausible operational guess, and sometimes true of new hardware, but it misses that the workload could not use the extra compute even when fully supported."},{"text":"The model is too small to need the extra compute; a larger model would have shown the speed-up.","correct":false,"tempting":"Backwards: a larger model has more bytes to read per token and is even more bandwidth-bound, so it would show the same non-result more starkly."},{"text":"Each generated token reads every weight once, so decode is bounded by bandwidth, which did not change.","correct":true},{"text":"The key-value cache dominates memory traffic at any realistic prompt length, and its size depends only on the context, so no change to the GPU can affect generation speed.","correct":false,"tempting":"Right that the cache adds traffic, wrong in every other clause: at ordinary lengths the weights dominate, and a higher-bandwidth GPU would speed up reading both."}]},"prev":{"slug":"language-imbalance","title":"Why it is worse in Hindi, and costs more","url":"https://addaly.com/learn/how-llms-work/language-imbalance"},"next":{"slug":"will-it-fit","title":"Will it fit: the memory budget of a running model","url":"https://addaly.com/learn/how-llms-work/will-it-fit"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The bandwidth wall: why decode speed is a division', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/the-bandwidth-wall (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The bandwidth wall: why decode speed is a division","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"Single-stream generation reads every weight once per token, so its speed is memory bandwidth divided by model bytes, which is why quantisation, mixture-of-experts and batching help and why a faster-compute GPU with the same bandwidth does not.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 7.1","figure":{"type":"bars","unit":"tokens per second, single stream, upper bound","title":"Bandwidth divided by bytes, five machines","series":[{"label":"Data-centre GPU, 8B at bf16 (3,350 ÷ 16)","value":209,"highlight":true},{"label":"Consumer GPU, 8B at bf16 (1,000 ÷ 16)","value":63},{"label":"Laptop, unified memory, 8B at 4-bit (200 ÷ 5)","value":40},{"label":"Flagship phone, 3B at 4-bit (60 ÷ 2)","value":30},{"label":"Desktop CPU, DDR5, 8B at 4-bit (90 ÷ 5)","value":18}],"caption":"Every number is one division, and real systems reach sixty to eighty per cent of it. Note what is absent: not one of these used a FLOPS figure. A card with twice the compute and the same bandwidth generates single-stream tokens at exactly the same rate."},"alt":"Figure: Bandwidth divided by bytes, five machines. Bars (tokens per second, single stream, upper bound): Data-centre GPU, 8B at bf16 (3,350 ÷ 16) 209; Consumer GPU, 8B at bf16 (1,000 ÷ 16) 63; Laptop, unified memory, 8B at 4-bit (200 ÷ 5) 40; Flagship phone, 3B at 4-bit (60 ÷ 2) 30; Desktop CPU, DDR5, 8B at 4-bit (90 ÷ 5) 18. Every number is one division, and real systems reach sixty to eighty per cent of it. Note what is absent: not one of these used a FLOPS figure. A card with twice the compute and the same bandwidth generates single-stream tokens at exactly the same rate."}},"blocks":[{"t":"h2","id":"one-division","runs":[{"text":"One division"}]},{"t":"p","runs":[{"text":"Here is the most useful single fact in this module. When a dense model generates one token for one user, it must read every weight once. So the fastest it can possibly go is:"}]},{"t":"code","lang":"text","label":"Text","text":"tokens per second  ≈  memory bandwidth  ÷  bytes of weights"},{"t":"p","runs":[{"text":"That is it. Not the number of cores, not the FLOPS on the spec sheet. Bandwidth divided by size."}]},{"t":"p","runs":[{"text":"Put numbers in. A data-centre GPU of the H100 class moves about 3.35 terabytes a second from its memory. An 8-billion-parameter model in bf16 is 16 gigabytes. 3,350 ÷ 16 is about 209 tokens a second, and a real system reaches sixty to eighty per cent of that bound."}]},{"t":"p","runs":[{"text":"A consumer card with a terabyte a second and the same model: about 60 a second. A laptop with unified memory at 200 gigabytes a second, running the same model quantised to four bits (call it 5 gigabytes) has a bound of 40 a second, and delivers 25 to 35. A flagship phone at 60 gigabytes a second, running a 3-billion-parameter model at four bits, has a bound near 30 and delivers half that. A desktop CPU with two channels of DDR5, around 90 gigabytes a second, gets a bound of 18 on the 5-gigabyte model and delivers about ten."}]},{"t":"p","runs":[{"text":"Every one of those matches what people report, and none of them needed a benchmark to predict."}]},{"t":"h2","id":"why-the-compute-sits-idle","runs":[{"text":"Why the compute sits idle"}]},{"t":"p","runs":[{"text":"Generating one token is a matrix-vector multiplication for each weight matrix: a vector of a few thousand numbers against a matrix of millions. Each weight is loaded from memory, used in one multiply-add, and discarded. That is about one floating-point operation per byte moved."}]},{"t":"p","runs":[{"text":"A modern GPU can perform hundreds of operations for every byte it loads. When the work offers one per byte, the arithmetic units wait on memory almost all of the time. The technical phrase is that decode is "},{"text":"memory-bandwidth bound","b":true},{"text":", and it is why the spec-sheet number that matters for single-stream generation is the memory number."}]},{"t":"h2","id":"prefill-is-the-other-regime","runs":[{"text":"Prefill is the other regime"}]},{"t":"p","runs":[{"text":"Processing the prompt is different. All the prompt’s tokens go through at once, so each weight loaded is used against thousands of vectors, not one. That is a matrix-matrix multiplication with high arithmetic intensity, and it is bounded by compute instead."}]},{"t":"p","runs":[{"text":"The 8-billion model costs about 2 × 8 billion operations per token (module five’s inference formula). The H100-class card does roughly a thousand trillion bf16 operations a second, so the theoretical prefill rate is around 60,000 tokens a second, and real systems manage 10,000 to 20,000. A 10,000-token prompt therefore takes roughly half a second to a second before the first output token: module three’s time-to-first-token, now with a cause."}]},{"t":"h2","id":"the-crossover-and-why-servers-batch","runs":[{"text":"The crossover, and why servers batch"}]},{"t":"p","runs":[{"text":"Divide compute by bandwidth for the card: a thousand trillion operations a second over 3.35 trillion bytes a second is about 300 operations per byte. Decode offers one. So you need about 300 tokens in flight (300 sequences generating at once) before the card is doing arithmetic as fast as it is reading memory."}]},{"t":"p","runs":[{"text":"That is the number behind module two’s batching lesson. A server holding a few hundred users’ sequences reads each weight once per step and uses it a few hundred times. Throughput per card rises by roughly the batch size, until the crossover, after which compute is the limit. Below the crossover, adding users is nearly free; that is the economic fact the whole serving industry is built on, and the next lessons develop it."}]},{"t":"h2","id":"three-consequences-you-can-now-predict","runs":[{"text":"Three consequences you can now predict"}]},{"t":"p","runs":[{"text":"Quantisation speeds decode.","b":true},{"text":" Halve the bytes and you double the bound. A four-bit model is not faster because the arithmetic is simpler; it is faster because there is less to read. The lesson after next explains what the halving costs."}]},{"t":"p","runs":[{"text":"Mixture-of-experts speeds decode.","b":true},{"text":" Only the active experts’ weights are read for a given token, mostly. In a batch, different tokens route to different experts, so a busy server reads more of the model per step than the per-token figure suggests."}]},{"t":"p","runs":[{"text":"Long contexts slow decode, not just prefill.","b":true},{"text":" The key-value cache is read at every step too. Module two’s arithmetic gave 131 kilobytes per token for the 8-billion model; at 32,000 tokens that is 4.2 gigabytes of cache read alongside 16 gigabytes of weights: a quarter more traffic, a quarter slower. At 128,000 tokens the cache exceeds the weights, and the model runs at less than half its short-context speed. People notice this and blame the model."}]},{"t":"h2","id":"measure-it-once","runs":[{"text":"Measure it once"}]},{"t":"p","runs":[{"text":"Run any model locally, note the tokens per second, and divide by the bound you computed. Fifty to eighty per cent is normal. Far below that means you are not bandwidth-bound: some layers are on the CPU, the weights are crossing a slow bus, or a kernel is mis-set. The ratio tells you whether to tune or to shrug."}]},{"t":"h2","id":"what-this-is-not-about","runs":[{"text":"What this is not about"}]},{"t":"p","runs":[{"text":"It is not about FLOPS. A card with twice the compute and the same bandwidth generates single-stream tokens at the same rate as the old one, and teams that have bought the upgrade have been surprised. The H100 to H200 step kept the compute and raised the bandwidth to 4.8 terabytes a second, and it was a real decode speed-up for exactly that reason. It is also why Apple silicon, with its wide unified memory, is competitive per pound for local models despite far less raw compute. When you are shopping for generation speed, you are shopping for bandwidth."}]}],"check":{"question":[{"text":"A team replaces a GPU with a newer one that has twice the compute and identical memory bandwidth. Single-user token generation speed does not change. Why?"}],"options":[{"text":[{"text":"The new card’s extra compute is not used until the driver and inference engine are updated to target it."}],"correct":false,"tempting":[{"text":"A plausible operational guess, and sometimes true of new hardware, but it misses that the workload could not use the extra compute even when fully supported."}]},{"text":[{"text":"The model is too small to need the extra compute; a larger model would have shown the speed-up."}],"correct":false,"tempting":[{"text":"Backwards: a larger model has more bytes to read per token and is even more bandwidth-bound, so it would show the same non-result more starkly."}]},{"text":[{"text":"Each generated token reads every weight once, so decode is bounded by bandwidth, which did not change."}],"correct":true,"tempting":null},{"text":[{"text":"The key-value cache dominates memory traffic at any realistic prompt length, and its size depends only on the context, so no change to the GPU can affect generation speed."}],"correct":false,"tempting":[{"text":"Right that the cache adds traffic, wrong in every other clause: at ordinary lengths the weights dominate, and a higher-bandwidth GPU would speed up reading both."}]}],"order":[3,1,0,2],"why":null},"prev":{"slug":"language-imbalance","title":"Why it is worse in Hindi, and costs more","minutes":9},"next":{"slug":"will-it-fit","title":"Will it fit: the memory budget of a running model","minutes":9}}}