{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"batching-and-throughput","title":"Throughput and latency are different problems","url":"https://addaly.com/learn/hugging-face/batching-and-throughput","position":16,"of":75,"minutes":9,"answer":"Batching trades latency for throughput by amortising weight reads, so it pays on GPUs and often not on CPUs; and because a generation batch runs until its slowest member finishes.","keyIdea":"Batching trades latency for throughput by amortising weight reads, so it pays on GPUs and often not on CPUs; and because a generation batch runs until its slowest member finishes, sorting by length and watching the KV cache matter more than raising the batch size."},"body":{"format":"markdown","text":"## Decide which number you are optimising\n\n**Latency** is how long one person waits. **Throughput** is how many items you finish per minute. Batching improves the second by making the first worse, and almost every confused performance discussion is two people optimising different ones.\n\n- Interactive chat: latency. Time to first token is what a user perceives.\n- Overnight classification of 400,000 rows: throughput. Nobody is waiting.\n- A queue behind a web request: both, resolved by a deadline.\n\n## Why batching helps at all\n\nA forward pass reads every weight from memory. On a GPU, moving 14 GB of weights costs far more time than the arithmetic does, so a batch of one is mostly the hardware waiting for memory. Push 32 items through together and the weights are read once for all of them. The arithmetic grows with the batch; the memory traffic barely does.\n\nThat is why batch size 32 is often nowhere near 32 times slower than batch size 1 on a GPU: five to ten times is typical for small models.\n\nOn a CPU the picture is different. There is no comparable memory-bandwidth headroom, and the cores are already saturated by one item. Batching on CPU gives modest gains and sometimes none. Measure rather than assume; this is the single most common false transfer from GPU tutorials to laptop reality.\n\n## Batching with the pipeline\n\n```python\npipe = pipeline(\"text-classification\", model=name, device=0, batch_size=32)\nresults = pipe(list_of_texts)\n```\n\nPass a list, not a loop. Better, pass a `Dataset` so items stream rather than all sitting in memory:\n\n```python\nfrom datasets import Dataset\nds = Dataset.from_dict({\"text\": list_of_texts})\nfor r in pipe(KeyDataset(ds, \"text\"), batch_size=32):\n    ...\n```\n\n## The padding tax, measured\n\nBatching sorted by length is the highest-return four lines in this course:\n\n```python\norder = sorted(range(len(texts)), key=lambda i: len(texts[i]))\nsorted_texts = [texts[i] for i in order]\n# run in batches, then restore original order with `order`\n```\n\nOn mixed-length support tickets (a few words to a few thousand) this routinely doubles throughput, because a batch no longer runs at the length of its longest member. Restoring the original order afterwards is the part people forget, and it produces a silent mislabelling of every row.\n\n## Generation is a different shape\n\nClassification is one forward pass. Generation is one pass per output token, and the batch finishes when its **slowest** member finishes. Batch eight prompts where seven produce 20 tokens and one produces 500, and you paid for 500 steps on all eight.\n\nThe KV cache is what makes each step cheap: attention keys and values for tokens already processed are kept rather than recomputed. It also costs memory that grows with batch size times sequence length, and it is why a batch that fits at 512 tokens runs out of memory at 4,096. When you see out-of-memory partway through generation rather than at load, the cache is the reason.\n\n## Measuring honestly\n\n```python\nimport time, torch\n\n# warm-up: the first call compiles kernels and allocates\n_ = pipe(texts[:8])\n\ntorch.cuda.synchronize()          # GPU work is asynchronous\nstart = time.perf_counter()\n_ = pipe(texts, batch_size=32)\ntorch.cuda.synchronize()\nprint(len(texts) / (time.perf_counter() - start), \"items/sec\")\n```\n\nThree things people get wrong here, all of which produce numbers that are too good:\n\n1. **No warm-up.** The first batch includes one-off costs and is two to ten times slower; excluding it is right, forgetting it is wrong in the other direction.\n2. **No synchronise.** CUDA calls return before the work is done. Without the sync you are timing how fast Python can queue work.\n3. **Reporting the mean.** A mean latency of 300 ms with a 99th percentile of 4 seconds is a system some users experience as broken. Report the tail.\n\n## When to stop tuning and change tool\n\nFor serving many concurrent generation requests, a plain transformers loop is the wrong shape however well you tune it. vLLM, text-generation-inference and llama.cpp's server implement continuous batching (new requests join a running batch instead of waiting for it) and paged attention, which stops the KV cache fragmenting memory. The gain is typically several times, not a few per cent.\n\nFor batch scoring of a fixed list, a sorted transformers loop is fine and needs no new dependency.\n\n## Do this now\n\nTime 200 mixed-length inputs three ways: one at a time, batched unsorted, batched sorted by length. Write the three numbers down. The gap between the second and third is free and most people never collect it."},"check":{"question":"A batch job of 50,000 short texts and a few very long ones runs far slower than the per-item timings suggest. Batch size is already 64. What is the most likely cause?","options":[{"text":"The batch size exceeds what the GPU can hold, so the framework is silently splitting batches and paying the overhead twice.","correct":false,"tempting":"Invents silent splitting; exceeding memory raises an out-of-memory error rather than degrading quietly."},{"text":"Short texts finish early but their results are held until the batch completes, so wall-clock time reflects queueing rather than compute.","correct":false,"tempting":"Describes a scheduling effect that does not change total compute, which is what a throughput shortfall measures."},{"text":"Every batch runs at the length of its longest member, so a few long items inflate the cost of the short ones padded beside them.","correct":true},{"text":"The tokenizer is the bottleneck at this volume, since fast tokenizers do not parallelise across a batch.","correct":false,"tempting":"Fast tokenizers do parallelise, and tokenization is rarely dominant against model forward passes."}]},"prev":{"slug":"chat-templates","title":"The chat template is not optional","url":"https://addaly.com/learn/hugging-face/chat-templates"},"next":{"slug":"streaming-and-stopping","title":"Showing tokens as they arrive, and making them stop","url":"https://addaly.com/learn/hugging-face/streaming-and-stopping"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Throughput and latency are different problems', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/batching-and-throughput (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Throughput and latency are different problems","courseTitle":"Hugging Face, End to End","module":{"index":2,"slug":"transformers-from-text-to-text","title":"Transformers, from characters to output"},"answer":null,"keyIdea":"Batching trades latency for throughput by amortising weight reads, so it pays on GPUs and often not on CPUs; and because a generation batch runs until its slowest member finishes, sorting by length and watching the KV cache matter more than raising the batch size.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"decide-which-number-you-are-optimising","runs":[{"text":"Decide which number you are optimising"}]},{"t":"p","runs":[{"text":"Latency","b":true},{"text":" is how long one person waits. "},{"text":"Throughput","b":true},{"text":" is how many items you finish per minute. Batching improves the second by making the first worse, and almost every confused performance discussion is two people optimising different ones."}]},{"t":"ul","items":[[{"text":"Interactive chat: latency. Time to first token is what a user perceives."}],[{"text":"Overnight classification of 400,000 rows: throughput. Nobody is waiting."}],[{"text":"A queue behind a web request: both, resolved by a deadline."}]]},{"t":"h2","id":"why-batching-helps-at-all","runs":[{"text":"Why batching helps at all"}]},{"t":"p","runs":[{"text":"A forward pass reads every weight from memory. On a GPU, moving 14 GB of weights costs far more time than the arithmetic does, so a batch of one is mostly the hardware waiting for memory. Push 32 items through together and the weights are read once for all of them. The arithmetic grows with the batch; the memory traffic barely does."}]},{"t":"p","runs":[{"text":"That is why batch size 32 is often nowhere near 32 times slower than batch size 1 on a GPU: five to ten times is typical for small models."}]},{"t":"p","runs":[{"text":"On a CPU the picture is different. There is no comparable memory-bandwidth headroom, and the cores are already saturated by one item. Batching on CPU gives modest gains and sometimes none. Measure rather than assume; this is the single most common false transfer from GPU tutorials to laptop reality."}]},{"t":"h2","id":"batching-with-the-pipeline","runs":[{"text":"Batching with the pipeline"}]},{"t":"code","lang":"python","label":"Python","text":"pipe = pipeline(\"text-classification\", model=name, device=0, batch_size=32)\nresults = pipe(list_of_texts)"},{"t":"p","runs":[{"text":"Pass a list, not a loop. Better, pass a "},{"text":"Dataset","code":true},{"text":" so items stream rather than all sitting in memory:"}]},{"t":"code","lang":"python","label":"Python","text":"from datasets import Dataset\nds = Dataset.from_dict({\"text\": list_of_texts})\nfor r in pipe(KeyDataset(ds, \"text\"), batch_size=32):\n    ..."},{"t":"h2","id":"the-padding-tax-measured","runs":[{"text":"The padding tax, measured"}]},{"t":"p","runs":[{"text":"Batching sorted by length is the highest-return four lines in this course:"}]},{"t":"code","lang":"python","label":"Python","text":"order = sorted(range(len(texts)), key=lambda i: len(texts[i]))\nsorted_texts = [texts[i] for i in order]\n# run in batches, then restore original order with `order`"},{"t":"p","runs":[{"text":"On mixed-length support tickets (a few words to a few thousand) this routinely doubles throughput, because a batch no longer runs at the length of its longest member. Restoring the original order afterwards is the part people forget, and it produces a silent mislabelling of every row."}]},{"t":"h2","id":"generation-is-a-different-shape","runs":[{"text":"Generation is a different shape"}]},{"t":"p","runs":[{"text":"Classification is one forward pass. Generation is one pass per output token, and the batch finishes when its "},{"text":"slowest","b":true},{"text":" member finishes. Batch eight prompts where seven produce 20 tokens and one produces 500, and you paid for 500 steps on all eight."}]},{"t":"p","runs":[{"text":"The KV cache is what makes each step cheap: attention keys and values for tokens already processed are kept rather than recomputed. It also costs memory that grows with batch size times sequence length, and it is why a batch that fits at 512 tokens runs out of memory at 4,096. When you see out-of-memory partway through generation rather than at load, the cache is the reason."}]},{"t":"h2","id":"measuring-honestly","runs":[{"text":"Measuring honestly"}]},{"t":"code","lang":"python","label":"Python","text":"import time, torch\n\n# warm-up: the first call compiles kernels and allocates\n_ = pipe(texts[:8])\n\ntorch.cuda.synchronize()          # GPU work is asynchronous\nstart = time.perf_counter()\n_ = pipe(texts, batch_size=32)\ntorch.cuda.synchronize()\nprint(len(texts) / (time.perf_counter() - start), \"items/sec\")"},{"t":"p","runs":[{"text":"Three things people get wrong here, all of which produce numbers that are too good:"}]},{"t":"ol","start":1,"items":[[{"text":"No warm-up.","b":true},{"text":" The first batch includes one-off costs and is two to ten times slower; excluding it is right, forgetting it is wrong in the other direction."}],[{"text":"No synchronise.","b":true},{"text":" CUDA calls return before the work is done. Without the sync you are timing how fast Python can queue work."}],[{"text":"Reporting the mean.","b":true},{"text":" A mean latency of 300 ms with a 99th percentile of 4 seconds is a system some users experience as broken. Report the tail."}]]},{"t":"h2","id":"when-to-stop-tuning-and-change-tool","runs":[{"text":"When to stop tuning and change tool"}]},{"t":"p","runs":[{"text":"For serving many concurrent generation requests, a plain transformers loop is the wrong shape however well you tune it. vLLM, text-generation-inference and llama.cpp’s server implement continuous batching (new requests join a running batch instead of waiting for it) and paged attention, which stops the KV cache fragmenting memory. The gain is typically several times, not a few per cent."}]},{"t":"p","runs":[{"text":"For batch scoring of a fixed list, a sorted transformers loop is fine and needs no new dependency."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Time 200 mixed-length inputs three ways: one at a time, batched unsorted, batched sorted by length. Write the three numbers down. The gap between the second and third is free and most people never collect it."}]}],"check":{"question":[{"text":"A batch job of 50,000 short texts and a few very long ones runs far slower than the per-item timings suggest. Batch size is already 64. What is the most likely cause?"}],"options":[{"text":[{"text":"The batch size exceeds what the GPU can hold, so the framework is silently splitting batches and paying the overhead twice."}],"correct":false,"tempting":[{"text":"Invents silent splitting; exceeding memory raises an out-of-memory error rather than degrading quietly."}]},{"text":[{"text":"Short texts finish early but their results are held until the batch completes, so wall-clock time reflects queueing rather than compute."}],"correct":false,"tempting":[{"text":"Describes a scheduling effect that does not change total compute, which is what a throughput shortfall measures."}]},{"text":[{"text":"Every batch runs at the length of its longest member, so a few long items inflate the cost of the short ones padded beside them."}],"correct":true,"tempting":null},{"text":[{"text":"The tokenizer is the bottleneck at this volume, since fast tokenizers do not parallelise across a batch."}],"correct":false,"tempting":[{"text":"Fast tokenizers do parallelise, and tokenization is rarely dominant against model forward passes."}]}],"order":[1,3,2,0],"why":null},"prev":{"slug":"chat-templates","title":"The chat template is not optional","minutes":9},"next":{"slug":"streaming-and-stopping","title":"Showing tokens as they arrive, and making them stop","minutes":8}}}