Throughput and latency are different problems

Decide which number you are optimising

Latency is how long one person waits. Throughput is how many items you finish per minute. Batching improves the second by making the first worse, and almost every confused performance discussion is two people optimising different ones.

• Interactive chat: latency. Time to first token is what a user perceives.
• Overnight classification of 400,000 rows: throughput. Nobody is waiting.
• A queue behind a web request: both, resolved by a deadline.

Why batching helps at all

A forward pass reads every weight from memory. On a GPU, moving 14 GB of weights costs far more time than the arithmetic does, so a batch of one is mostly the hardware waiting for memory. Push 32 items through together and the weights are read once for all of them. The arithmetic grows with the batch; the memory traffic barely does.

That is why batch size 32 is often nowhere near 32 times slower than batch size 1 on a GPU: five to ten times is typical for small models.

On a CPU the picture is different. There is no comparable memory-bandwidth headroom, and the cores are already saturated by one item. Batching on CPU gives modest gains and sometimes none. Measure rather than assume; this is the single most common false transfer from GPU tutorials to laptop reality.

Batching with the pipeline

pipe = pipeline("text-classification", model=name, device=0, batch_size=32)
results = pipe(list_of_texts)

Pass a list, not a loop. Better, pass a Dataset so items stream rather than all sitting in memory:

from datasets import Dataset
ds = Dataset.from_dict({"text": list_of_texts})
for r in pipe(KeyDataset(ds, "text"), batch_size=32):
    ...

The padding tax, measured

Batching sorted by length is the highest-return four lines in this course:

order = sorted(range(len(texts)), key=lambda i: len(texts[i]))
sorted_texts = [texts[i] for i in order]
# run in batches, then restore original order with `order`

On mixed-length support tickets (a few words to a few thousand) this routinely doubles throughput, because a batch no longer runs at the length of its longest member. Restoring the original order afterwards is the part people forget, and it produces a silent mislabelling of every row.

Generation is a different shape

Classification is one forward pass. Generation is one pass per output token, and the batch finishes when its slowest member finishes. Batch eight prompts where seven produce 20 tokens and one produces 500, and you paid for 500 steps on all eight.

The KV cache is what makes each step cheap: attention keys and values for tokens already processed are kept rather than recomputed. It also costs memory that grows with batch size times sequence length, and it is why a batch that fits at 512 tokens runs out of memory at 4,096. When you see out-of-memory partway through generation rather than at load, the cache is the reason.

Measuring honestly

import time, torch

# warm-up: the first call compiles kernels and allocates
_ = pipe(texts[:8])

torch.cuda.synchronize()          # GPU work is asynchronous
start = time.perf_counter()
_ = pipe(texts, batch_size=32)
torch.cuda.synchronize()
print(len(texts) / (time.perf_counter() - start), "items/sec")

Three things people get wrong here, all of which produce numbers that are too good:

1. No warm-up. The first batch includes one-off costs and is two to ten times slower; excluding it is right, forgetting it is wrong in the other direction.
2. No synchronise. CUDA calls return before the work is done. Without the sync you are timing how fast Python can queue work.
3. Reporting the mean. A mean latency of 300 ms with a 99th percentile of 4 seconds is a system some users experience as broken. Report the tail.

When to stop tuning and change tool

For serving many concurrent generation requests, a plain transformers loop is the wrong shape however well you tune it. vLLM, text-generation-inference and llama.cpp’s server implement continuous batching (new requests join a running batch instead of waiting for it) and paged attention, which stops the KV cache fragmenting memory. The gain is typically several times, not a few per cent.

For batch scoring of a fixed list, a sorted transformers loop is fine and needs no new dependency.

Do this now

Time 200 mixed-length inputs three ways: one at a time, batched unsorted, batched sorted by length. Write the three numbers down. The gap between the second and third is free and most people never collect it.