Serving a hundred people from one card

The idle machine, one more time

The bandwidth lesson left a GPU generating for one user using about a three-hundredth of its arithmetic capacity, because every decode step reads all the weights to produce one token. A server exists to fill the other 299 parts. The way it does that has changed twice since 2022, and each change is a mechanism you can reason about.

Static batching, and why it was abandoned

The first idea: wait until sixteen requests have arrived, run them together as one batch, return the results. Throughput rises sixteen-fold. Three things go wrong. Requests wait for the batch to fill. Every sequence in the batch is padded to the longest, wasting cache. And the batch finishes when its longest response finishes: a user who asked for one word waits for the user who asked for an essay. Static batching gave good throughput numbers and a dreadful experience, and no serious server uses it now.

Continuous batching

The fix, published in 2022 and standard by 2023, schedules at the level of the token step rather than the request. At each decode step the batch is whichever sequences are alive. A sequence that emits its stop token leaves the batch at that step; a request waiting in the queue joins at the next. There is no padding, no waiting for a batch to fill, and no short request held hostage by a long one.

It sounds obvious. It required rewriting how memory is managed, because a batch whose membership changes every few milliseconds cannot pre-allocate a fixed slab of cache per sequence.

Paged cache

The memory answer, from 2023, borrowed the idea of virtual memory. The key-value cache is allocated in fixed blocks of sixteen tokens or so, mapped through a table, instead of one contiguous region sized for the maximum possible length. Before this, servers reserved the maximum context for every sequence and wasted, in published measurements, sixty to eighty per cent of the cache on space that was never used. With paging the waste fell to a few per cent, which by the fitting lesson’s arithmetic is directly a multiple on how many users one card can hold.

Paging also gives module four’s prompt caching its mechanism on the server side: two sequences that share a prefix (the same system prompt, the same document) can point at the same physical blocks. The prefix is computed once and shared, not copied.

Chunked prefill

A new request with a 10,000-token prompt arrives. Prefill is compute-bound, and running it in one go would occupy the card for most of a second, during which every user mid-generation sees their stream freeze. Chunked prefill cuts the new prompt into pieces of a few hundred to a couple of thousand tokens and interleaves them with decode steps. Everyone else’s inter-token latency stays smooth; the newcomer’s time to first token is slightly longer. It is a scheduling choice about whose latency to protect, and current engines default to protecting the users already talking.

The curve you are operating on

Add concurrent users and watch two numbers. Aggregate throughput (tokens a second across everyone) rises almost linearly at first, because each added sequence uses compute the card had idle. Each user’s inter-token latency rises slowly. Then, somewhere around the crossover from the bandwidth lesson (hundreds of sequences for a small model, fewer for a large one, fewer still with long contexts) compute saturates. Throughput flattens and latency climbs steeply.

Concretely: a data-centre card serving an 8-billion model in bf16 can deliver several thousand output tokens a second in aggregate at around a hundred concurrent users, with each of them seeing thirty to fifty tokens a second. One user alone saw perhaps 150. The card is doing thirty times the work for a third of the per-user speed, and that trade is the entire economics of the API you pay for.

The word for the number you actually want is goodput: throughput counted only for requests that met a latency target, say first token under a second and under fifty milliseconds between tokens. You choose an operating point on the curve by deciding what target you owe your users, then load-testing until you find the concurrency at which you stop meeting it. That is the card’s capacity, and it is a measurement, not a spec.

The knobs, and one interaction

Every engine exposes roughly the same settings under different names: a maximum number of sequences in the batch, a maximum number of tokens processed per step, the fraction of GPU memory to reserve, the maximum context per sequence, and a switch for prefix caching. The fitting lesson’s arithmetic tells you what each costs in cache.

One interaction is worth knowing. Speculative decoding, from module three, uses idle compute to verify drafted tokens. Under heavy batching there is no idle compute: batching has already spent it. So speculation helps a lightly loaded server and can slow a saturated one, and engines now switch it off dynamically for that reason.

Where it is going

Prefill and decode are different regimes (compute-bound and bandwidth-bound) so the largest deployments now run them on separate pools of GPUs and ship the finished cache from one to the other. It adds a transfer and removes the interference between the two, and it is the current frontier of serving design rather than something you need on day one.

The free path, and the measurement

vLLM and SGLang are open source and are what many commercial APIs run underneath. llama-server does continuous batching too, at the scale of a handful of users. Rent a card by the hour, load your model, and run a load test with a realistic mix of prompt and output lengths, not the same short prompt repeated, which flatters everything through prefix sharing. Plot throughput and the 95th percentile of inter-token latency against concurrency. The knee in that plot is the number you were looking for.