Continuous batching, and the queue underneath it

Static batching wastes most of the machine

The naive way to batch is to collect eight requests, run them together, and return all eight when the last one finishes. The trouble is that generation lengths differ enormously. If seven requests want 40 tokens and one wants 900, the seven finish early and their slots sit idle for the remaining 860 steps, computing padding. Utilisation collapses towards the reciprocal of the length variance, which on real traffic is dreadful.

Continuous batching (also called iteration-level scheduling) makes the batch a living thing. After every single decoding step the scheduler asks: has anything finished, and is anything waiting? Finished sequences are retired and their slots given to waiting requests immediately.

The consequences are large and worth stating plainly. A new request does not wait for the current batch to end, only for the current step, which is milliseconds. The batch stays full, so the expensive weight read is amortised across as many sequences as possible. And a very long generation no longer blocks everything behind it.

This is the single biggest reason a purpose-built server outperforms a loop around a model, and it is not a small factor: it is often five to ten times on realistic traffic.

What the scheduler is actually deciding

At each step the engine holds two queues: running sequences and waiting ones. Admitting a waiting request means allocating cache blocks for its prompt and adding it to the batch. The constraints are the memory available for cache blocks and a configured cap on the batch.

When memory runs short, the engine has to make a choice. It can stop admitting new work, which grows the queue and raises time to first token. Or it can preempt a running sequence (evict its cache and put it back in the waiting queue) which means recomputing its prefill later. vLLM does the latter under pressure and logs it. Frequent preemption is a clear signal that --gpu-memory-utilization is too low or --max-model-len too high for your traffic, and it shows up to users as unpredictable latency rather than as an error.

Prefill and decode compete

A subtlety that explains a common complaint. Decode steps are short and frequent; a prefill of 20,000 tokens is one long operation. If the scheduler runs that prefill as a single unit, every other user’s stream pauses for the duration. They experience a stutter mid-sentence and report the server as unreliable.

Chunked prefill splits the prompt and interleaves the pieces with decoding steps, trading a slightly slower prefill for far lower latency variance across everyone else. On busy multi-user servers it should generally be on.

vllm serve <model> \
  --max-num-seqs 128 \
  --max-num-batched-tokens 4096 \
  --enable-chunked-prefill

--max-num-seqs caps concurrent sequences. --max-num-batched-tokens caps the tokens processed in one step, which is the knob that governs the prefill-versus-decode balance: larger values favour prefill throughput, smaller ones favour smooth streaming.

Where the gains stop

Throughput against batch size rises steeply, then bends, then flattens. Three reasons, in the order they bite:

1. The memory-bound to compute-bound transition. At batch one, the weight read produces one token; at batch 32 it produces 32. Once the arithmetic intensity reaches the hardware’s balance point, you are compute-limited and further batching adds little.
2. Cache capacity. Every sequence needs its own KV cache. At long context you run out of blocks long before you run out of compute.
3. Attention does not batch as cleanly as the feed-forward layers. Each sequence attends over its own distinct cache, so that part of the work scales with the batch rather than being amortised.

Measure your own curve. Run the same load at concurrency 1, 4, 16, 32 and 64, and plot aggregate throughput and p95 latency. The point where throughput flattens while latency keeps climbing is your operating limit, and it is a property of your model, your context length and your card, not something to read from a benchmark.

In llama.cpp

llama-server -m model.gguf -c 32768 --parallel 4 --cont-batching

Continuous batching is present and works. Aggregate throughput at high concurrency remains well below the purpose-built servers, because the kernels and the scheduler were built for a different priority. For a handful of users it is entirely adequate, and remember that -c is the total budget divided across the slots.