Serving many users at once

Why a different engine exists

Everything so far optimised one conversation. Serving thirty at once is a different problem, and the tool for it is vLLM, or SGLang, or Hugging Face’s TGI, or TensorRT-LLM on Nvidia hardware. The ideas below are shared; vLLM is the common starting point.

pip install vllm
vllm serve Qwen/Qwen3-8B   --max-model-len 8192   --gpu-memory-utilization 0.90   --max-num-seqs 64   --enable-prefix-caching

Same OpenAI-compatible endpoint as everything else, so your client code does not change.

Continuous batching

A naive server batches requests together, runs them to completion, and starts the next batch. Requests finish at different times, so the batch runs at the speed of its slowest member and the freed capacity sits idle.

Continuous batching schedules at the level of a single decoding step. When one sequence finishes, a waiting request joins the batch immediately. Nothing waits for a batch boundary. Combined with the bandwidth argument from the previous lesson (the weights are read once per step regardless of how many sequences are in the batch) this is where the throughput comes from.

The numbers are dramatic. A single stream on a given GPU might generate 100 tokens per second; the same GPU serving 32 concurrent sequences might produce 2,000 tokens per second in aggregate, with each individual user seeing perhaps 60. Total work goes up manyfold; each person’s experience degrades a little.

Paged attention

The KV cache was the memory problem in an earlier lesson. A naive implementation reserves the maximum context length for every request, so a request that turns out to be 300 tokens long occupies space for 8,192.

Paged attention allocates the cache in fixed-size blocks, held in a table, exactly as an operating system pages memory. A sequence uses the blocks it needs and returns them when it finishes. Fragmentation and over-reservation mostly disappear, and the number of concurrent sequences the same card supports rises several times over. This is the single largest reason vLLM outperforms simpler servers.

It also makes two features cheap. Prefix caching shares the blocks of a common prompt prefix across requests: if every request begins with the same 2,000-token system prompt, it is computed once. On a chat product this is often the biggest single saving available. And copy-on-write sharing lets several samples from the same prompt share the prompt’s blocks.

The dials

--max-model-len is the context limit, and it is a capacity decision as much as a capability one. Serving a 128k model at 8k lets you hold many more users.

--gpu-memory-utilization is the fraction of the card vLLM claims, default 0.9. Raise it if nothing else runs on the card; lower it if something does.

--max-num-seqs caps concurrent sequences. Higher gives more throughput and worse tail latency.

--tensor-parallel-size N splits the model across N GPUs, for models too large for one card. It costs communication between cards on every layer, so use it because you must, not to speed up a model that already fits.

Chunked prefill interleaves long prompt processing with ongoing decoding, so one user pasting a huge document does not stall everybody else’s generation. On mixed traffic it is the difference between a smooth service and periodic freezes.

Throughput and latency are the same dial

You cannot maximise both. A larger batch means more total tokens per second and a longer wait for each individual token. Decide which one your product is:

• Interactive chat. Time to first token is what people feel. Keep batches moderate, enable prefix caching, and consider streaming so the wait is hidden.
• Batch processing. Nobody is watching. Push concurrency until memory limits it, and take the throughput.
• Mixed. Run two deployments rather than one compromise, if you can afford the memory. Two servers with different settings usually beat one with settings that suit neither.

When not to reach for this

vLLM wants a GPU, expects Linux, and its advantages appear under concurrency. For one person on a laptop it is slower to set up and no faster in use than llama.cpp, and on CPU-only hardware llama.cpp is the better engine. Choose by your concurrency, not by which project is more discussed.