{"course":{"slug":"open-models","title":"The Open Model Ecosystem","url":"https://addaly.com/learn/open-models","level":"intermediate","track":"deep"},"lesson":{"slug":"serving-many-users","title":"Serving many users at once","url":"https://addaly.com/learn/open-models/serving-many-users","position":37,"of":71,"minutes":9,"answer":"Continuous batching and paged attention let one GPU serve many sequences because weights are read once per step and cache memory is allocated in blocks rather than reserved per request.","keyIdea":"Continuous batching and paged attention let one GPU serve many sequences because weights are read once per step and cache memory is allocated in blocks rather than reserved per request, trading a little per-user latency for several times the total throughput."},"body":{"format":"markdown","text":"## Why a different engine exists\n\nEverything so far optimised one conversation. Serving thirty at once is a different problem, and the tool for it is vLLM, or SGLang, or Hugging Face's TGI, or TensorRT-LLM on Nvidia hardware. The ideas below are shared; vLLM is the common starting point.\n\n```bash\npip install vllm\nvllm serve Qwen/Qwen3-8B   --max-model-len 8192   --gpu-memory-utilization 0.90   --max-num-seqs 64   --enable-prefix-caching\n```\n\nSame OpenAI-compatible endpoint as everything else, so your client code does not change.\n\n## Continuous batching\n\nA naive server batches requests together, runs them to completion, and starts the next batch. Requests finish at different times, so the batch runs at the speed of its slowest member and the freed capacity sits idle.\n\nContinuous batching schedules at the level of a single decoding step. When one sequence finishes, a waiting request joins the batch immediately. Nothing waits for a batch boundary. Combined with the bandwidth argument from the previous lesson (the weights are read once per step regardless of how many sequences are in the batch) this is where the throughput comes from.\n\nThe numbers are dramatic. A single stream on a given GPU might generate 100 tokens per second; the same GPU serving 32 concurrent sequences might produce 2,000 tokens per second in aggregate, with each individual user seeing perhaps 60. Total work goes up manyfold; each person's experience degrades a little.\n\n> Figure: The same GPU, two scheduling decisions. One stream: Weights read once per token, for one sequence; About 100 tokens per second in total; That one user gets all 100; Cache reserved for the maximum context; Capacity idle between requests. Thirty-two streams, continuous batching: Weights read once per step, for all 32; About 2,000 tokens per second in total; Each user sees perhaps 60; Paged attention hands out blocks as needed; A finished sequence is replaced immediately. Total work rises many times over and each person's experience degrades a little. That is the whole economic basis of hosted inference, and the reason llama.cpp, which optimises one conversation beautifully, is the wrong engine for thirty.\n\n## Paged attention\n\nThe KV cache was the memory problem in an earlier lesson. A naive implementation reserves the maximum context length for every request, so a request that turns out to be 300 tokens long occupies space for 8,192.\n\nPaged attention allocates the cache in fixed-size blocks, held in a table, exactly as an operating system pages memory. A sequence uses the blocks it needs and returns them when it finishes. Fragmentation and over-reservation mostly disappear, and the number of concurrent sequences the same card supports rises several times over. This is the single largest reason vLLM outperforms simpler servers.\n\nIt also makes two features cheap. **Prefix caching** shares the blocks of a common prompt prefix across requests: if every request begins with the same 2,000-token system prompt, it is computed once. On a chat product this is often the biggest single saving available. And **copy-on-write** sharing lets several samples from the same prompt share the prompt's blocks.\n\n## The dials\n\n**`--max-model-len`** is the context limit, and it is a capacity decision as much as a capability one. Serving a 128k model at 8k lets you hold many more users.\n\n**`--gpu-memory-utilization`** is the fraction of the card vLLM claims, default 0.9. Raise it if nothing else runs on the card; lower it if something does.\n\n**`--max-num-seqs`** caps concurrent sequences. Higher gives more throughput and worse tail latency.\n\n**`--tensor-parallel-size N`** splits the model across N GPUs, for models too large for one card. It costs communication between cards on every layer, so use it because you must, not to speed up a model that already fits.\n\n**Chunked prefill** interleaves long prompt processing with ongoing decoding, so one user pasting a huge document does not stall everybody else's generation. On mixed traffic it is the difference between a smooth service and periodic freezes.\n\n## Throughput and latency are the same dial\n\nYou cannot maximise both. A larger batch means more total tokens per second and a longer wait for each individual token. Decide which one your product is:\n\n- **Interactive chat.** Time to first token is what people feel. Keep batches moderate, enable prefix caching, and consider streaming so the wait is hidden.\n- **Batch processing.** Nobody is watching. Push concurrency until memory limits it, and take the throughput.\n- **Mixed.** Run two deployments rather than one compromise, if you can afford the memory. Two servers with different settings usually beat one with settings that suit neither.\n\n## When not to reach for this\n\nvLLM wants a GPU, expects Linux, and its advantages appear under concurrency. For one person on a laptop it is slower to set up and no faster in use than llama.cpp, and on CPU-only hardware llama.cpp is the better engine. Choose by your concurrency, not by which project is more discussed."},"check":{"question":"A chat product with a 2,000-token system prompt on every request moves from a naive server to vLLM with prefix caching and sees a large drop in cost per request. Which mechanism explains most of it?","options":[{"text":"Quantisation applied automatically by vLLM, which reduces the memory read per token.","correct":false,"tempting":"vLLM does not quantise a model unless asked, and quantisation would affect decoding rather than the shared prompt."},{"text":"Tensor parallelism, which spreads the shared prompt across GPUs.","correct":false,"tempting":"Tensor parallelism splits a model too large for one card and adds communication cost; it does not deduplicate repeated prompts."},{"text":"Continuous batching alone, since more requests fit in each batch.","correct":false,"tempting":"Continuous batching does raise throughput, but it does not stop the identical 2,000-token prefix being recomputed for every request."},{"text":"The shared prefix is computed once and its cache blocks are reused across requests, so the repeated prefill work disappears.","correct":true}]},"prev":{"slug":"predicting-tokens-per-second","title":"Predicting speed before you download","url":"https://addaly.com/learn/open-models/predicting-tokens-per-second"},"next":{"slug":"free-compute","title":"Running all of this without money","url":"https://addaly.com/learn/open-models/free-compute"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Serving many users at once', in The Open Model Ecosystem. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/open-models/serving-many-users (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Serving many users at once","courseTitle":"The Open Model Ecosystem","module":{"index":4,"slug":"making-it-run","title":"Making it run on the machine you have"},"answer":null,"keyIdea":"Continuous batching and paged attention let one GPU serve many sequences because weights are read once per step and cache memory is allocated in blocks rather than reserved per request, trading a little per-user latency for several times the total throughput.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"left":{"title":"One stream","items":["Weights read once per token, for one sequence","About 100 tokens per second in total","That one user gets all 100","Cache reserved for the maximum context","Capacity idle between requests"]},"type":"compare","right":{"title":"Thirty-two streams, continuous batching","items":["Weights read once per step, for all 32","About 2,000 tokens per second in total","Each user sees perhaps 60","Paged attention hands out blocks as needed","A finished sequence is replaced immediately"]},"title":"The same GPU, two scheduling decisions","caption":"Total work rises many times over and each person’s experience degrades a little. That is the whole economic basis of hosted inference, and the reason llama.cpp, which optimises one conversation beautifully, is the wrong engine for thirty."},"alt":"Figure: The same GPU, two scheduling decisions. One stream: Weights read once per token, for one sequence; About 100 tokens per second in total; That one user gets all 100; Cache reserved for the maximum context; Capacity idle between requests. Thirty-two streams, continuous batching: Weights read once per step, for all 32; About 2,000 tokens per second in total; Each user sees perhaps 60; Paged attention hands out blocks as needed; A finished sequence is replaced immediately. Total work rises many times over and each person’s experience degrades a little. That is the whole economic basis of hosted inference, and the reason llama.cpp, which optimises one conversation beautifully, is the wrong engine for thirty."}},"blocks":[{"t":"h2","id":"why-a-different-engine-exists","runs":[{"text":"Why a different engine exists"}]},{"t":"p","runs":[{"text":"Everything so far optimised one conversation. Serving thirty at once is a different problem, and the tool for it is vLLM, or SGLang, or Hugging Face’s TGI, or TensorRT-LLM on Nvidia hardware. The ideas below are shared; vLLM is the common starting point."}]},{"t":"code","lang":"bash","label":"Bash","text":"pip install vllm\nvllm serve Qwen/Qwen3-8B   --max-model-len 8192   --gpu-memory-utilization 0.90   --max-num-seqs 64   --enable-prefix-caching"},{"t":"p","runs":[{"text":"Same OpenAI-compatible endpoint as everything else, so your client code does not change."}]},{"t":"h2","id":"continuous-batching","runs":[{"text":"Continuous batching"}]},{"t":"p","runs":[{"text":"A naive server batches requests together, runs them to completion, and starts the next batch. Requests finish at different times, so the batch runs at the speed of its slowest member and the freed capacity sits idle."}]},{"t":"p","runs":[{"text":"Continuous batching schedules at the level of a single decoding step. When one sequence finishes, a waiting request joins the batch immediately. Nothing waits for a batch boundary. Combined with the bandwidth argument from the previous lesson (the weights are read once per step regardless of how many sequences are in the batch) this is where the throughput comes from."}]},{"t":"p","runs":[{"text":"The numbers are dramatic. A single stream on a given GPU might generate 100 tokens per second; the same GPU serving 32 concurrent sequences might produce 2,000 tokens per second in aggregate, with each individual user seeing perhaps 60. Total work goes up manyfold; each person’s experience degrades a little."}]},{"t":"h2","id":"paged-attention","runs":[{"text":"Paged attention"}]},{"t":"p","runs":[{"text":"The KV cache was the memory problem in an earlier lesson. A naive implementation reserves the maximum context length for every request, so a request that turns out to be 300 tokens long occupies space for 8,192."}]},{"t":"p","runs":[{"text":"Paged attention allocates the cache in fixed-size blocks, held in a table, exactly as an operating system pages memory. A sequence uses the blocks it needs and returns them when it finishes. Fragmentation and over-reservation mostly disappear, and the number of concurrent sequences the same card supports rises several times over. This is the single largest reason vLLM outperforms simpler servers."}]},{"t":"p","runs":[{"text":"It also makes two features cheap. "},{"text":"Prefix caching","b":true},{"text":" shares the blocks of a common prompt prefix across requests: if every request begins with the same 2,000-token system prompt, it is computed once. On a chat product this is often the biggest single saving available. And "},{"text":"copy-on-write","b":true},{"text":" sharing lets several samples from the same prompt share the prompt’s blocks."}]},{"t":"h2","id":"the-dials","runs":[{"text":"The dials"}]},{"t":"p","runs":[{"text":"--max-model-len","b":true,"code":true},{"text":" is the context limit, and it is a capacity decision as much as a capability one. Serving a 128k model at 8k lets you hold many more users."}]},{"t":"p","runs":[{"text":"--gpu-memory-utilization","b":true,"code":true},{"text":" is the fraction of the card vLLM claims, default 0.9. Raise it if nothing else runs on the card; lower it if something does."}]},{"t":"p","runs":[{"text":"--max-num-seqs","b":true,"code":true},{"text":" caps concurrent sequences. Higher gives more throughput and worse tail latency."}]},{"t":"p","runs":[{"text":"--tensor-parallel-size N","b":true,"code":true},{"text":" splits the model across N GPUs, for models too large for one card. It costs communication between cards on every layer, so use it because you must, not to speed up a model that already fits."}]},{"t":"p","runs":[{"text":"Chunked prefill","b":true},{"text":" interleaves long prompt processing with ongoing decoding, so one user pasting a huge document does not stall everybody else’s generation. On mixed traffic it is the difference between a smooth service and periodic freezes."}]},{"t":"h2","id":"throughput-and-latency-are-the-same-dial","runs":[{"text":"Throughput and latency are the same dial"}]},{"t":"p","runs":[{"text":"You cannot maximise both. A larger batch means more total tokens per second and a longer wait for each individual token. Decide which one your product is:"}]},{"t":"ul","items":[[{"text":"Interactive chat.","b":true},{"text":" Time to first token is what people feel. Keep batches moderate, enable prefix caching, and consider streaming so the wait is hidden."}],[{"text":"Batch processing.","b":true},{"text":" Nobody is watching. Push concurrency until memory limits it, and take the throughput."}],[{"text":"Mixed.","b":true},{"text":" Run two deployments rather than one compromise, if you can afford the memory. Two servers with different settings usually beat one with settings that suit neither."}]]},{"t":"h2","id":"when-not-to-reach-for-this","runs":[{"text":"When not to reach for this"}]},{"t":"p","runs":[{"text":"vLLM wants a GPU, expects Linux, and its advantages appear under concurrency. For one person on a laptop it is slower to set up and no faster in use than llama.cpp, and on CPU-only hardware llama.cpp is the better engine. Choose by your concurrency, not by which project is more discussed."}]}],"check":{"question":[{"text":"A chat product with a 2,000-token system prompt on every request moves from a naive server to vLLM with prefix caching and sees a large drop in cost per request. Which mechanism explains most of it?"}],"options":[{"text":[{"text":"Quantisation applied automatically by vLLM, which reduces the memory read per token."}],"correct":false,"tempting":[{"text":"vLLM does not quantise a model unless asked, and quantisation would affect decoding rather than the shared prompt."}]},{"text":[{"text":"Tensor parallelism, which spreads the shared prompt across GPUs."}],"correct":false,"tempting":[{"text":"Tensor parallelism splits a model too large for one card and adds communication cost; it does not deduplicate repeated prompts."}]},{"text":[{"text":"Continuous batching alone, since more requests fit in each batch."}],"correct":false,"tempting":[{"text":"Continuous batching does raise throughput, but it does not stop the identical 2,000-token prefix being recomputed for every request."}]},{"text":[{"text":"The shared prefix is computed once and its cache blocks are reused across requests, so the repeated prefill work disappears."}],"correct":true,"tempting":null}],"order":[2,3,0,1],"why":null},"prev":{"slug":"predicting-tokens-per-second","title":"Predicting speed before you download","minutes":9},"next":{"slug":"free-compute","title":"Running all of this without money","minutes":8}}}