{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"serving-many-users","title":"Serving a hundred people from one card","url":"https://addaly.com/learn/how-llms-work/serving-many-users","position":63,"of":78,"minutes":9,"answer":"A server fills the compute that single-stream decode leaves idle by scheduling at the token step with a paged cache, so throughput rises almost free with concurrency until compute saturates.","keyIdea":"A server fills the compute that single-stream decode leaves idle by scheduling at the token step with a paged cache, so throughput rises almost free with concurrency until compute saturates, and the capacity of a card is the concurrency at which it stops meeting your latency target."},"body":{"format":"markdown","text":"## The idle machine, one more time\n\nThe bandwidth lesson left a GPU generating for one user using about a three-hundredth of its arithmetic capacity, because every decode step reads all the weights to produce one token. A server exists to fill the other 299 parts. The way it does that has changed twice since 2022, and each change is a mechanism you can reason about.\n\n## Static batching, and why it was abandoned\n\nThe first idea: wait until sixteen requests have arrived, run them together as one batch, return the results. Throughput rises sixteen-fold. Three things go wrong. Requests wait for the batch to fill. Every sequence in the batch is padded to the longest, wasting cache. And the batch finishes when its longest response finishes: a user who asked for one word waits for the user who asked for an essay. Static batching gave good throughput numbers and a dreadful experience, and no serious server uses it now.\n\n## Continuous batching\n\nThe fix, published in 2022 and standard by 2023, schedules at the level of the **token step** rather than the request. At each decode step the batch is whichever sequences are alive. A sequence that emits its stop token leaves the batch at that step; a request waiting in the queue joins at the next. There is no padding, no waiting for a batch to fill, and no short request held hostage by a long one.\n\nIt sounds obvious. It required rewriting how memory is managed, because a batch whose membership changes every few milliseconds cannot pre-allocate a fixed slab of cache per sequence.\n\n## Paged cache\n\nThe memory answer, from 2023, borrowed the idea of virtual memory. The key-value cache is allocated in fixed blocks of sixteen tokens or so, mapped through a table, instead of one contiguous region sized for the maximum possible length. Before this, servers reserved the maximum context for every sequence and wasted, in published measurements, sixty to eighty per cent of the cache on space that was never used. With paging the waste fell to a few per cent, which by the fitting lesson's arithmetic is directly a multiple on how many users one card can hold.\n\nPaging also gives module four's prompt caching its mechanism on the server side: two sequences that share a prefix (the same system prompt, the same document) can point at the same physical blocks. The prefix is computed once and shared, not copied.\n\n## Chunked prefill\n\nA new request with a 10,000-token prompt arrives. Prefill is compute-bound, and running it in one go would occupy the card for most of a second, during which every user mid-generation sees their stream freeze. Chunked prefill cuts the new prompt into pieces of a few hundred to a couple of thousand tokens and interleaves them with decode steps. Everyone else's inter-token latency stays smooth; the newcomer's time to first token is slightly longer. It is a scheduling choice about whose latency to protect, and current engines default to protecting the users already talking.\n\n## The curve you are operating on\n\nAdd concurrent users and watch two numbers. Aggregate throughput (tokens a second across everyone) rises almost linearly at first, because each added sequence uses compute the card had idle. Each user's inter-token latency rises slowly. Then, somewhere around the crossover from the bandwidth lesson (hundreds of sequences for a small model, fewer for a large one, fewer still with long contexts) compute saturates. Throughput flattens and latency climbs steeply.\n\nConcretely: a data-centre card serving an 8-billion model in bf16 can deliver several thousand output tokens a second in aggregate at around a hundred concurrent users, with each of them seeing thirty to fifty tokens a second. One user alone saw perhaps 150. The card is doing thirty times the work for a third of the per-user speed, and that trade is the entire economics of the API you pay for.\n\nThe word for the number you actually want is **goodput**: throughput counted only for requests that met a latency target, say first token under a second and under fifty milliseconds between tokens. You choose an operating point on the curve by deciding what target you owe your users, then load-testing until you find the concurrency at which you stop meeting it. That is the card's capacity, and it is a measurement, not a spec.\n\n## The knobs, and one interaction\n\nEvery engine exposes roughly the same settings under different names: a maximum number of sequences in the batch, a maximum number of tokens processed per step, the fraction of GPU memory to reserve, the maximum context per sequence, and a switch for prefix caching. The fitting lesson's arithmetic tells you what each costs in cache.\n\nOne interaction is worth knowing. Speculative decoding, from module three, uses idle compute to verify drafted tokens. Under heavy batching there is no idle compute: batching has already spent it. So speculation helps a lightly loaded server and can slow a saturated one, and engines now switch it off dynamically for that reason.\n\n## Where it is going\n\nPrefill and decode are different regimes (compute-bound and bandwidth-bound) so the largest deployments now run them on separate pools of GPUs and ship the finished cache from one to the other. It adds a transfer and removes the interference between the two, and it is the current frontier of serving design rather than something you need on day one.\n\n## The free path, and the measurement\n\nvLLM and SGLang are open source and are what many commercial APIs run underneath. `llama-server` does continuous batching too, at the scale of a handful of users. Rent a card by the hour, load your model, and run a load test with a realistic mix of prompt and output lengths, not the same short prompt repeated, which flatters everything through prefix sharing. Plot throughput and the 95th percentile of inter-token latency against concurrency. The knee in that plot is the number you were looking for."},"check":{"question":"A server delivers four times the aggregate throughput at 64 concurrent users that it did at one, yet each user's inter-token latency has only risen from 20 to 28 milliseconds. Why is the latency rise so small?","options":[{"text":"The scheduler serves users round-robin, giving each the whole GPU for a short slice, so each user's own steps run at full single-user speed.","correct":false,"tempting":"Describes time-slicing, which would make latency scale with the number of users; the observed near-flat latency is the opposite of what slicing produces."},{"text":"Chunked prefill spreads each new prompt across many steps, which hides its cost from every user's latency figure.","correct":false,"tempting":"Chunked prefill does smooth latency, but it protects against stalls from new arrivals; it does not explain why 64 simultaneous decoders barely slow each other."},{"text":"The key-value cache is shared between all concurrent users, so additional users add no memory traffic to each step.","correct":false,"tempting":"Cache is per sequence, not shared (only common prefixes are shared through paging) so extra users do add cache reads; what they do not add is another read of the weights."},{"text":"Below the crossover, one read of the weights per step serves everyone, and the extra sequences use compute that was idle.","correct":true}]},"prev":{"slug":"running-on-your-own-machine","title":"Running a model on your own machine","url":"https://addaly.com/learn/how-llms-work/running-on-your-own-machine"},"next":{"slug":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","url":"https://addaly.com/learn/how-llms-work/splitting-a-model-across-gpus"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Serving a hundred people from one card', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/serving-many-users (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Serving a hundred people from one card","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"A server fills the compute that single-stream decode leaves idle by scheduling at the token step with a paged cache, so throughput rises almost free with concurrency until compute saturates, and the capacity of a card is the concurrency at which it stops meeting your latency target.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-idle-machine-one-more-time","runs":[{"text":"The idle machine, one more time"}]},{"t":"p","runs":[{"text":"The bandwidth lesson left a GPU generating for one user using about a three-hundredth of its arithmetic capacity, because every decode step reads all the weights to produce one token. A server exists to fill the other 299 parts. The way it does that has changed twice since 2022, and each change is a mechanism you can reason about."}]},{"t":"h2","id":"static-batching-and-why-it-was-abandoned","runs":[{"text":"Static batching, and why it was abandoned"}]},{"t":"p","runs":[{"text":"The first idea: wait until sixteen requests have arrived, run them together as one batch, return the results. Throughput rises sixteen-fold. Three things go wrong. Requests wait for the batch to fill. Every sequence in the batch is padded to the longest, wasting cache. And the batch finishes when its longest response finishes: a user who asked for one word waits for the user who asked for an essay. Static batching gave good throughput numbers and a dreadful experience, and no serious server uses it now."}]},{"t":"h2","id":"continuous-batching","runs":[{"text":"Continuous batching"}]},{"t":"p","runs":[{"text":"The fix, published in 2022 and standard by 2023, schedules at the level of the "},{"text":"token step","b":true},{"text":" rather than the request. At each decode step the batch is whichever sequences are alive. A sequence that emits its stop token leaves the batch at that step; a request waiting in the queue joins at the next. There is no padding, no waiting for a batch to fill, and no short request held hostage by a long one."}]},{"t":"p","runs":[{"text":"It sounds obvious. It required rewriting how memory is managed, because a batch whose membership changes every few milliseconds cannot pre-allocate a fixed slab of cache per sequence."}]},{"t":"h2","id":"paged-cache","runs":[{"text":"Paged cache"}]},{"t":"p","runs":[{"text":"The memory answer, from 2023, borrowed the idea of virtual memory. The key-value cache is allocated in fixed blocks of sixteen tokens or so, mapped through a table, instead of one contiguous region sized for the maximum possible length. Before this, servers reserved the maximum context for every sequence and wasted, in published measurements, sixty to eighty per cent of the cache on space that was never used. With paging the waste fell to a few per cent, which by the fitting lesson’s arithmetic is directly a multiple on how many users one card can hold."}]},{"t":"p","runs":[{"text":"Paging also gives module four’s prompt caching its mechanism on the server side: two sequences that share a prefix (the same system prompt, the same document) can point at the same physical blocks. The prefix is computed once and shared, not copied."}]},{"t":"h2","id":"chunked-prefill","runs":[{"text":"Chunked prefill"}]},{"t":"p","runs":[{"text":"A new request with a 10,000-token prompt arrives. Prefill is compute-bound, and running it in one go would occupy the card for most of a second, during which every user mid-generation sees their stream freeze. Chunked prefill cuts the new prompt into pieces of a few hundred to a couple of thousand tokens and interleaves them with decode steps. Everyone else’s inter-token latency stays smooth; the newcomer’s time to first token is slightly longer. It is a scheduling choice about whose latency to protect, and current engines default to protecting the users already talking."}]},{"t":"h2","id":"the-curve-you-are-operating-on","runs":[{"text":"The curve you are operating on"}]},{"t":"p","runs":[{"text":"Add concurrent users and watch two numbers. Aggregate throughput (tokens a second across everyone) rises almost linearly at first, because each added sequence uses compute the card had idle. Each user’s inter-token latency rises slowly. Then, somewhere around the crossover from the bandwidth lesson (hundreds of sequences for a small model, fewer for a large one, fewer still with long contexts) compute saturates. Throughput flattens and latency climbs steeply."}]},{"t":"p","runs":[{"text":"Concretely: a data-centre card serving an 8-billion model in bf16 can deliver several thousand output tokens a second in aggregate at around a hundred concurrent users, with each of them seeing thirty to fifty tokens a second. One user alone saw perhaps 150. The card is doing thirty times the work for a third of the per-user speed, and that trade is the entire economics of the API you pay for."}]},{"t":"p","runs":[{"text":"The word for the number you actually want is "},{"text":"goodput","b":true},{"text":": throughput counted only for requests that met a latency target, say first token under a second and under fifty milliseconds between tokens. You choose an operating point on the curve by deciding what target you owe your users, then load-testing until you find the concurrency at which you stop meeting it. That is the card’s capacity, and it is a measurement, not a spec."}]},{"t":"h2","id":"the-knobs-and-one-interaction","runs":[{"text":"The knobs, and one interaction"}]},{"t":"p","runs":[{"text":"Every engine exposes roughly the same settings under different names: a maximum number of sequences in the batch, a maximum number of tokens processed per step, the fraction of GPU memory to reserve, the maximum context per sequence, and a switch for prefix caching. The fitting lesson’s arithmetic tells you what each costs in cache."}]},{"t":"p","runs":[{"text":"One interaction is worth knowing. Speculative decoding, from module three, uses idle compute to verify drafted tokens. Under heavy batching there is no idle compute: batching has already spent it. So speculation helps a lightly loaded server and can slow a saturated one, and engines now switch it off dynamically for that reason."}]},{"t":"h2","id":"where-it-is-going","runs":[{"text":"Where it is going"}]},{"t":"p","runs":[{"text":"Prefill and decode are different regimes (compute-bound and bandwidth-bound) so the largest deployments now run them on separate pools of GPUs and ship the finished cache from one to the other. It adds a transfer and removes the interference between the two, and it is the current frontier of serving design rather than something you need on day one."}]},{"t":"h2","id":"the-free-path-and-the-measurement","runs":[{"text":"The free path, and the measurement"}],"role":"free"},{"t":"p","runs":[{"text":"vLLM and SGLang are open source and are what many commercial APIs run underneath. "},{"text":"llama-server","code":true},{"text":" does continuous batching too, at the scale of a handful of users. Rent a card by the hour, load your model, and run a load test with a realistic mix of prompt and output lengths, not the same short prompt repeated, which flatters everything through prefix sharing. Plot throughput and the 95th percentile of inter-token latency against concurrency. The knee in that plot is the number you were looking for."}]}],"check":{"question":[{"text":"A server delivers four times the aggregate throughput at 64 concurrent users that it did at one, yet each user’s inter-token latency has only risen from 20 to 28 milliseconds. Why is the latency rise so small?"}],"options":[{"text":[{"text":"The scheduler serves users round-robin, giving each the whole GPU for a short slice, so each user’s own steps run at full single-user speed."}],"correct":false,"tempting":[{"text":"Describes time-slicing, which would make latency scale with the number of users; the observed near-flat latency is the opposite of what slicing produces."}]},{"text":[{"text":"Chunked prefill spreads each new prompt across many steps, which hides its cost from every user’s latency figure."}],"correct":false,"tempting":[{"text":"Chunked prefill does smooth latency, but it protects against stalls from new arrivals; it does not explain why 64 simultaneous decoders barely slow each other."}]},{"text":[{"text":"The key-value cache is shared between all concurrent users, so additional users add no memory traffic to each step."}],"correct":false,"tempting":[{"text":"Cache is per sequence, not shared (only common prefixes are shared through paging) so extra users do add cache reads; what they do not add is another read of the weights."}]},{"text":[{"text":"Below the crossover, one read of the weights per step serves everyone, and the extra sequences use compute that was idle."}],"correct":true,"tempting":null}],"order":[1,0,3,2],"why":null},"prev":{"slug":"running-on-your-own-machine","title":"Running a model on your own machine","minutes":9},"next":{"slug":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","minutes":8}}}