{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"continuous-batching","title":"Continuous batching, and the queue underneath it","url":"https://addaly.com/learn/running-models-yourself/continuous-batching","position":39,"of":82,"minutes":9,"answer":"Continuous batching re-forms the batch after every decoding step, so a new request waits milliseconds rather than for the previous batch to drain.","keyIdea":"Continuous batching re-forms the batch after every decoding step, so a new request waits milliseconds rather than for the previous batch to drain, and preemption in the logs means your cache memory is too small for the traffic."},"body":{"format":"markdown","text":"## Static batching wastes most of the machine\n\nThe naive way to batch is to collect eight requests, run them together, and return all eight when the last one finishes. The trouble is that generation lengths differ enormously. If seven requests want 40 tokens and one wants 900, the seven finish early and their slots sit idle for the remaining 860 steps, computing padding. Utilisation collapses towards the reciprocal of the length variance, which on real traffic is dreadful.\n\n**Continuous batching** (also called iteration-level scheduling) makes the batch a living thing. After every single decoding step the scheduler asks: has anything finished, and is anything waiting? Finished sequences are retired and their slots given to waiting requests immediately.\n\nThe consequences are large and worth stating plainly. A new request does not wait for the current batch to end, only for the current *step*, which is milliseconds. The batch stays full, so the expensive weight read is amortised across as many sequences as possible. And a very long generation no longer blocks everything behind it.\n\nThis is the single biggest reason a purpose-built server outperforms a loop around a model, and it is not a small factor: it is often five to ten times on realistic traffic.\n\n## What the scheduler is actually deciding\n\nAt each step the engine holds two queues: running sequences and waiting ones. Admitting a waiting request means allocating cache blocks for its prompt and adding it to the batch. The constraints are the memory available for cache blocks and a configured cap on the batch.\n\nWhen memory runs short, the engine has to make a choice. It can stop admitting new work, which grows the queue and raises time to first token. Or it can **preempt** a running sequence (evict its cache and put it back in the waiting queue) which means recomputing its prefill later. vLLM does the latter under pressure and logs it. Frequent preemption is a clear signal that `--gpu-memory-utilization` is too low or `--max-model-len` too high for your traffic, and it shows up to users as unpredictable latency rather than as an error.\n\n## Prefill and decode compete\n\nA subtlety that explains a common complaint. Decode steps are short and frequent; a prefill of 20,000 tokens is one long operation. If the scheduler runs that prefill as a single unit, every other user's stream pauses for the duration. They experience a stutter mid-sentence and report the server as unreliable.\n\nChunked prefill splits the prompt and interleaves the pieces with decoding steps, trading a slightly slower prefill for far lower latency variance across everyone else. On busy multi-user servers it should generally be on.\n\n```bash\nvllm serve <model> \\\n  --max-num-seqs 128 \\\n  --max-num-batched-tokens 4096 \\\n  --enable-chunked-prefill\n```\n\n`--max-num-seqs` caps concurrent sequences. `--max-num-batched-tokens` caps the tokens processed in one step, which is the knob that governs the prefill-versus-decode balance: larger values favour prefill throughput, smaller ones favour smooth streaming.\n\n## Where the gains stop\n\nThroughput against batch size rises steeply, then bends, then flattens. Three reasons, in the order they bite:\n\n1. **The memory-bound to compute-bound transition.** At batch one, the weight read produces one token; at batch 32 it produces 32. Once the arithmetic intensity reaches the hardware's balance point, you are compute-limited and further batching adds little.\n2. **Cache capacity.** Every sequence needs its own KV cache. At long context you run out of blocks long before you run out of compute.\n3. **Attention does not batch as cleanly as the feed-forward layers.** Each sequence attends over its own distinct cache, so that part of the work scales with the batch rather than being amortised.\n\nMeasure your own curve. Run the same load at concurrency 1, 4, 16, 32 and 64, and plot aggregate throughput and p95 latency. The point where throughput flattens while latency keeps climbing is your operating limit, and it is a property of your model, your context length and your card, not something to read from a benchmark.\n\n## In llama.cpp\n\n```bash\nllama-server -m model.gguf -c 32768 --parallel 4 --cont-batching\n```\n\nContinuous batching is present and works. Aggregate throughput at high concurrency remains well below the purpose-built servers, because the kernels and the scheduler were built for a different priority. For a handful of users it is entirely adequate, and remember that `-c` is the total budget divided across the slots."},"check":{"question":"A vLLM server logs frequent preemption events during busy periods, and users report latency that varies wildly rather than errors. What is happening and what is the appropriate first response?","options":[{"text":"The batch size cap is too low, so requests queue behind a full batch; raising max-num-seqs will admit more work.","correct":false,"tempting":"Would increase memory pressure and therefore preemption, making the observed symptom worse."},{"text":"Cache memory runs short, so running sequences are evicted and their prefill recomputed later; raise the memory fraction for cache or lower the maximum context length.","correct":true},{"text":"Chunked prefill is fragmenting long prompts across steps, and disabling it will restore predictable latency.","correct":false,"tempting":"Reverses the purpose of chunked prefill, which reduces latency variance; disabling it makes stutter worse, and it does not cause preemption."},{"text":"The GPU is thermally throttling under sustained load, and preemption is the engine's response to reduced throughput.","correct":false,"tempting":"Invents a link between thermals and scheduling; preemption is a memory decision and engines do not monitor temperature to make it."}]},"prev":{"slug":"engines-and-what-faster-means","title":"Serving engines, and what \"faster\" means","url":"https://addaly.com/learn/running-models-yourself/engines-and-what-faster-means"},"next":{"slug":"prefix-caching-properly","title":"Prefix caching, and how to design prompts for it","url":"https://addaly.com/learn/running-models-yourself/prefix-caching-properly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Continuous batching, and the queue underneath it', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/continuous-batching (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Continuous batching, and the queue underneath it","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Continuous batching re-forms the batch after every decoding step, so a new request waits milliseconds rather than for the previous batch to drain, and preemption in the logs means your cache memory is too small for the traffic.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"static-batching-wastes-most-of-the-machine","runs":[{"text":"Static batching wastes most of the machine"}]},{"t":"p","runs":[{"text":"The naive way to batch is to collect eight requests, run them together, and return all eight when the last one finishes. The trouble is that generation lengths differ enormously. If seven requests want 40 tokens and one wants 900, the seven finish early and their slots sit idle for the remaining 860 steps, computing padding. Utilisation collapses towards the reciprocal of the length variance, which on real traffic is dreadful."}]},{"t":"p","runs":[{"text":"Continuous batching","b":true},{"text":" (also called iteration-level scheduling) makes the batch a living thing. After every single decoding step the scheduler asks: has anything finished, and is anything waiting? Finished sequences are retired and their slots given to waiting requests immediately."}]},{"t":"p","runs":[{"text":"The consequences are large and worth stating plainly. A new request does not wait for the current batch to end, only for the current "},{"text":"step","i":true},{"text":", which is milliseconds. The batch stays full, so the expensive weight read is amortised across as many sequences as possible. And a very long generation no longer blocks everything behind it."}]},{"t":"p","runs":[{"text":"This is the single biggest reason a purpose-built server outperforms a loop around a model, and it is not a small factor: it is often five to ten times on realistic traffic."}]},{"t":"h2","id":"what-the-scheduler-is-actually-deciding","runs":[{"text":"What the scheduler is actually deciding"}]},{"t":"p","runs":[{"text":"At each step the engine holds two queues: running sequences and waiting ones. Admitting a waiting request means allocating cache blocks for its prompt and adding it to the batch. The constraints are the memory available for cache blocks and a configured cap on the batch."}]},{"t":"p","runs":[{"text":"When memory runs short, the engine has to make a choice. It can stop admitting new work, which grows the queue and raises time to first token. Or it can "},{"text":"preempt","b":true},{"text":" a running sequence (evict its cache and put it back in the waiting queue) which means recomputing its prefill later. vLLM does the latter under pressure and logs it. Frequent preemption is a clear signal that "},{"text":"--gpu-memory-utilization","code":true},{"text":" is too low or "},{"text":"--max-model-len","code":true},{"text":" too high for your traffic, and it shows up to users as unpredictable latency rather than as an error."}]},{"t":"h2","id":"prefill-and-decode-compete","runs":[{"text":"Prefill and decode compete"}]},{"t":"p","runs":[{"text":"A subtlety that explains a common complaint. Decode steps are short and frequent; a prefill of 20,000 tokens is one long operation. If the scheduler runs that prefill as a single unit, every other user’s stream pauses for the duration. They experience a stutter mid-sentence and report the server as unreliable."}]},{"t":"p","runs":[{"text":"Chunked prefill splits the prompt and interleaves the pieces with decoding steps, trading a slightly slower prefill for far lower latency variance across everyone else. On busy multi-user servers it should generally be on."}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve <model> \\\n  --max-num-seqs 128 \\\n  --max-num-batched-tokens 4096 \\\n  --enable-chunked-prefill"},{"t":"p","runs":[{"text":"--max-num-seqs","code":true},{"text":" caps concurrent sequences. "},{"text":"--max-num-batched-tokens","code":true},{"text":" caps the tokens processed in one step, which is the knob that governs the prefill-versus-decode balance: larger values favour prefill throughput, smaller ones favour smooth streaming."}]},{"t":"h2","id":"where-the-gains-stop","runs":[{"text":"Where the gains stop"}]},{"t":"p","runs":[{"text":"Throughput against batch size rises steeply, then bends, then flattens. Three reasons, in the order they bite:"}]},{"t":"ol","start":1,"items":[[{"text":"The memory-bound to compute-bound transition.","b":true},{"text":" At batch one, the weight read produces one token; at batch 32 it produces 32. Once the arithmetic intensity reaches the hardware’s balance point, you are compute-limited and further batching adds little."}],[{"text":"Cache capacity.","b":true},{"text":" Every sequence needs its own KV cache. At long context you run out of blocks long before you run out of compute."}],[{"text":"Attention does not batch as cleanly as the feed-forward layers.","b":true},{"text":" Each sequence attends over its own distinct cache, so that part of the work scales with the batch rather than being amortised."}]]},{"t":"p","runs":[{"text":"Measure your own curve. Run the same load at concurrency 1, 4, 16, 32 and 64, and plot aggregate throughput and p95 latency. The point where throughput flattens while latency keeps climbing is your operating limit, and it is a property of your model, your context length and your card, not something to read from a benchmark."}]},{"t":"h2","id":"in-llama-cpp","runs":[{"text":"In llama.cpp"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -c 32768 --parallel 4 --cont-batching"},{"t":"p","runs":[{"text":"Continuous batching is present and works. Aggregate throughput at high concurrency remains well below the purpose-built servers, because the kernels and the scheduler were built for a different priority. For a handful of users it is entirely adequate, and remember that "},{"text":"-c","code":true},{"text":" is the total budget divided across the slots."}]}],"check":{"question":[{"text":"A vLLM server logs frequent preemption events during busy periods, and users report latency that varies wildly rather than errors. What is happening and what is the appropriate first response?"}],"options":[{"text":[{"text":"The batch size cap is too low, so requests queue behind a full batch; raising max-num-seqs will admit more work."}],"correct":false,"tempting":[{"text":"Would increase memory pressure and therefore preemption, making the observed symptom worse."}]},{"text":[{"text":"Cache memory runs short, so running sequences are evicted and their prefill recomputed later; raise the memory fraction for cache or lower the maximum context length."}],"correct":true,"tempting":null},{"text":[{"text":"Chunked prefill is fragmenting long prompts across steps, and disabling it will restore predictable latency."}],"correct":false,"tempting":[{"text":"Reverses the purpose of chunked prefill, which reduces latency variance; disabling it makes stutter worse, and it does not cause preemption."}]},{"text":[{"text":"The GPU is thermally throttling under sustained load, and preemption is the engine’s response to reduced throughput."}],"correct":false,"tempting":[{"text":"Invents a link between thermals and scheduling; preemption is a memory decision and engines do not monitor temperature to make it."}]}],"order":[0,3,2,1],"why":null},"prev":{"slug":"engines-and-what-faster-means","title":"Serving engines, and what “faster” means","minutes":11},"next":{"slug":"prefix-caching-properly","title":"Prefix caching, and how to design prompts for it","minutes":9}}}