{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"queueing-and-backpressure","title":"Queues, timeouts and refusing work gracefully","url":"https://addaly.com/learn/running-models-yourself/queueing-and-backpressure","position":46,"of":82,"minutes":8,"answer":"Past capacity, an unbounded queue turns a throughput problem into an outage: cap the queue, refuse fast with 429, cancel on client disconnect, and decide the fallback before the incident rather than during it.","keyIdea":"Past capacity, an unbounded queue turns a throughput problem into an outage: cap the queue, refuse fast with 429, cancel on client disconnect, and decide the fallback before the incident rather than during it."},"body":{"format":"markdown","text":"## What happens past capacity\n\nEvery server has a rate above which it cannot keep up. The question is not whether you reach it but what happens when you do, and the default answer in most local deployments is the worst one: accept everything, queue without limit, and let every user wait.\n\nA queue that grows without bound converts a throughput problem into a total outage. Requests time out at the client, the client retries, the retries join the queue, and the server spends its capacity generating answers nobody is waiting for any more. This is a well-known failure shape and it arrives quickly once the arrival rate crosses service rate.\n\n## Bound the queue and fail fast\n\nThe fix is to decide a maximum acceptable wait and refuse work that cannot meet it.\n\n- **Cap the queue length.** Beyond it, return 429 immediately. A fast refusal lets a client back off or fall back; a nine-second wait followed by a timeout gives it nothing.\n- **Set a server-side timeout** shorter than the client's. If the client gives up at 30 seconds, the server should abandon at 25 rather than finishing an answer into a closed connection.\n- **Cancel on disconnect.** When a client goes away, the sequence should be dropped from the batch. Good engines do this; verify yours does, because a server generating for departed clients is spending its whole capacity on nothing.\n\nA reverse proxy is the natural place for the first two:\n\n```\n# Caddy\nreverse_proxy localhost:8000 {\n    transport http { response_header_timeout 30s }\n}\n```\n\n## Admission control by cost\n\nNot all requests cost the same. One asking for 4,000 output tokens from a 30,000-token prompt occupies a slot for a very long time. Sensible controls:\n\n- **A maximum `max_tokens`,** enforced server-side rather than trusted from the client.\n- **A maximum prompt length,** rejected with a clear message rather than truncated silently.\n- **Separate queues for interactive and batch work.** A nightly classification job and a person waiting for a chat reply should not compete on equal terms. Two servers, or two priority classes, or a schedule.\n\n## Retries that do not amplify\n\nIf clients retry, they must do so in a way that helps.\n\n- **Exponential backoff with jitter.** Fixed-interval retries from many clients synchronise into waves.\n- **A retry budget.** Two attempts, not indefinite.\n- **Never retry a request that failed because it was too large.** It will fail again identically.\n\n## Degrade deliberately\n\nWhen a local model is unavailable (loading, out of memory, or the machine is down) decide in advance what happens, because the alternative is deciding it during an incident.\n\nOptions, in rough order of preference: queue briefly and serve when free; fall back to a smaller local model that is always resident; fall back to a hosted API, if the data policy permits it, which is exactly the question to have settled beforehand; or refuse clearly and tell the user when to come back.\n\nAll four are defensible. Hanging is not.\n\n## Watch the right numbers\n\nThree signals tell you where you are:\n\n1. **Queue depth over time.** Rising steadily means arrival rate exceeds service rate and nothing will fix it but less work or more capacity.\n2. **p95 time to first token.** The user-visible symptom, and the first to move.\n3. **Rejection rate.** Should be zero normally and non-zero under overload. A rejection rate that is always zero while latency climbs means you have no admission control at all.\n\n## The uncomfortable conclusion\n\nA single GPU has a fixed capacity, and no configuration exceeds it. Tuning buys you a factor, not an order of magnitude. Past that the honest options are fewer requests, smaller requests, another machine, or an API for the overflow.\n\nSaying that clearly to whoever asked for the service is more useful than another week of flags, and it is the conversation the measurements from this block let you have with evidence rather than apology."},"check":{"question":"During a traffic spike, a local inference server's queue grows steadily, clients time out at 30 seconds and retry automatically, and throughput of successfully delivered answers falls to almost nothing even though the GPU stays fully busy. What is the mechanism?","options":[{"text":"The GPU is thermally throttling under sustained load, so real throughput has dropped even though utilisation reads high.","correct":false,"tempting":"Throttling reduces speed by tens of per cent, not to almost nothing, and would not be triggered by queueing."},{"text":"Continuous batching is admitting too many sequences, so each individual sequence generates too slowly to finish before its client gives up.","correct":false,"tempting":"Names a real effect of an oversized batch, but the collapse here is driven by work for departed clients rather than by per-sequence slowness alone."},{"text":"The server is generating answers for clients that have already given up, and each retry adds another request, so capacity is consumed by work nobody will receive.","correct":true},{"text":"Prefix caching is thrashing under diverse traffic, so every request pays full prefill and the effective service rate has halved.","correct":false,"tempting":"Would reduce capacity by a factor, not eliminate delivered throughput while the GPU remains saturated."}]},"prev":{"slug":"running-several-models","title":"One machine, several models","url":"https://addaly.com/learn/running-models-yourself/running-several-models"},"next":{"slug":"sampling-is-often-the-problem","title":"Sampling: why output quality is often not the model","url":"https://addaly.com/learn/running-models-yourself/sampling-is-often-the-problem"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Queues, timeouts and refusing work gracefully', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/queueing-and-backpressure (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Queues, timeouts and refusing work gracefully","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Past capacity, an unbounded queue turns a throughput problem into an outage: cap the queue, refuse fast with 429, cancel on client disconnect, and decide the fallback before the incident rather than during it.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"what-happens-past-capacity","runs":[{"text":"What happens past capacity"}]},{"t":"p","runs":[{"text":"Every server has a rate above which it cannot keep up. The question is not whether you reach it but what happens when you do, and the default answer in most local deployments is the worst one: accept everything, queue without limit, and let every user wait."}]},{"t":"p","runs":[{"text":"A queue that grows without bound converts a throughput problem into a total outage. Requests time out at the client, the client retries, the retries join the queue, and the server spends its capacity generating answers nobody is waiting for any more. This is a well-known failure shape and it arrives quickly once the arrival rate crosses service rate."}]},{"t":"h2","id":"bound-the-queue-and-fail-fast","runs":[{"text":"Bound the queue and fail fast"}]},{"t":"p","runs":[{"text":"The fix is to decide a maximum acceptable wait and refuse work that cannot meet it."}]},{"t":"ul","items":[[{"text":"Cap the queue length.","b":true},{"text":" Beyond it, return 429 immediately. A fast refusal lets a client back off or fall back; a nine-second wait followed by a timeout gives it nothing."}],[{"text":"Set a server-side timeout","b":true},{"text":" shorter than the client’s. If the client gives up at 30 seconds, the server should abandon at 25 rather than finishing an answer into a closed connection."}],[{"text":"Cancel on disconnect.","b":true},{"text":" When a client goes away, the sequence should be dropped from the batch. Good engines do this; verify yours does, because a server generating for departed clients is spending its whole capacity on nothing."}]]},{"t":"p","runs":[{"text":"A reverse proxy is the natural place for the first two:"}]},{"t":"code","lang":"","label":"Text","text":"# Caddy\nreverse_proxy localhost:8000 {\n    transport http { response_header_timeout 30s }\n}"},{"t":"h2","id":"admission-control-by-cost","runs":[{"text":"Admission control by cost"}]},{"t":"p","runs":[{"text":"Not all requests cost the same. One asking for 4,000 output tokens from a 30,000-token prompt occupies a slot for a very long time. Sensible controls:"}]},{"t":"ul","items":[[{"text":"A maximum ","b":true},{"text":"max_tokens","b":true,"code":true},{"text":",","b":true},{"text":" enforced server-side rather than trusted from the client."}],[{"text":"A maximum prompt length,","b":true},{"text":" rejected with a clear message rather than truncated silently."}],[{"text":"Separate queues for interactive and batch work.","b":true},{"text":" A nightly classification job and a person waiting for a chat reply should not compete on equal terms. Two servers, or two priority classes, or a schedule."}]]},{"t":"h2","id":"retries-that-do-not-amplify","runs":[{"text":"Retries that do not amplify"}]},{"t":"p","runs":[{"text":"If clients retry, they must do so in a way that helps."}]},{"t":"ul","items":[[{"text":"Exponential backoff with jitter.","b":true},{"text":" Fixed-interval retries from many clients synchronise into waves."}],[{"text":"A retry budget.","b":true},{"text":" Two attempts, not indefinite."}],[{"text":"Never retry a request that failed because it was too large.","b":true},{"text":" It will fail again identically."}]]},{"t":"h2","id":"degrade-deliberately","runs":[{"text":"Degrade deliberately"}]},{"t":"p","runs":[{"text":"When a local model is unavailable (loading, out of memory, or the machine is down) decide in advance what happens, because the alternative is deciding it during an incident."}]},{"t":"p","runs":[{"text":"Options, in rough order of preference: queue briefly and serve when free; fall back to a smaller local model that is always resident; fall back to a hosted API, if the data policy permits it, which is exactly the question to have settled beforehand; or refuse clearly and tell the user when to come back."}]},{"t":"p","runs":[{"text":"All four are defensible. Hanging is not."}]},{"t":"h2","id":"watch-the-right-numbers","runs":[{"text":"Watch the right numbers"}]},{"t":"p","runs":[{"text":"Three signals tell you where you are:"}]},{"t":"ol","start":1,"items":[[{"text":"Queue depth over time.","b":true},{"text":" Rising steadily means arrival rate exceeds service rate and nothing will fix it but less work or more capacity."}],[{"text":"p95 time to first token.","b":true},{"text":" The user-visible symptom, and the first to move."}],[{"text":"Rejection rate.","b":true},{"text":" Should be zero normally and non-zero under overload. A rejection rate that is always zero while latency climbs means you have no admission control at all."}]]},{"t":"h2","id":"the-uncomfortable-conclusion","runs":[{"text":"The uncomfortable conclusion"}]},{"t":"p","runs":[{"text":"A single GPU has a fixed capacity, and no configuration exceeds it. Tuning buys you a factor, not an order of magnitude. Past that the honest options are fewer requests, smaller requests, another machine, or an API for the overflow."}]},{"t":"p","runs":[{"text":"Saying that clearly to whoever asked for the service is more useful than another week of flags, and it is the conversation the measurements from this block let you have with evidence rather than apology."}]}],"check":{"question":[{"text":"During a traffic spike, a local inference server’s queue grows steadily, clients time out at 30 seconds and retry automatically, and throughput of successfully delivered answers falls to almost nothing even though the GPU stays fully busy. What is the mechanism?"}],"options":[{"text":[{"text":"The GPU is thermally throttling under sustained load, so real throughput has dropped even though utilisation reads high."}],"correct":false,"tempting":[{"text":"Throttling reduces speed by tens of per cent, not to almost nothing, and would not be triggered by queueing."}]},{"text":[{"text":"Continuous batching is admitting too many sequences, so each individual sequence generates too slowly to finish before its client gives up."}],"correct":false,"tempting":[{"text":"Names a real effect of an oversized batch, but the collapse here is driven by work for departed clients rather than by per-sequence slowness alone."}]},{"text":[{"text":"The server is generating answers for clients that have already given up, and each retry adds another request, so capacity is consumed by work nobody will receive."}],"correct":true,"tempting":null},{"text":[{"text":"Prefix caching is thrashing under diverse traffic, so every request pays full prefill and the effective service rate has halved."}],"correct":false,"tempting":[{"text":"Would reduce capacity by a factor, not eliminate delivered throughput while the GPU remains saturated."}]}],"order":[0,3,1,2],"why":null},"prev":{"slug":"running-several-models","title":"One machine, several models","minutes":9},"next":{"slug":"sampling-is-often-the-problem","title":"Sampling: why output quality is often not the model","minutes":8}}}