Queues, timeouts and refusing work gracefully

What happens past capacity

Every server has a rate above which it cannot keep up. The question is not whether you reach it but what happens when you do, and the default answer in most local deployments is the worst one: accept everything, queue without limit, and let every user wait.

A queue that grows without bound converts a throughput problem into a total outage. Requests time out at the client, the client retries, the retries join the queue, and the server spends its capacity generating answers nobody is waiting for any more. This is a well-known failure shape and it arrives quickly once the arrival rate crosses service rate.

Bound the queue and fail fast

The fix is to decide a maximum acceptable wait and refuse work that cannot meet it.

• Cap the queue length. Beyond it, return 429 immediately. A fast refusal lets a client back off or fall back; a nine-second wait followed by a timeout gives it nothing.
• Set a server-side timeout shorter than the client’s. If the client gives up at 30 seconds, the server should abandon at 25 rather than finishing an answer into a closed connection.
• Cancel on disconnect. When a client goes away, the sequence should be dropped from the batch. Good engines do this; verify yours does, because a server generating for departed clients is spending its whole capacity on nothing.

A reverse proxy is the natural place for the first two:

# Caddy
reverse_proxy localhost:8000 {
    transport http { response_header_timeout 30s }
}

Admission control by cost

Not all requests cost the same. One asking for 4,000 output tokens from a 30,000-token prompt occupies a slot for a very long time. Sensible controls:

• A maximum max_tokens, enforced server-side rather than trusted from the client.
• A maximum prompt length, rejected with a clear message rather than truncated silently.
• Separate queues for interactive and batch work. A nightly classification job and a person waiting for a chat reply should not compete on equal terms. Two servers, or two priority classes, or a schedule.

Retries that do not amplify

If clients retry, they must do so in a way that helps.

• Exponential backoff with jitter. Fixed-interval retries from many clients synchronise into waves.
• A retry budget. Two attempts, not indefinite.
• Never retry a request that failed because it was too large. It will fail again identically.

Degrade deliberately

When a local model is unavailable (loading, out of memory, or the machine is down) decide in advance what happens, because the alternative is deciding it during an incident.

Options, in rough order of preference: queue briefly and serve when free; fall back to a smaller local model that is always resident; fall back to a hosted API, if the data policy permits it, which is exactly the question to have settled beforehand; or refuse clearly and tell the user when to come back.

All four are defensible. Hanging is not.

Watch the right numbers

Three signals tell you where you are:

1. Queue depth over time. Rising steadily means arrival rate exceeds service rate and nothing will fix it but less work or more capacity.
2. p95 time to first token. The user-visible symptom, and the first to move.
3. Rejection rate. Should be zero normally and non-zero under overload. A rejection rate that is always zero while latency climbs means you have no admission control at all.

The uncomfortable conclusion

A single GPU has a fixed capacity, and no configuration exceeds it. Tuning buys you a factor, not an order of magnitude. Past that the honest options are fewer requests, smaller requests, another machine, or an API for the overflow.

Saying that clearly to whoever asked for the service is more useful than another week of flags, and it is the conversation the measurements from this block let you have with evidence rather than apology.