One card, several people

Fairness is a scheduling problem, not a politeness problem

A single GPU serving several people has no built-in notion of fairness. Inference engines run a first-come, first-served queue over a fixed number of slots. Whoever submits most gets most. On a shared research box this reliably produces the same complaint: “it was fine last week”, meaning somebody started a batch job.

There are three mechanisms available, and they solve different problems.

Per-key concurrency: the one that matters most

The engine can run some number of sequences at once: set by --max-num-seqs in vLLM, -np in llama.cpp’s server. Every slot that a batch job holds is a slot an interactive user waits for. Capping how many of one key’s requests may be in flight simultaneously leaves headroom for everybody else, and it is enforced in the gateway, not the engine, because the engine does not know who is calling.

A workable split on a card running eight sequences: interactive keys get a concurrency of two each, batch keys get two total and are told to expect queueing. The batch job finishes a little later and the chat interface stays responsive, which is almost always the right trade because a human is waiting on one and not the other.

Priority queueing, and the starvation it invites

The next step up is a priority queue in the gateway: interactive requests jump ahead of batch ones. vLLM supports a priority scheduling policy directly, and any gateway can implement it with two queues.

The failure is classic and worth anticipating. With enough interactive traffic, low-priority work never runs at all: it starves. The standard fix is ageing: a request’s effective priority rises the longer it waits, so a batch job delayed ten minutes eventually outranks a fresh chat message. Without ageing, someone’s nightly job silently never completes and nobody notices for a week.

Time-slicing the whole card, and why it usually disappoints

NVIDIA offers MPS (Multi-Process Service) and, on data-centre cards, MIG (Multi-Instance GPU). MIG carves a card into hardware-isolated partitions with their own memory and compute: genuine isolation, available on A100/H100-class hardware and not on consumer cards. MPS lets several processes share the card with less context-switching overhead than the default, with no memory isolation: one process’s out-of-memory error takes down its neighbours.

Neither is usually what a small deployment wants, for an arithmetic reason. Splitting a 24GB card into two 12GB halves means two copies of engine overhead, two KV caches sized for half the work, and two models that must each fit in 12GB. One engine serving both users with continuous batching gets more total throughput from the same silicon, because batching shares the weight reads that dominate decode. Partition only when isolation is a requirement rather than a preference.

Session affinity: the free throughput nobody claims

If you run more than one replica, route a given conversation back to the replica that served its previous turn. The reason is the prefix cache: that replica already holds the conversation’s key and value tensors, so the next turn’s prefill is nearly free. Round-robin routing throws that away and re-computes the whole history on a different machine every turn.

The effect is large on long conversations. A 6,000-token history re-prefilled on every turn is thousands of tokens of wasted compute per message; served from a warm prefix cache it is the new tokens only. Hash the conversation identifier to a replica and keep it there. Fall back to another replica when that one is unhealthy, accepting the one-off cost.

Tell people the number

The cheapest fairness mechanism is publishing what is available. “This endpoint serves roughly 1,500 tokens per second in total; at eight concurrent users expect about 25 tokens per second each” converts an argument into arithmetic. People schedule around a number they can see. They cannot schedule around a vague sense that the machine is busy.

Expose two figures from the engine’s metrics endpoint and put them somewhere visible: current queue depth and the ratio of running to waiting sequences. vLLM publishes both on /metrics in Prometheus format, which a free Grafana instance renders in ten minutes.

curl -s http://127.0.0.1:8000/metrics | grep -E 'num_requests_(running|waiting)'
# vllm:num_requests_running 8.0
# vllm:num_requests_waiting 23.0

A waiting count that is persistently many times the running count means the card is saturated and no amount of fairness tuning will fix it: you are rationing a shortage, and the decision has become whether to buy more capacity or cap demand. Fairness mechanisms distribute a shortage; they do not remove one.

A queue that is always full is a capacity decision wearing a scheduling costume.