{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"sharing-one-gpu-fairly","title":"One card, several people","url":"https://addaly.com/learn/running-models-yourself/sharing-one-gpu-fairly","position":63,"of":82,"minutes":9,"answer":"Engines schedule first-come, first-served, so fairness on a shared card comes from per-key concurrency caps and a priority queue with ageing to prevent starvation.","keyIdea":"Engines schedule first-come, first-served, so fairness on a shared card comes from per-key concurrency caps and a priority queue with ageing to prevent starvation, and routing a conversation back to the replica holding its prefix cache is free throughput that round-robin discards."},"body":{"format":"markdown","text":"## Fairness is a scheduling problem, not a politeness problem\n\nA single GPU serving several people has no built-in notion of fairness. Inference engines run a first-come, first-served queue over a fixed number of slots. Whoever submits most gets most. On a shared research box this reliably produces the same complaint: \"it was fine last week\", meaning somebody started a batch job.\n\nThere are three mechanisms available, and they solve different problems.\n\n## Per-key concurrency: the one that matters most\n\nThe engine can run some number of sequences at once: set by `--max-num-seqs` in vLLM, `-np` in llama.cpp's server. Every slot that a batch job holds is a slot an interactive user waits for. Capping how many of one key's requests may be in flight simultaneously leaves headroom for everybody else, and it is enforced in the gateway, not the engine, because the engine does not know who is calling.\n\nA workable split on a card running eight sequences: interactive keys get a concurrency of two each, batch keys get two total and are told to expect queueing. The batch job finishes a little later and the chat interface stays responsive, which is almost always the right trade because a human is waiting on one and not the other.\n\n## Priority queueing, and the starvation it invites\n\nThe next step up is a priority queue in the gateway: interactive requests jump ahead of batch ones. vLLM supports a priority scheduling policy directly, and any gateway can implement it with two queues.\n\nThe failure is classic and worth anticipating. With enough interactive traffic, low-priority work never runs at all: it starves. The standard fix is ageing: a request's effective priority rises the longer it waits, so a batch job delayed ten minutes eventually outranks a fresh chat message. Without ageing, someone's nightly job silently never completes and nobody notices for a week.\n\n## Time-slicing the whole card, and why it usually disappoints\n\nNVIDIA offers MPS (Multi-Process Service) and, on data-centre cards, MIG (Multi-Instance GPU). MIG carves a card into hardware-isolated partitions with their own memory and compute: genuine isolation, available on A100/H100-class hardware and not on consumer cards. MPS lets several processes share the card with less context-switching overhead than the default, with no memory isolation: one process's out-of-memory error takes down its neighbours.\n\nNeither is usually what a small deployment wants, for an arithmetic reason. Splitting a 24GB card into two 12GB halves means two copies of engine overhead, two KV caches sized for half the work, and two models that must each fit in 12GB. One engine serving both users with continuous batching gets more total throughput from the same silicon, because batching shares the weight reads that dominate decode. Partition only when isolation is a requirement rather than a preference.\n\n## Session affinity: the free throughput nobody claims\n\nIf you run more than one replica, route a given conversation back to the replica that served its previous turn. The reason is the prefix cache: that replica already holds the conversation's key and value tensors, so the next turn's prefill is nearly free. Round-robin routing throws that away and re-computes the whole history on a different machine every turn.\n\nThe effect is large on long conversations. A 6,000-token history re-prefilled on every turn is thousands of tokens of wasted compute per message; served from a warm prefix cache it is the new tokens only. Hash the conversation identifier to a replica and keep it there. Fall back to another replica when that one is unhealthy, accepting the one-off cost.\n\n## Tell people the number\n\nThe cheapest fairness mechanism is publishing what is available. \"This endpoint serves roughly 1,500 tokens per second in total; at eight concurrent users expect about 25 tokens per second each\" converts an argument into arithmetic. People schedule around a number they can see. They cannot schedule around a vague sense that the machine is busy.\n\nExpose two figures from the engine's metrics endpoint and put them somewhere visible: current queue depth and the ratio of running to waiting sequences. vLLM publishes both on `/metrics` in Prometheus format, which a free Grafana instance renders in ten minutes.\n\n```bash\ncurl -s http://127.0.0.1:8000/metrics | grep -E 'num_requests_(running|waiting)'\n# vllm:num_requests_running 8.0\n# vllm:num_requests_waiting 23.0\n```\n\nA waiting count that is persistently many times the running count means the card is saturated and no amount of fairness tuning will fix it: you are rationing a shortage, and the decision has become whether to buy more capacity or cap demand. Fairness mechanisms distribute a shortage; they do not remove one.\n\n> A queue that is always full is a capacity decision wearing a scheduling costume."},"check":{"question":"A team splits a 24GB card into two 12GB MIG partitions so that two users cannot slow each other down. Aggregate throughput falls noticeably. Why?","options":[{"text":"MIG partitions do not support continuous batching, so each half processes requests sequentially.","correct":false,"tempting":"Invents a limitation; each partition runs a normal engine with normal batching, just over a smaller pool of work."},{"text":"Memory bandwidth is divided between partitions, so each half runs at half speed and the total is unchanged.","correct":false,"tempting":"Correctly identifies that bandwidth is shared but predicts no aggregate loss, which does not match the observed fall; the loss comes from lost batching, not from the split itself."},{"text":"Each partition duplicates overhead and runs smaller batches, and batching is what amortises weight reads.","correct":true},{"text":"The model must be loaded twice, and the second copy is served from host memory over PCIe.","correct":false,"tempting":"The model is loaded twice, but into each partition's own VRAM; nothing is being served across PCIe here."}]},"prev":{"slug":"guardrails-on-your-own-endpoint","title":"You are now the moderation team","url":"https://addaly.com/learn/running-models-yourself/guardrails-on-your-own-endpoint"},"next":{"slug":"hybrid-local-and-api","title":"Local first, API behind it","url":"https://addaly.com/learn/running-models-yourself/hybrid-local-and-api"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'One card, several people', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/sharing-one-gpu-fairly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"One card, several people","courseTitle":"Running Models Yourself","module":{"index":7,"slug":"serving-it-to-others","title":"Serving it to other people"},"answer":null,"keyIdea":"Engines schedule first-come, first-served, so fairness on a shared card comes from per-key concurrency caps and a priority queue with ageing to prevent starvation, and routing a conversation back to the replica holding its prefix cache is free throughput that round-robin discards.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"fairness-is-a-scheduling-problem-not-a-politeness-problem","runs":[{"text":"Fairness is a scheduling problem, not a politeness problem"}]},{"t":"p","runs":[{"text":"A single GPU serving several people has no built-in notion of fairness. Inference engines run a first-come, first-served queue over a fixed number of slots. Whoever submits most gets most. On a shared research box this reliably produces the same complaint: “it was fine last week”, meaning somebody started a batch job."}]},{"t":"p","runs":[{"text":"There are three mechanisms available, and they solve different problems."}]},{"t":"h2","id":"per-key-concurrency-the-one-that-matters-most","runs":[{"text":"Per-key concurrency: the one that matters most"}]},{"t":"p","runs":[{"text":"The engine can run some number of sequences at once: set by "},{"text":"--max-num-seqs","code":true},{"text":" in vLLM, "},{"text":"-np","code":true},{"text":" in llama.cpp’s server. Every slot that a batch job holds is a slot an interactive user waits for. Capping how many of one key’s requests may be in flight simultaneously leaves headroom for everybody else, and it is enforced in the gateway, not the engine, because the engine does not know who is calling."}]},{"t":"p","runs":[{"text":"A workable split on a card running eight sequences: interactive keys get a concurrency of two each, batch keys get two total and are told to expect queueing. The batch job finishes a little later and the chat interface stays responsive, which is almost always the right trade because a human is waiting on one and not the other."}]},{"t":"h2","id":"priority-queueing-and-the-starvation-it-invites","runs":[{"text":"Priority queueing, and the starvation it invites"}]},{"t":"p","runs":[{"text":"The next step up is a priority queue in the gateway: interactive requests jump ahead of batch ones. vLLM supports a priority scheduling policy directly, and any gateway can implement it with two queues."}]},{"t":"p","runs":[{"text":"The failure is classic and worth anticipating. With enough interactive traffic, low-priority work never runs at all: it starves. The standard fix is ageing: a request’s effective priority rises the longer it waits, so a batch job delayed ten minutes eventually outranks a fresh chat message. Without ageing, someone’s nightly job silently never completes and nobody notices for a week."}]},{"t":"h2","id":"time-slicing-the-whole-card-and-why-it-usually-disappoints","runs":[{"text":"Time-slicing the whole card, and why it usually disappoints"}]},{"t":"p","runs":[{"text":"NVIDIA offers MPS (Multi-Process Service) and, on data-centre cards, MIG (Multi-Instance GPU). MIG carves a card into hardware-isolated partitions with their own memory and compute: genuine isolation, available on A100/H100-class hardware and not on consumer cards. MPS lets several processes share the card with less context-switching overhead than the default, with no memory isolation: one process’s out-of-memory error takes down its neighbours."}]},{"t":"p","runs":[{"text":"Neither is usually what a small deployment wants, for an arithmetic reason. Splitting a 24GB card into two 12GB halves means two copies of engine overhead, two KV caches sized for half the work, and two models that must each fit in 12GB. One engine serving both users with continuous batching gets more total throughput from the same silicon, because batching shares the weight reads that dominate decode. Partition only when isolation is a requirement rather than a preference."}]},{"t":"h2","id":"session-affinity-the-free-throughput-nobody-claims","runs":[{"text":"Session affinity: the free throughput nobody claims"}]},{"t":"p","runs":[{"text":"If you run more than one replica, route a given conversation back to the replica that served its previous turn. The reason is the prefix cache: that replica already holds the conversation’s key and value tensors, so the next turn’s prefill is nearly free. Round-robin routing throws that away and re-computes the whole history on a different machine every turn."}]},{"t":"p","runs":[{"text":"The effect is large on long conversations. A 6,000-token history re-prefilled on every turn is thousands of tokens of wasted compute per message; served from a warm prefix cache it is the new tokens only. Hash the conversation identifier to a replica and keep it there. Fall back to another replica when that one is unhealthy, accepting the one-off cost."}]},{"t":"h2","id":"tell-people-the-number","runs":[{"text":"Tell people the number"}]},{"t":"p","runs":[{"text":"The cheapest fairness mechanism is publishing what is available. “This endpoint serves roughly 1,500 tokens per second in total; at eight concurrent users expect about 25 tokens per second each” converts an argument into arithmetic. People schedule around a number they can see. They cannot schedule around a vague sense that the machine is busy."}]},{"t":"p","runs":[{"text":"Expose two figures from the engine’s metrics endpoint and put them somewhere visible: current queue depth and the ratio of running to waiting sequences. vLLM publishes both on "},{"text":"/metrics","code":true},{"text":" in Prometheus format, which a free Grafana instance renders in ten minutes."}]},{"t":"code","lang":"bash","label":"Bash","text":"curl -s http://127.0.0.1:8000/metrics | grep -E 'num_requests_(running|waiting)'\n# vllm:num_requests_running 8.0\n# vllm:num_requests_waiting 23.0"},{"t":"p","runs":[{"text":"A waiting count that is persistently many times the running count means the card is saturated and no amount of fairness tuning will fix it: you are rationing a shortage, and the decision has become whether to buy more capacity or cap demand. Fairness mechanisms distribute a shortage; they do not remove one."}]},{"t":"quote","runs":[{"text":"A queue that is always full is a capacity decision wearing a scheduling costume."}]}],"check":{"question":[{"text":"A team splits a 24GB card into two 12GB MIG partitions so that two users cannot slow each other down. Aggregate throughput falls noticeably. Why?"}],"options":[{"text":[{"text":"MIG partitions do not support continuous batching, so each half processes requests sequentially."}],"correct":false,"tempting":[{"text":"Invents a limitation; each partition runs a normal engine with normal batching, just over a smaller pool of work."}]},{"text":[{"text":"Memory bandwidth is divided between partitions, so each half runs at half speed and the total is unchanged."}],"correct":false,"tempting":[{"text":"Correctly identifies that bandwidth is shared but predicts no aggregate loss, which does not match the observed fall; the loss comes from lost batching, not from the split itself."}]},{"text":[{"text":"Each partition duplicates overhead and runs smaller batches, and batching is what amortises weight reads."}],"correct":true,"tempting":null},{"text":[{"text":"The model must be loaded twice, and the second copy is served from host memory over PCIe."}],"correct":false,"tempting":[{"text":"The model is loaded twice, but into each partition’s own VRAM; nothing is being served across PCIe here."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"guardrails-on-your-own-endpoint","title":"You are now the moderation team","minutes":9},"next":{"slug":"hybrid-local-and-api","title":"Local first, API behind it","minutes":8}}}