{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"engines-and-what-faster-means","title":"Serving engines, and what \"faster\" means","url":"https://addaly.com/learn/running-models-yourself/engines-and-what-faster-means","position":38,"of":82,"minutes":11,"answer":"Choose an engine by your concurrency pattern; single-stream speed and aggregate throughput are different, opposing numbers.","keyIdea":"Choose an engine by your concurrency pattern; single-stream speed and aggregate throughput are different, opposing numbers."},"body":{"format":"markdown","text":"Every engine claims to be fastest. Most of those claims are true, under conditions the benchmark does not print. Before comparing anything, fix which number you mean.\n\n## Three different speeds\n\n**Single-stream decode**: tokens per second for one user with nothing else running. This is what you feel when you chat alone.\n\n**Aggregate throughput**: tokens per second summed across all concurrent users. This is what determines cost per token and how many people one GPU serves.\n\n**Time to first token**: queue wait plus prompt processing. This is what users call \"slow\" even when generation is fast.\n\n> Figure: Fastest depends entirely on which number you mean. A two by two grid, rows ExLlamaV2 and vLLM, columns One user and Sixty-four users: ExLlamaV2 and One user: Fastest alone; ExLlamaV2 and Sixty-four users: Falls away; vLLM and One user: Not faster; vLLM and Sixty-four users: Many times more. Both benchmarks are honest. Doubling the users on a paged, continuously batched server costs almost nothing per user; doubling them on an engine tuned for one person roughly halves everyone's speed. Pick for the shape of your load rather than for the headline.\n\nThese trade against each other. Batching many requests together raises aggregate throughput enormously while slightly raising each individual's latency. An engine tuned for one number will lose on another, and both benchmarks are honest.\n\n## llama.cpp\n\nRuns on CPU, CUDA, Metal, Vulkan, ROCm and SYCL, and can split a model between GPU and CPU. GGUF only.\n\n**Wins when:** the model does not fit in VRAM, the hardware is not NVIDIA, you are on a Mac, or you have one or a few users. It is the only engine on this list that runs at all in most of those situations.\n\n**Weakness:** it has continuous batching (`--parallel`, `-cb`), but aggregate throughput at high concurrency is well below the purpose-built servers.\n\n## vLLM\n\nThe throughput default. Two ideas do most of the work. **PagedAttention** stores the KV cache in fixed-size blocks like virtual memory pages, so there is almost no fragmentation and no need to reserve worst-case context per request, which is what lets it hold many more concurrent sequences on the same card. **Continuous batching** admits and retires requests token by token instead of waiting for a whole batch to finish.\n\n**Wins when:** you have concurrent users and the model fits entirely in GPU memory.\n\n**Weakness:** no meaningful CPU offload, a heavy install, slow startup, and single-user latency no better than the alternatives.\n\n## ExLlamaV2 / EXL2\n\nHand-tuned kernels for consumer NVIDIA cards, variable-bit-rate quantisation, quantised KV cache, speculative decoding. TabbyAPI puts an OpenAI-compatible server in front of it.\n\n**Wins when:** one user, one NVIDIA card, and you want the highest possible tokens per second for yourself.\n\n**Weakness:** NVIDIA only, and it degrades far more sharply than vLLM as users are added.\n\n## TGI\n\nHugging Face's Rust and Python server. Feature set close to vLLM, with a stronger operational story: metrics, tracing, tight integration with the Hugging Face ecosystem.\n\n**Status:** in maintenance mode since December 2025 (minor bug fixes only), and its repository is now archived. Hugging Face itself recommends vLLM or SGLang instead, so use TGI only where it is already deployed.\n\n## SGLang\n\nBuilt around **RadixAttention**: the KV cache is organised as a prefix tree, so any request sharing a prefix with an earlier one reuses that computation automatically. Also very fast constrained decoding for structured output.\n\n**Wins when:** many requests share a long system prompt (agent loops, classification pipelines, anything with a fixed preamble) or when you need reliable JSON at high rates.\n\n## Ballpark numbers, and why you should not trust them\n\nEach project publishes its own single-user figures. On an RTX 4090 (a 24 GB consumer card) with Llama 2 7B at about 4 bits, llama.cpp's CUDA scoreboard shows about 186 tok/s (Q4_0) and ExLlamaV2's README about 211 tok/s (4.0 bits per weight). Single-user speeds sit within tens of per cent of each other; the large differences appear in aggregate throughput under concurrency, where batching servers such as vLLM pull far ahead and ExLlamaV2 falls off sharply.\n\nThese are order-of-magnitude guides on hardware that is not yours, with versions that have since changed. Measure: `llama-bench` for llama.cpp, `vllm bench serve` or the serving benchmark script for vLLM, and any HTTP load generator against the OpenAI endpoint for a fair cross-engine comparison.\n\n## The sentence to remember\n\nDoubling the users on vLLM costs almost nothing per user. Doubling the users on ExLlamaV2 roughly halves everyone's speed. Pick the engine for the shape of your load, not for the headline."},"check":{"question":"You benchmark one user on your 24 GB card: vLLM gives 60 tok/s, ExLlamaV2 gives 110 tok/s. You pick ExLlamaV2 for an internal tool that around 40 colleagues will use through the day. What is wrong with that reasoning?","options":[{"text":"The benchmark measured single-stream speed, but with 40 concurrent users vLLM's continuous batching produces far more total throughput and lower queue times.","correct":true},{"text":"Nothing is wrong; per-token speed is a property of the engine and does not change with the number of users.","correct":false,"tempting":"Assumes latency is independent of load, when the whole difference between these engines is how they behave under it."},{"text":"Nothing is wrong, as long as you run 40 ExLlamaV2 processes so each user has their own.","correct":false,"tempting":"Each process would need its own copy of the weights and KV cache, so the card runs out long before 40."},{"text":"vLLM's advantage is only its kernels, so switching ExLlamaV2 to the same quantisation format would close the gap.","correct":false,"tempting":"Credits throughput to kernel speed when it comes from batching and KV memory management."}]},"prev":{"slug":"speech-and-vision-locally","title":"Speech, vision and the rest of the local stack","url":"https://addaly.com/learn/running-models-yourself/speech-and-vision-locally"},"next":{"slug":"continuous-batching","title":"Continuous batching, and the queue underneath it","url":"https://addaly.com/learn/running-models-yourself/continuous-batching"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Serving engines, and what \"faster\" means', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/engines-and-what-faster-means (Accessed: 31 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Serving engines, and what “faster” means","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Choose an engine by your concurrency pattern; single-stream speed and aggregate throughput are different, opposing numbers.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure 5.1","figure":{"cols":["One user","Sixty-four users"],"rows":["ExLlamaV2","vLLM"],"type":"matrix","cells":[["Fastest alone","Falls away"],["Not faster","Many times more"]],"title":"Fastest depends entirely on which number you mean","caption":"Both benchmarks are honest. Doubling the users on a paged, continuously batched server costs almost nothing per user; doubling them on an engine tuned for one person roughly halves everyone’s speed. Pick for the shape of your load rather than for the headline.","cellLabels":[["Hand-tuned kernels","Everyone slows down"],["No better alone","Paged cache, batching"]]},"alt":"Figure: Fastest depends entirely on which number you mean. A two by two grid, rows ExLlamaV2 and vLLM, columns One user and Sixty-four users: ExLlamaV2 and One user: Fastest alone; ExLlamaV2 and Sixty-four users: Falls away; vLLM and One user: Not faster; vLLM and Sixty-four users: Many times more. Both benchmarks are honest. Doubling the users on a paged, continuously batched server costs almost nothing per user; doubling them on an engine tuned for one person roughly halves everyone’s speed. Pick for the shape of your load rather than for the headline."}},"blocks":[{"t":"p","runs":[{"text":"Every engine claims to be fastest. Most of those claims are true, under conditions the benchmark does not print. Before comparing anything, fix which number you mean."}],"lede":true},{"t":"h2","id":"three-different-speeds","runs":[{"text":"Three different speeds"}]},{"t":"p","runs":[{"text":"Single-stream decode","b":true},{"text":": tokens per second for one user with nothing else running. This is what you feel when you chat alone."}]},{"t":"p","runs":[{"text":"Aggregate throughput","b":true},{"text":": tokens per second summed across all concurrent users. This is what determines cost per token and how many people one GPU serves."}]},{"t":"p","runs":[{"text":"Time to first token","b":true},{"text":": queue wait plus prompt processing. This is what users call “slow” even when generation is fast."}]},{"t":"p","runs":[{"text":"These trade against each other. Batching many requests together raises aggregate throughput enormously while slightly raising each individual’s latency. An engine tuned for one number will lose on another, and both benchmarks are honest."}]},{"t":"h2","id":"llama-cpp","runs":[{"text":"llama.cpp"}]},{"t":"p","runs":[{"text":"Runs on CPU, CUDA, Metal, Vulkan, ROCm and SYCL, and can split a model between GPU and CPU. GGUF only."}]},{"t":"p","runs":[{"text":"Wins when:","b":true},{"text":" the model does not fit in VRAM, the hardware is not NVIDIA, you are on a Mac, or you have one or a few users. It is the only engine on this list that runs at all in most of those situations."}]},{"t":"p","runs":[{"text":"Weakness:","b":true},{"text":" it has continuous batching ("},{"text":"--parallel","code":true},{"text":", "},{"text":"-cb","code":true},{"text":"), but aggregate throughput at high concurrency is well below the purpose-built servers."}]},{"t":"h2","id":"vllm","runs":[{"text":"vLLM"}]},{"t":"p","runs":[{"text":"The throughput default. Two ideas do most of the work. "},{"text":"PagedAttention","b":true},{"text":" stores the KV cache in fixed-size blocks like virtual memory pages, so there is almost no fragmentation and no need to reserve worst-case context per request, which is what lets it hold many more concurrent sequences on the same card. "},{"text":"Continuous batching","b":true},{"text":" admits and retires requests token by token instead of waiting for a whole batch to finish."}]},{"t":"p","runs":[{"text":"Wins when:","b":true},{"text":" you have concurrent users and the model fits entirely in GPU memory."}]},{"t":"p","runs":[{"text":"Weakness:","b":true},{"text":" no meaningful CPU offload, a heavy install, slow startup, and single-user latency no better than the alternatives."}]},{"t":"h2","id":"exllamav2-exl2","runs":[{"text":"ExLlamaV2 / EXL2"}]},{"t":"p","runs":[{"text":"Hand-tuned kernels for consumer NVIDIA cards, variable-bit-rate quantisation, quantised KV cache, speculative decoding. TabbyAPI puts an OpenAI-compatible server in front of it."}]},{"t":"p","runs":[{"text":"Wins when:","b":true},{"text":" one user, one NVIDIA card, and you want the highest possible tokens per second for yourself."}]},{"t":"p","runs":[{"text":"Weakness:","b":true},{"text":" NVIDIA only, and it degrades far more sharply than vLLM as users are added."}]},{"t":"h2","id":"tgi","runs":[{"text":"TGI"}]},{"t":"p","runs":[{"text":"Hugging Face’s Rust and Python server. Feature set close to vLLM, with a stronger operational story: metrics, tracing, tight integration with the Hugging Face ecosystem."}]},{"t":"p","runs":[{"text":"Status:","b":true},{"text":" in maintenance mode since December 2025 (minor bug fixes only), and its repository is now archived. Hugging Face itself recommends vLLM or SGLang instead, so use TGI only where it is already deployed."}]},{"t":"h2","id":"sglang","runs":[{"text":"SGLang"}]},{"t":"p","runs":[{"text":"Built around "},{"text":"RadixAttention","b":true},{"text":": the KV cache is organised as a prefix tree, so any request sharing a prefix with an earlier one reuses that computation automatically. Also very fast constrained decoding for structured output."}]},{"t":"p","runs":[{"text":"Wins when:","b":true},{"text":" many requests share a long system prompt (agent loops, classification pipelines, anything with a fixed preamble) or when you need reliable JSON at high rates."}]},{"t":"h2","id":"ballpark-numbers-and-why-you-should-not-trust-them","runs":[{"text":"Ballpark numbers, and why you should not trust them"}]},{"t":"p","runs":[{"text":"Each project publishes its own single-user figures. On an RTX 4090 (a 24 GB consumer card) with Llama 2 7B at about 4 bits, llama.cpp’s CUDA scoreboard shows about 186 tok/s (Q4_0) and ExLlamaV2’s README about 211 tok/s (4.0 bits per weight). Single-user speeds sit within tens of per cent of each other; the large differences appear in aggregate throughput under concurrency, where batching servers such as vLLM pull far ahead and ExLlamaV2 falls off sharply."}]},{"t":"p","runs":[{"text":"These are order-of-magnitude guides on hardware that is not yours, with versions that have since changed. Measure: "},{"text":"llama-bench","code":true},{"text":" for llama.cpp, "},{"text":"vllm bench serve","code":true},{"text":" or the serving benchmark script for vLLM, and any HTTP load generator against the OpenAI endpoint for a fair cross-engine comparison."}]},{"t":"h2","id":"the-sentence-to-remember","runs":[{"text":"The sentence to remember"}]},{"t":"p","runs":[{"text":"Doubling the users on vLLM costs almost nothing per user. Doubling the users on ExLlamaV2 roughly halves everyone’s speed. Pick the engine for the shape of your load, not for the headline."}]}],"check":{"question":[{"text":"You benchmark one user on your 24 GB card: vLLM gives 60 tok/s, ExLlamaV2 gives 110 tok/s. You pick ExLlamaV2 for an internal tool that around 40 colleagues will use through the day. What is wrong with that reasoning?"}],"options":[{"text":[{"text":"The benchmark measured single-stream speed, but with 40 concurrent users vLLM’s continuous batching produces far more total throughput and lower queue times."}],"correct":true,"tempting":null},{"text":[{"text":"Nothing is wrong; per-token speed is a property of the engine and does not change with the number of users."}],"correct":false,"tempting":[{"text":"Assumes latency is independent of load, when the whole difference between these engines is how they behave under it."}]},{"text":[{"text":"Nothing is wrong, as long as you run 40 ExLlamaV2 processes so each user has their own."}],"correct":false,"tempting":[{"text":"Each process would need its own copy of the weights and KV cache, so the card runs out long before 40."}]},{"text":[{"text":"vLLM’s advantage is only its kernels, so switching ExLlamaV2 to the same quantisation format would close the gap."}],"correct":false,"tempting":[{"text":"Credits throughput to kernel speed when it comes from batching and KV memory management."}]}],"order":[3,0,2,1],"why":null},"prev":{"slug":"speech-and-vision-locally","title":"Speech, vision and the rest of the local stack","minutes":9},"next":{"slug":"continuous-batching","title":"Continuous batching, and the queue underneath it","minutes":9}}}