Every engine claims to be fastest. Most of those claims are true, under conditions the benchmark does not print. Before comparing anything, fix which number you mean.
Three different speeds
Single-stream decode — tokens per second for one user with nothing else running. This is what you feel when you chat alone.
Aggregate throughput — tokens per second summed across all concurrent users. This is what determines cost per token and how many people one GPU serves.
Time to first token — queue wait plus prompt processing. This is what users call "slow" even when generation is fast.
These trade against each other. Batching many requests together raises aggregate throughput enormously while slightly raising each individual's latency. An engine tuned for one number will lose on another, and both benchmarks are honest.
llama.cpp
Runs on CPU, CUDA, Metal, Vulkan, ROCm and SYCL, and can split a model between GPU and CPU. GGUF only.
Wins when: the model does not fit in VRAM, the hardware is not NVIDIA, you are on a Mac, or you have one or a few users. It is the only engine on this list that runs at all in most of those situations.
Weakness: it has continuous batching (--parallel, -cb), but aggregate throughput at high concurrency is well below the purpose-built servers.
vLLM
The throughput default. Two ideas do most of the work. PagedAttention stores the KV cache in fixed-size blocks like virtual memory pages, so there is almost no fragmentation and no need to reserve worst-case context per request — which is what lets it hold many more concurrent sequences on the same card. Continuous batching admits and retires requests token by token instead of waiting for a whole batch to finish.
Wins when: you have concurrent users and the model fits entirely in GPU memory.
Weakness: no meaningful CPU offload, a heavy install, slow startup, and single-user latency no better than the alternatives.
ExLlamaV2 / EXL2
Hand-tuned kernels for consumer NVIDIA cards, variable-bit-rate quantisation, quantised KV cache, speculative decoding. TabbyAPI puts an OpenAI-compatible server in front of it.
Wins when: one user, one NVIDIA card, and you want the highest possible tokens per second for yourself.
Weakness: NVIDIA only, and it degrades far more sharply than vLLM as users are added.
TGI
Hugging Face's Rust and Python server. Feature set close to vLLM, with a stronger operational story — metrics, tracing, tight integration with the Hugging Face ecosystem.
Wins when: you already run on Hugging Face infrastructure and want the same stack locally.
SGLang
Built around RadixAttention: the KV cache is organised as a prefix tree, so any request sharing a prefix with an earlier one reuses that computation automatically. Also very fast constrained decoding for structured output.
Wins when: many requests share a long system prompt — agent loops, classification pipelines, anything with a fixed preamble — or when you need reliable JSON at high rates.
Ballpark numbers, and why you should not trust them
One 24 GB consumer card, an 8B model, roughly 4-bit:
| engine | one user | 64 concurrent, aggregate | |---|---|---| | llama.cpp | 55–70 tok/s | 300–600 tok/s | | ExLlamaV2 | 90–130 tok/s | falls off sharply | | vLLM | 55–75 tok/s | 1,200–2,500 tok/s |
These are order-of-magnitude guides on hardware that is not yours, with versions that have since changed. Measure: llama-bench for llama.cpp, vllm bench serve or the serving benchmark script for vLLM, and any HTTP load generator against the OpenAI endpoint for a fair cross-engine comparison.
The sentence to remember
Doubling the users on vLLM costs almost nothing per user. Doubling the users on ExLlamaV2 roughly halves everyone's speed. Pick the engine for the shape of your load, not for the headline.
Before you move on