Serving engines, and what “faster” means

Every engine claims to be fastest. Most of those claims are true, under conditions the benchmark does not print. Before comparing anything, fix which number you mean.

Three different speeds

Single-stream decode: tokens per second for one user with nothing else running. This is what you feel when you chat alone.

Aggregate throughput: tokens per second summed across all concurrent users. This is what determines cost per token and how many people one GPU serves.

Time to first token: queue wait plus prompt processing. This is what users call “slow” even when generation is fast.

These trade against each other. Batching many requests together raises aggregate throughput enormously while slightly raising each individual’s latency. An engine tuned for one number will lose on another, and both benchmarks are honest.

llama.cpp

Runs on CPU, CUDA, Metal, Vulkan, ROCm and SYCL, and can split a model between GPU and CPU. GGUF only.

Wins when: the model does not fit in VRAM, the hardware is not NVIDIA, you are on a Mac, or you have one or a few users. It is the only engine on this list that runs at all in most of those situations.

Weakness: it has continuous batching (--parallel, -cb), but aggregate throughput at high concurrency is well below the purpose-built servers.

vLLM

The throughput default. Two ideas do most of the work. PagedAttention stores the KV cache in fixed-size blocks like virtual memory pages, so there is almost no fragmentation and no need to reserve worst-case context per request, which is what lets it hold many more concurrent sequences on the same card. Continuous batching admits and retires requests token by token instead of waiting for a whole batch to finish.

Wins when: you have concurrent users and the model fits entirely in GPU memory.

Weakness: no meaningful CPU offload, a heavy install, slow startup, and single-user latency no better than the alternatives.

ExLlamaV2 / EXL2

Hand-tuned kernels for consumer NVIDIA cards, variable-bit-rate quantisation, quantised KV cache, speculative decoding. TabbyAPI puts an OpenAI-compatible server in front of it.

Wins when: one user, one NVIDIA card, and you want the highest possible tokens per second for yourself.

Weakness: NVIDIA only, and it degrades far more sharply than vLLM as users are added.

TGI

Hugging Face’s Rust and Python server. Feature set close to vLLM, with a stronger operational story: metrics, tracing, tight integration with the Hugging Face ecosystem.

Status: in maintenance mode since December 2025 (minor bug fixes only), and its repository is now archived. Hugging Face itself recommends vLLM or SGLang instead, so use TGI only where it is already deployed.

SGLang

Built around RadixAttention: the KV cache is organised as a prefix tree, so any request sharing a prefix with an earlier one reuses that computation automatically. Also very fast constrained decoding for structured output.

Wins when: many requests share a long system prompt (agent loops, classification pipelines, anything with a fixed preamble) or when you need reliable JSON at high rates.

Ballpark numbers, and why you should not trust them

Each project publishes its own single-user figures. On an RTX 4090 (a 24 GB consumer card) with Llama 2 7B at about 4 bits, llama.cpp’s CUDA scoreboard shows about 186 tok/s (Q4_0) and ExLlamaV2’s README about 211 tok/s (4.0 bits per weight). Single-user speeds sit within tens of per cent of each other; the large differences appear in aggregate throughput under concurrency, where batching servers such as vLLM pull far ahead and ExLlamaV2 falls off sharply.

These are order-of-magnitude guides on hardware that is not yours, with versions that have since changed. Measure: llama-bench for llama.cpp, vllm bench serve or the serving benchmark script for vLLM, and any HTTP load generator against the OpenAI endpoint for a fair cross-engine comparison.

The sentence to remember

Doubling the users on vLLM costs almost nothing per user. Doubling the users on ExLlamaV2 roughly halves everyone’s speed. Pick the engine for the shape of your load, not for the headline.