A load test whose numbers mean something

“It felt fast” is a measurement of one condition that will not recur

Trying a prompt yourself on an idle machine measures single-stream latency with a warm cache and no competition. If anyone else will use the server, that number predicts almost nothing about what they will experience.

A load test worth running produces four numbers: aggregate output tokens per second, median and 95th-percentile time to first token, 95th-percentile inter-token latency, and the error or timeout rate. Averages alone hide the problem: a mean time to first token of 400 ms is compatible with one user in twenty waiting nine seconds, and that user is the one who complains.

Closed loop and open loop

This distinction decides whether your test is realistic.

A closed-loop test keeps N workers busy, each sending a new request as soon as its previous one returns. Concurrency is fixed by construction. This measures capacity at a chosen concurrency, and it cannot show you a queue building, because the load automatically slows when the server does.

An open-loop test sends requests at a fixed arrival rate regardless of whether earlier ones have returned. This is how real traffic behaves, and it is the only way to see the point where the queue grows without bound. Push the rate up in steps and watch p95 time to first token: it stays flat, then bends, then goes vertical. The rate just before the bend is your real capacity.

If you run only one kind of test, run the open-loop one.

Use realistic prompts

The shape of the input changes the result more than most configuration does.

• Prompt length decides how much prefill competes with decoding. A test with 50-token prompts against production traffic averaging 3,000 tokens is measuring a different machine.
• Output length decides how long each sequence holds a slot. Fixing every response at 128 tokens hides the long-tail behaviour that causes real queueing.
• Prefix overlap decides your cache hit rate. Sending the same prompt a thousand times measures a cache hit rate of essentially 100%, which flatters the server enormously and is unlike any real workload.

Sample real prompts from your logs if you have them, or generate a distribution with the right mean and spread if you do not.

Running one

vLLM ships a benchmark that handles the details:

vllm bench serve \
  --backend openai-chat --model <model> \
  --dataset-name random --random-input-len 1024 --random-output-len 256 \
  --request-rate 8 --num-prompts 400

For a cross-engine comparison, drive the OpenAI endpoints of both with the same generic HTTP load generator and the same request file. That is the only comparison where the engines are the variable rather than the harness.

Whatever tool you use:

1. Warm up first and discard the initial requests. The first calls pay model load, kernel autotuning and an empty cache.
2. Run long enough to see a steady state: several minutes, not thirty seconds.
3. Watch the server, not just the client. GPU utilisation, memory used, and preemption or queue-depth counters. The client tells you what happened; the server tells you why.
4. Change one thing at a time. Batch cap, then cache fraction, then chunked prefill. Two changes at once produce a result you cannot attribute.

Reading the result

Throughput flat while latency climbs: you are past capacity and requests are queueing. Reduce arrival rate, add a replica, or accept the latency.

GPU utilisation low and latency high: you are not compute-bound. Look at cache capacity, at preemption, and at whether prefill is being serialised.

p95 far above the median: something is occasionally very slow. Usually long prompts blocking decode, which chunked prefill addresses, or preemption, which memory settings address.

Errors rising with rate: you have found a timeout or a queue limit. Decide deliberately whether to reject early or queue longer; both are legitimate and silence is not.

Write it down

Date, engine version, model, quant, context length, flags, and the four numbers. Engines change weekly and regressions are ordinary. A baseline file is how you find out that an upgrade cost you 30% of throughput, rather than vaguely suspecting it for a month.