The tuning checklist, in order

Change one thing, measure, keep or revert

Most tuning goes wrong the same way: several flags change at once, the result is better or worse, and nobody knows which one did it. The discipline below is boring and it is the whole method: establish a baseline, change one setting, measure the same way, and write down what happened.

Before anything, decide which number you are optimising. Single-user speed, aggregate throughput and p95 time to first token pull in different directions, and a change that helps one will often hurt another. If you cannot name the number, you cannot tune.

The order

1. Confirm the accelerator is being used. nvidia-smi during generation. A model silently on the CPU is a thirty-fold problem and no other setting matters until it is fixed.

2. Confirm the fast kernels are active. Read the startup log for the attention backend and the quantisation kernel. A fallback path here is worth several times any flag below.

3. Fit the model entirely in accelerator memory. Because of the harmonic-mean effect, moving the last few layers off the CPU is usually worth more than any other single change. Give up context or a quantisation tier to achieve it.

4. Set the context window to what you need. Not to the model’s maximum. Reclaimed cache memory becomes concurrency, or a better quant.

5. Quantise the KV cache to 8 bits. Nearly free quality-wise, halves the term that limits concurrency. Requires flash attention.

6. Enable prefix caching, then reorder the prompt for it. Fixed content first, variable content last, no timestamps in the preamble. On agent and pipeline workloads this is frequently the largest win available.

7. Enable chunked prefill if several people share the server and complain about stutter rather than general slowness.

8. Tune the batch caps. Raise the concurrent-sequence cap until p95 latency starts to climb faster than throughput rises. That crossover is your operating point.

9. Raise the memory fraction reserved for cache, vLLM’s --gpu-memory-utilization, commonly 0.90, until preemption stops appearing in the logs, leaving headroom for the rest of the system.

10. Add speculative decoding, but only if single-stream latency is the target and acceptance rate on your text exceeds roughly 0.6.

11. Only now consider a different engine. By this point you know which number is limiting you, which makes the engine choice obvious instead of a matter of opinion.

The settings people reach for that rarely help

More CPU threads on a GPU deployment. The CPU is not the bottleneck; extra threads add contention.

A larger batch cap when memory is already tight. Produces preemption, which is slower than a shorter queue.

Disabling chunked prefill to speed up prefill. It does, slightly, at the cost of everyone else’s latency. Only correct on a single-user machine.

A different front-end. Interfaces do not change the model. If two apps behave differently the difference is in the request, and the request is inspectable.

A newer engine version as a first move. Sometimes a large win, sometimes a regression. Do it deliberately, with a baseline, not as a guess.

Define the benchmark once, then never change it

Every step above depends on comparing a number to yesterday’s number, and that only works if the measurement is fixed. Write it down once and treat it as immovable.

A workable definition: twenty real prompts drawn from your actual traffic, with the prompt lengths they actually have; a fixed max_tokens; temperature 0 so sampling noise does not enter; a stated concurrency; and three repetitions reporting the median rather than the best.

Two details make the difference between a benchmark and a number.

Discard a warm-up run. The first request after a start pays for weight loading from disk and for kernel compilation, and including it makes every configuration look worse than it is by a wildly varying amount.

Fix the concurrency you care about, and say which. Single-stream latency and saturated throughput move in opposite directions under most changes. A configuration that improves one while damaging the other will look like an improvement or a regression depending only on which you happened to measure.

If the benchmark ever changes (new prompts, a different length cap) the old numbers are no longer comparable, so start a new section in the record rather than overwriting it. A performance history with one silent redefinition in it is worse than no history, because it will be trusted.

Keep the record

One file. Date, engine version, model and quant, every flag, and the four measured numbers. Ten lines per configuration.

This pays back in three ways. You can answer “was it always this slow” with evidence. You can find which upgrade cost you throughput. And when somebody asks why the server is configured the way it is, the answer is a measurement rather than a memory of an afternoon.