{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"the-tuning-checklist","title":"The tuning checklist, in order","url":"https://addaly.com/learn/running-models-yourself/the-tuning-checklist","position":44,"of":82,"minutes":8,"answer":"Tune in order (accelerator active, fast kernels, model fully resident, context sized, cache quantised, prefix caching, then batching) changing one thing at a time against a named number and writing down what happened.","keyIdea":"Tune in order (accelerator active, fast kernels, model fully resident, context sized, cache quantised, prefix caching, then batching) changing one thing at a time against a named number and writing down what happened."},"body":{"format":"markdown","text":"## Change one thing, measure, keep or revert\n\nMost tuning goes wrong the same way: several flags change at once, the result is better or worse, and nobody knows which one did it. The discipline below is boring and it is the whole method: establish a baseline, change one setting, measure the same way, and write down what happened.\n\nBefore anything, decide which number you are optimising. Single-user speed, aggregate throughput and p95 time to first token pull in different directions, and a change that helps one will often hurt another. If you cannot name the number, you cannot tune.\n\n## The order\n\n**1. Confirm the accelerator is being used.** `nvidia-smi` during generation. A model silently on the CPU is a thirty-fold problem and no other setting matters until it is fixed.\n\n**2. Confirm the fast kernels are active.** Read the startup log for the attention backend and the quantisation kernel. A fallback path here is worth several times any flag below.\n\n**3. Fit the model entirely in accelerator memory.** Because of the harmonic-mean effect, moving the last few layers off the CPU is usually worth more than any other single change. Give up context or a quantisation tier to achieve it.\n\n**4. Set the context window to what you need.** Not to the model's maximum. Reclaimed cache memory becomes concurrency, or a better quant.\n\n**5. Quantise the KV cache to 8 bits.** Nearly free quality-wise, halves the term that limits concurrency. Requires flash attention.\n\n**6. Enable prefix caching, then reorder the prompt for it.** Fixed content first, variable content last, no timestamps in the preamble. On agent and pipeline workloads this is frequently the largest win available.\n\n**7. Enable chunked prefill** if several people share the server and complain about stutter rather than general slowness.\n\n**8. Tune the batch caps.** Raise the concurrent-sequence cap until p95 latency starts to climb faster than throughput rises. That crossover is your operating point.\n\n**9. Raise the memory fraction reserved for cache**, vLLM's `--gpu-memory-utilization`, commonly 0.90, until preemption stops appearing in the logs, leaving headroom for the rest of the system.\n\n**10. Add speculative decoding**, but only if single-stream latency is the target and acceptance rate on your text exceeds roughly 0.6.\n\n**11. Only now consider a different engine.** By this point you know which number is limiting you, which makes the engine choice obvious instead of a matter of opinion.\n\n## The settings people reach for that rarely help\n\n**More CPU threads on a GPU deployment.** The CPU is not the bottleneck; extra threads add contention.\n\n**A larger batch cap when memory is already tight.** Produces preemption, which is slower than a shorter queue.\n\n**Disabling chunked prefill to speed up prefill.** It does, slightly, at the cost of everyone else's latency. Only correct on a single-user machine.\n\n**A different front-end.** Interfaces do not change the model. If two apps behave differently the difference is in the request, and the request is inspectable.\n\n**A newer engine version as a first move.** Sometimes a large win, sometimes a regression. Do it deliberately, with a baseline, not as a guess.\n\n## Define the benchmark once, then never change it\n\nEvery step above depends on comparing a number to yesterday's number, and that only works if the measurement is fixed. Write it down once and treat it as immovable.\n\nA workable definition: twenty real prompts drawn from your actual traffic, with the prompt lengths they actually have; a fixed `max_tokens`; temperature 0 so sampling noise does not enter; a stated concurrency; and three repetitions reporting the median rather than the best.\n\nTwo details make the difference between a benchmark and a number.\n\n**Discard a warm-up run.** The first request after a start pays for weight loading from disk and for kernel compilation, and including it makes every configuration look worse than it is by a wildly varying amount.\n\n**Fix the concurrency you care about, and say which.** Single-stream latency and saturated throughput move in opposite directions under most changes. A configuration that improves one while damaging the other will look like an improvement or a regression depending only on which you happened to measure.\n\nIf the benchmark ever changes (new prompts, a different length cap) the old numbers are no longer comparable, so start a new section in the record rather than overwriting it. A performance history with one silent redefinition in it is worse than no history, because it will be trusted.\n\n## Keep the record\n\nOne file. Date, engine version, model and quant, every flag, and the four measured numbers. Ten lines per configuration.\n\nThis pays back in three ways. You can answer \"was it always this slow\" with evidence. You can find which upgrade cost you throughput. And when somebody asks why the server is configured the way it is, the answer is a measurement rather than a memory of an afternoon."},"check":{"question":"A server handling many users with a long shared system prompt has p95 time to first token of nine seconds. The team's plan is to raise the concurrent-sequence cap. Why is that unlikely to help, and what should come first?","options":[{"text":"The cap is not the constraint; more concurrent sequences means more cache pressure and probably preemption. Enabling prefix caching removes most of the repeated prefill that is causing the wait.","correct":true},{"text":"Raising the cap helps throughput but not latency, so speculative decoding should come first to reduce per-request time.","correct":false,"tempting":"Speculation improves decode, while the symptom is time to first token, which is dominated by prefill and queueing."},{"text":"Time to first token is set by model load time, so keeping the model resident with a longer keep-alive is the fix.","correct":false,"tempting":"Confuses a one-off startup cost with a per-request one; the model stays loaded across requests."},{"text":"The context window is too large, and reducing it will cut prefill work proportionally.","correct":false,"tempting":"Confuses allocated window with tokens actually sent; shrinking the window frees memory but does not reduce the prompt being processed."}]},"prev":{"slug":"load-testing-properly","title":"A load test whose numbers mean something","url":"https://addaly.com/learn/running-models-yourself/load-testing-properly"},"next":{"slug":"running-several-models","title":"One machine, several models","url":"https://addaly.com/learn/running-models-yourself/running-several-models"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'The tuning checklist, in order', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/the-tuning-checklist (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"The tuning checklist, in order","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Tune in order (accelerator active, fast kernels, model fully resident, context sized, cache quantised, prefix caching, then batching) changing one thing at a time against a named number and writing down what happened.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"change-one-thing-measure-keep-or-revert","runs":[{"text":"Change one thing, measure, keep or revert"}]},{"t":"p","runs":[{"text":"Most tuning goes wrong the same way: several flags change at once, the result is better or worse, and nobody knows which one did it. The discipline below is boring and it is the whole method: establish a baseline, change one setting, measure the same way, and write down what happened."}]},{"t":"p","runs":[{"text":"Before anything, decide which number you are optimising. Single-user speed, aggregate throughput and p95 time to first token pull in different directions, and a change that helps one will often hurt another. If you cannot name the number, you cannot tune."}]},{"t":"h2","id":"the-order","runs":[{"text":"The order"}]},{"t":"p","runs":[{"text":"1. Confirm the accelerator is being used.","b":true},{"text":" "},{"text":"nvidia-smi","code":true},{"text":" during generation. A model silently on the CPU is a thirty-fold problem and no other setting matters until it is fixed."}]},{"t":"p","runs":[{"text":"2. Confirm the fast kernels are active.","b":true},{"text":" Read the startup log for the attention backend and the quantisation kernel. A fallback path here is worth several times any flag below."}]},{"t":"p","runs":[{"text":"3. Fit the model entirely in accelerator memory.","b":true},{"text":" Because of the harmonic-mean effect, moving the last few layers off the CPU is usually worth more than any other single change. Give up context or a quantisation tier to achieve it."}]},{"t":"p","runs":[{"text":"4. Set the context window to what you need.","b":true},{"text":" Not to the model’s maximum. Reclaimed cache memory becomes concurrency, or a better quant."}]},{"t":"p","runs":[{"text":"5. Quantise the KV cache to 8 bits.","b":true},{"text":" Nearly free quality-wise, halves the term that limits concurrency. Requires flash attention."}]},{"t":"p","runs":[{"text":"6. Enable prefix caching, then reorder the prompt for it.","b":true},{"text":" Fixed content first, variable content last, no timestamps in the preamble. On agent and pipeline workloads this is frequently the largest win available."}]},{"t":"p","runs":[{"text":"7. Enable chunked prefill","b":true},{"text":" if several people share the server and complain about stutter rather than general slowness."}]},{"t":"p","runs":[{"text":"8. Tune the batch caps.","b":true},{"text":" Raise the concurrent-sequence cap until p95 latency starts to climb faster than throughput rises. That crossover is your operating point."}]},{"t":"p","runs":[{"text":"9. Raise the memory fraction reserved for cache","b":true},{"text":", vLLM’s "},{"text":"--gpu-memory-utilization","code":true},{"text":", commonly 0.90, until preemption stops appearing in the logs, leaving headroom for the rest of the system."}]},{"t":"p","runs":[{"text":"10. Add speculative decoding","b":true},{"text":", but only if single-stream latency is the target and acceptance rate on your text exceeds roughly 0.6."}]},{"t":"p","runs":[{"text":"11. Only now consider a different engine.","b":true},{"text":" By this point you know which number is limiting you, which makes the engine choice obvious instead of a matter of opinion."}]},{"t":"h2","id":"the-settings-people-reach-for-that-rarely-help","runs":[{"text":"The settings people reach for that rarely help"}]},{"t":"p","runs":[{"text":"More CPU threads on a GPU deployment.","b":true},{"text":" The CPU is not the bottleneck; extra threads add contention."}]},{"t":"p","runs":[{"text":"A larger batch cap when memory is already tight.","b":true},{"text":" Produces preemption, which is slower than a shorter queue."}]},{"t":"p","runs":[{"text":"Disabling chunked prefill to speed up prefill.","b":true},{"text":" It does, slightly, at the cost of everyone else’s latency. Only correct on a single-user machine."}]},{"t":"p","runs":[{"text":"A different front-end.","b":true},{"text":" Interfaces do not change the model. If two apps behave differently the difference is in the request, and the request is inspectable."}]},{"t":"p","runs":[{"text":"A newer engine version as a first move.","b":true},{"text":" Sometimes a large win, sometimes a regression. Do it deliberately, with a baseline, not as a guess."}]},{"t":"h2","id":"define-the-benchmark-once-then-never-change-it","runs":[{"text":"Define the benchmark once, then never change it"}]},{"t":"p","runs":[{"text":"Every step above depends on comparing a number to yesterday’s number, and that only works if the measurement is fixed. Write it down once and treat it as immovable."}]},{"t":"p","runs":[{"text":"A workable definition: twenty real prompts drawn from your actual traffic, with the prompt lengths they actually have; a fixed "},{"text":"max_tokens","code":true},{"text":"; temperature 0 so sampling noise does not enter; a stated concurrency; and three repetitions reporting the median rather than the best."}]},{"t":"p","runs":[{"text":"Two details make the difference between a benchmark and a number."}]},{"t":"p","runs":[{"text":"Discard a warm-up run.","b":true},{"text":" The first request after a start pays for weight loading from disk and for kernel compilation, and including it makes every configuration look worse than it is by a wildly varying amount."}]},{"t":"p","runs":[{"text":"Fix the concurrency you care about, and say which.","b":true},{"text":" Single-stream latency and saturated throughput move in opposite directions under most changes. A configuration that improves one while damaging the other will look like an improvement or a regression depending only on which you happened to measure."}]},{"t":"p","runs":[{"text":"If the benchmark ever changes (new prompts, a different length cap) the old numbers are no longer comparable, so start a new section in the record rather than overwriting it. A performance history with one silent redefinition in it is worse than no history, because it will be trusted."}]},{"t":"h2","id":"keep-the-record","runs":[{"text":"Keep the record"}]},{"t":"p","runs":[{"text":"One file. Date, engine version, model and quant, every flag, and the four measured numbers. Ten lines per configuration."}]},{"t":"p","runs":[{"text":"This pays back in three ways. You can answer “was it always this slow” with evidence. You can find which upgrade cost you throughput. And when somebody asks why the server is configured the way it is, the answer is a measurement rather than a memory of an afternoon."}]}],"check":{"question":[{"text":"A server handling many users with a long shared system prompt has p95 time to first token of nine seconds. The team’s plan is to raise the concurrent-sequence cap. Why is that unlikely to help, and what should come first?"}],"options":[{"text":[{"text":"The cap is not the constraint; more concurrent sequences means more cache pressure and probably preemption. Enabling prefix caching removes most of the repeated prefill that is causing the wait."}],"correct":true,"tempting":null},{"text":[{"text":"Raising the cap helps throughput but not latency, so speculative decoding should come first to reduce per-request time."}],"correct":false,"tempting":[{"text":"Speculation improves decode, while the symptom is time to first token, which is dominated by prefill and queueing."}]},{"text":[{"text":"Time to first token is set by model load time, so keeping the model resident with a longer keep-alive is the fix."}],"correct":false,"tempting":[{"text":"Confuses a one-off startup cost with a per-request one; the model stays loaded across requests."}]},{"text":[{"text":"The context window is too large, and reducing it will cut prefill work proportionally."}],"correct":false,"tempting":[{"text":"Confuses allocated window with tokens actually sent; shrinking the window frees memory but does not reduce the prompt being processed."}]}],"order":[2,1,0,3],"why":null},"prev":{"slug":"load-testing-properly","title":"A load test whose numbers mean something","minutes":9},"next":{"slug":"running-several-models","title":"One machine, several models","minutes":9}}}