{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"load-testing-properly","title":"A load test whose numbers mean something","url":"https://addaly.com/learn/running-models-yourself/load-testing-properly","position":43,"of":82,"minutes":9,"answer":"Test with a fixed arrival rate rather than fixed concurrency, with prompt and output lengths drawn from real traffic.","keyIdea":"Test with a fixed arrival rate rather than fixed concurrency, with prompt and output lengths drawn from real traffic, and report p95 alongside the median: a closed-loop test on uniform prompts cannot show you the queue that will actually break the service."},"body":{"format":"markdown","text":"## \"It felt fast\" is a measurement of one condition that will not recur\n\nTrying a prompt yourself on an idle machine measures single-stream latency with a warm cache and no competition. If anyone else will use the server, that number predicts almost nothing about what they will experience.\n\nA load test worth running produces four numbers: aggregate output tokens per second, median and 95th-percentile time to first token, 95th-percentile inter-token latency, and the error or timeout rate. Averages alone hide the problem: a mean time to first token of 400 ms is compatible with one user in twenty waiting nine seconds, and that user is the one who complains.\n\n## Closed loop and open loop\n\nThis distinction decides whether your test is realistic.\n\nA **closed-loop** test keeps N workers busy, each sending a new request as soon as its previous one returns. Concurrency is fixed by construction. This measures capacity at a chosen concurrency, and it cannot show you a queue building, because the load automatically slows when the server does.\n\nAn **open-loop** test sends requests at a fixed arrival rate regardless of whether earlier ones have returned. This is how real traffic behaves, and it is the only way to see the point where the queue grows without bound. Push the rate up in steps and watch p95 time to first token: it stays flat, then bends, then goes vertical. The rate just before the bend is your real capacity.\n\nIf you run only one kind of test, run the open-loop one.\n\n## Use realistic prompts\n\nThe shape of the input changes the result more than most configuration does.\n\n- **Prompt length** decides how much prefill competes with decoding. A test with 50-token prompts against production traffic averaging 3,000 tokens is measuring a different machine.\n- **Output length** decides how long each sequence holds a slot. Fixing every response at 128 tokens hides the long-tail behaviour that causes real queueing.\n- **Prefix overlap** decides your cache hit rate. Sending the same prompt a thousand times measures a cache hit rate of essentially 100%, which flatters the server enormously and is unlike any real workload.\n\nSample real prompts from your logs if you have them, or generate a distribution with the right mean and spread if you do not.\n\n## Running one\n\nvLLM ships a benchmark that handles the details:\n\n```bash\nvllm bench serve \\\n  --backend openai-chat --model <model> \\\n  --dataset-name random --random-input-len 1024 --random-output-len 256 \\\n  --request-rate 8 --num-prompts 400\n```\n\nFor a cross-engine comparison, drive the OpenAI endpoints of both with the same generic HTTP load generator and the same request file. That is the only comparison where the engines are the variable rather than the harness.\n\nWhatever tool you use:\n\n1. **Warm up first** and discard the initial requests. The first calls pay model load, kernel autotuning and an empty cache.\n2. **Run long enough** to see a steady state: several minutes, not thirty seconds.\n3. **Watch the server, not just the client.** GPU utilisation, memory used, and preemption or queue-depth counters. The client tells you what happened; the server tells you why.\n4. **Change one thing at a time.** Batch cap, then cache fraction, then chunked prefill. Two changes at once produce a result you cannot attribute.\n\n## Reading the result\n\n**Throughput flat while latency climbs**: you are past capacity and requests are queueing. Reduce arrival rate, add a replica, or accept the latency.\n\n**GPU utilisation low and latency high**: you are not compute-bound. Look at cache capacity, at preemption, and at whether prefill is being serialised.\n\n**p95 far above the median**: something is occasionally very slow. Usually long prompts blocking decode, which chunked prefill addresses, or preemption, which memory settings address.\n\n**Errors rising with rate**: you have found a timeout or a queue limit. Decide deliberately whether to reject early or queue longer; both are legitimate and silence is not.\n\n## Write it down\n\nDate, engine version, model, quant, context length, flags, and the four numbers. Engines change weekly and regressions are ordinary. A baseline file is how you find out that an upgrade cost you 30% of throughput, rather than vaguely suspecting it for a month."},"check":{"question":"A closed-loop test with 32 workers sending identical 2,000-token prompts reports excellent throughput and a p95 time to first token of 300 ms. In production at similar volume, users see multi-second waits. Which feature of the test most likely produced the flattering result?","options":[{"text":"Identical prompts gave a near-perfect prefix cache hit rate, so almost no prefill was performed: real traffic has diverse prompts and pays the full prefill cost.","correct":true},{"text":"Thirty-two workers is below production concurrency, so the test simply measured a lighter load.","correct":false,"tempting":"Would show as lower throughput too, and the stated volume was similar; it also does not explain the specific latency gap."},{"text":"Closed-loop testing under-reports latency because workers wait for responses, so the measured time excludes queue time.","correct":false,"tempting":"Half right about the mechanism (closed loop cannot show an unbounded queue) but the measured time does include queueing at that fixed concurrency."},{"text":"A 2,000-token prompt is longer than production averages, so the test was harder than reality and the gap must come from the client side.","correct":false,"tempting":"Draws the wrong conclusion from an assumption about production, and would predict production being faster rather than slower."}]},"prev":{"slug":"attention-kernels","title":"The kernels under everything: flash and paged attention","url":"https://addaly.com/learn/running-models-yourself/attention-kernels"},"next":{"slug":"the-tuning-checklist","title":"The tuning checklist, in order","url":"https://addaly.com/learn/running-models-yourself/the-tuning-checklist"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'A load test whose numbers mean something', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/load-testing-properly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"A load test whose numbers mean something","courseTitle":"Running Models Yourself","module":{"index":5,"slug":"serving-and-performance","title":"Serving engines and performance"},"answer":null,"keyIdea":"Test with a fixed arrival rate rather than fixed concurrency, with prompt and output lengths drawn from real traffic, and report p95 alongside the median: a closed-loop test on uniform prompts cannot show you the queue that will actually break the service.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"it-felt-fast-is-a-measurement-of-one-condition-that-will-not","runs":[{"text":"“It felt fast” is a measurement of one condition that will not recur"}]},{"t":"p","runs":[{"text":"Trying a prompt yourself on an idle machine measures single-stream latency with a warm cache and no competition. If anyone else will use the server, that number predicts almost nothing about what they will experience."}]},{"t":"p","runs":[{"text":"A load test worth running produces four numbers: aggregate output tokens per second, median and 95th-percentile time to first token, 95th-percentile inter-token latency, and the error or timeout rate. Averages alone hide the problem: a mean time to first token of 400 ms is compatible with one user in twenty waiting nine seconds, and that user is the one who complains."}]},{"t":"h2","id":"closed-loop-and-open-loop","runs":[{"text":"Closed loop and open loop"}]},{"t":"p","runs":[{"text":"This distinction decides whether your test is realistic."}]},{"t":"p","runs":[{"text":"A "},{"text":"closed-loop","b":true},{"text":" test keeps N workers busy, each sending a new request as soon as its previous one returns. Concurrency is fixed by construction. This measures capacity at a chosen concurrency, and it cannot show you a queue building, because the load automatically slows when the server does."}]},{"t":"p","runs":[{"text":"An "},{"text":"open-loop","b":true},{"text":" test sends requests at a fixed arrival rate regardless of whether earlier ones have returned. This is how real traffic behaves, and it is the only way to see the point where the queue grows without bound. Push the rate up in steps and watch p95 time to first token: it stays flat, then bends, then goes vertical. The rate just before the bend is your real capacity."}]},{"t":"p","runs":[{"text":"If you run only one kind of test, run the open-loop one."}]},{"t":"h2","id":"use-realistic-prompts","runs":[{"text":"Use realistic prompts"}]},{"t":"p","runs":[{"text":"The shape of the input changes the result more than most configuration does."}]},{"t":"ul","items":[[{"text":"Prompt length","b":true},{"text":" decides how much prefill competes with decoding. A test with 50-token prompts against production traffic averaging 3,000 tokens is measuring a different machine."}],[{"text":"Output length","b":true},{"text":" decides how long each sequence holds a slot. Fixing every response at 128 tokens hides the long-tail behaviour that causes real queueing."}],[{"text":"Prefix overlap","b":true},{"text":" decides your cache hit rate. Sending the same prompt a thousand times measures a cache hit rate of essentially 100%, which flatters the server enormously and is unlike any real workload."}]]},{"t":"p","runs":[{"text":"Sample real prompts from your logs if you have them, or generate a distribution with the right mean and spread if you do not."}]},{"t":"h2","id":"running-one","runs":[{"text":"Running one"}]},{"t":"p","runs":[{"text":"vLLM ships a benchmark that handles the details:"}]},{"t":"code","lang":"bash","label":"Bash","text":"vllm bench serve \\\n  --backend openai-chat --model <model> \\\n  --dataset-name random --random-input-len 1024 --random-output-len 256 \\\n  --request-rate 8 --num-prompts 400"},{"t":"p","runs":[{"text":"For a cross-engine comparison, drive the OpenAI endpoints of both with the same generic HTTP load generator and the same request file. That is the only comparison where the engines are the variable rather than the harness."}]},{"t":"p","runs":[{"text":"Whatever tool you use:"}]},{"t":"ol","start":1,"items":[[{"text":"Warm up first","b":true},{"text":" and discard the initial requests. The first calls pay model load, kernel autotuning and an empty cache."}],[{"text":"Run long enough","b":true},{"text":" to see a steady state: several minutes, not thirty seconds."}],[{"text":"Watch the server, not just the client.","b":true},{"text":" GPU utilisation, memory used, and preemption or queue-depth counters. The client tells you what happened; the server tells you why."}],[{"text":"Change one thing at a time.","b":true},{"text":" Batch cap, then cache fraction, then chunked prefill. Two changes at once produce a result you cannot attribute."}]]},{"t":"h2","id":"reading-the-result","runs":[{"text":"Reading the result"}]},{"t":"p","runs":[{"text":"Throughput flat while latency climbs","b":true},{"text":": you are past capacity and requests are queueing. Reduce arrival rate, add a replica, or accept the latency."}]},{"t":"p","runs":[{"text":"GPU utilisation low and latency high","b":true},{"text":": you are not compute-bound. Look at cache capacity, at preemption, and at whether prefill is being serialised."}]},{"t":"p","runs":[{"text":"p95 far above the median","b":true},{"text":": something is occasionally very slow. Usually long prompts blocking decode, which chunked prefill addresses, or preemption, which memory settings address."}]},{"t":"p","runs":[{"text":"Errors rising with rate","b":true},{"text":": you have found a timeout or a queue limit. Decide deliberately whether to reject early or queue longer; both are legitimate and silence is not."}]},{"t":"h2","id":"write-it-down","runs":[{"text":"Write it down"}]},{"t":"p","runs":[{"text":"Date, engine version, model, quant, context length, flags, and the four numbers. Engines change weekly and regressions are ordinary. A baseline file is how you find out that an upgrade cost you 30% of throughput, rather than vaguely suspecting it for a month."}]}],"check":{"question":[{"text":"A closed-loop test with 32 workers sending identical 2,000-token prompts reports excellent throughput and a p95 time to first token of 300 ms. In production at similar volume, users see multi-second waits. Which feature of the test most likely produced the flattering result?"}],"options":[{"text":[{"text":"Identical prompts gave a near-perfect prefix cache hit rate, so almost no prefill was performed: real traffic has diverse prompts and pays the full prefill cost."}],"correct":true,"tempting":null},{"text":[{"text":"Thirty-two workers is below production concurrency, so the test simply measured a lighter load."}],"correct":false,"tempting":[{"text":"Would show as lower throughput too, and the stated volume was similar; it also does not explain the specific latency gap."}]},{"text":[{"text":"Closed-loop testing under-reports latency because workers wait for responses, so the measured time excludes queue time."}],"correct":false,"tempting":[{"text":"Half right about the mechanism (closed loop cannot show an unbounded queue) but the measured time does include queueing at that fixed concurrency."}]},{"text":[{"text":"A 2,000-token prompt is longer than production averages, so the test was harder than reality and the gap must come from the client side."}],"correct":false,"tempting":[{"text":"Draws the wrong conclusion from an assumption about production, and would predict production being faster rather than slower."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"attention-kernels","title":"The kernels under everything: flash and paged attention","minutes":9},"next":{"slug":"the-tuning-checklist","title":"The tuning checklist, in order","minutes":8}}}