{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"measuring-a-hardware-change","title":"Proving the change helped","url":"https://addaly.com/learn/hugging-face/measuring-a-hardware-change","position":27,"of":75,"minutes":8,"answer":"A hardware change has to be reported as peak memory, time to first token, throughput and quality on your own thirty examples together.","keyIdea":"A hardware change has to be reported as peak memory, time to first token, throughput and quality on your own thirty examples together, because each configuration trades along all four axes at once and any one number alone can be improved while the system gets worse."},"body":{"format":"markdown","text":"## Four numbers, not one\n\nEvery decision in this module (dtype, quantization, offloading, runtime, model size) trades along four axes at once. Report all four or you have not measured anything:\n\n1. **Peak memory.** The constraint that decides whether it runs at all.\n2. **Time to first token.** What an interactive user perceives.\n3. **Throughput**, in tokens or items per second. What a batch job costs.\n4. **Quality**, on your own examples. The one people skip, and the one that invalidates the rest.\n\nA change that halves memory and costs three points of accuracy is a good trade or a disaster depending on the application, and you cannot tell which without the fourth number.\n\n## A harness small enough to actually use\n\n```python\nimport time, torch\n\ndef measure(pipe, prompts, max_new_tokens=128):\n    _ = pipe(prompts[0], max_new_tokens=8)           # warm-up\n    if torch.cuda.is_available():\n        torch.cuda.synchronize(); torch.cuda.reset_peak_memory_stats()\n\n    t0 = time.perf_counter()\n    outs = [pipe(p, max_new_tokens=max_new_tokens) for p in prompts]\n    if torch.cuda.is_available():\n        torch.cuda.synchronize()\n    elapsed = time.perf_counter() - t0\n\n    peak = torch.cuda.max_memory_allocated()/1e9 if torch.cuda.is_available() else None\n    return {\"seconds\": round(elapsed, 2),\n            \"per_prompt\": round(elapsed/len(prompts), 3),\n            \"peak_gb\": round(peak, 2) if peak else None}\n```\n\nThree things this gets right that ad-hoc timing gets wrong:\n\n- **Warm-up excluded.** The first call allocates, compiles kernels and pages weights in. It is two to ten times slower and it is not representative.\n- **Synchronised.** CUDA work is queued asynchronously; without `synchronize()` you are timing Python.\n- **Peak, not current.** Out-of-memory happens at the peak, which may last milliseconds.\n\nMeasure time to first token separately, because it is a different quantity:\n\n```python\nt0 = time.perf_counter()\nfor chunk in streamer:\n    ttft = time.perf_counter() - t0\n    break\n```\n\n## Report the tail\n\nA mean of 300 ms with a 99th percentile of 4 seconds describes a system that one user in a hundred experiences as broken. Collect at least fifty runs and report the median and the 95th percentile. On shared free hardware the spread is wide and the mean is close to meaningless.\n\n## Measuring the quality side\n\nThis is twenty minutes of work and it is the part that gets skipped.\n\n1. Fix **thirty examples** from your real inputs, with expected outputs where an expected output exists.\n2. Run them through configuration A and configuration B.\n3. Score whatever is objective: exact match, valid JSON, correct number, correct label.\n4. Read the disagreements yourself. Thirty is few enough to read.\n\nThirty examples will not detect a two-point difference: it is too small a sample for that, and pretending otherwise is how people convince themselves of improvements that are noise. It will reliably detect the failure that matters here: a configuration that has stopped producing valid structured output, or lost a language, or started truncating. That is a large, visible effect and thirty items find it.\n\nKeep the examples in a file in the repository, not in a notebook cell. They will outlive the notebook.\n\n## Change one thing\n\nThe universal discipline, and the one people abandon when they are in a hurry. If you switch from bfloat16 on a T4 to 4-bit AWQ on an A10 with a new batch size, and it is faster, you have learned nothing transferable.\n\nWrite the configurations down before you run them:\n\n```\nA: bf16, batch 1, T4        → 8.1 tok/s, 14.2 GB, 30/30 valid JSON\nB: 4-bit nf4, batch 1, T4   → 6.4 tok/s,  5.1 GB, 27/30 valid JSON\nC: Q4_K_M GGUF, CPU laptop  → 7.9 tok/s,  4.4 GB, 29/30 valid JSON\n```\n\nThat table (three rows, four columns, an afternoon) is a decision. Note the result in row B that surprises people: 4-bit bitsandbytes is *slower* than bfloat16, because it buys memory rather than speed.\n\n## Do this now\n\nRun your current setup through the harness and write down the four numbers. Even with nothing to compare against, you now have a baseline, and every later claim that something got faster becomes checkable rather than remembered."},"check":{"question":"Why does timing generation on a GPU without calling `torch.cuda.synchronize()` typically produce a figure that is far too fast?","options":[{"text":"CUDA calls return once the work is queued, so the timer stops before the GPU has finished executing it.","correct":true},{"text":"The first call includes kernel compilation, which distorts the average unless it is excluded by a warm-up.","correct":false,"tempting":"Describes a real distortion that pushes timings the wrong way, slower, not faster, and is a separate issue from synchronisation."},{"text":"Without synchronisation the framework batches several generate calls together, so per-call time is divided across them.","correct":false,"tempting":"Invents implicit batching across independent calls."},{"text":"Peak memory statistics are only flushed on synchronisation, so timings taken before it read from a stale counter.","correct":false,"tempting":"Conflates memory reporting with wall-clock timing; the counter and the clock are unrelated."}]},"prev":{"slug":"small-models-and-old-machines","title":"What actually runs on eight gigabytes","url":"https://addaly.com/learn/hugging-face/small-models-and-old-machines"},"next":{"slug":"spaces-in-practice","title":"A Space is someone else's computer, already configured","url":"https://addaly.com/learn/hugging-face/spaces-in-practice"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Proving the change helped', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/measuring-a-hardware-change (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Proving the change helped","courseTitle":"Hugging Face, End to End","module":{"index":3,"slug":"running-it-on-what-you-have","title":"Running it on the hardware you actually have"},"answer":null,"keyIdea":"A hardware change has to be reported as peak memory, time to first token, throughput and quality on your own thirty examples together, because each configuration trades along all four axes at once and any one number alone can be improved while the system gets worse.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"four-numbers-not-one","runs":[{"text":"Four numbers, not one"}]},{"t":"p","runs":[{"text":"Every decision in this module (dtype, quantization, offloading, runtime, model size) trades along four axes at once. Report all four or you have not measured anything:"}]},{"t":"ol","start":1,"items":[[{"text":"Peak memory.","b":true},{"text":" The constraint that decides whether it runs at all."}],[{"text":"Time to first token.","b":true},{"text":" What an interactive user perceives."}],[{"text":"Throughput","b":true},{"text":", in tokens or items per second. What a batch job costs."}],[{"text":"Quality","b":true},{"text":", on your own examples. The one people skip, and the one that invalidates the rest."}]]},{"t":"p","runs":[{"text":"A change that halves memory and costs three points of accuracy is a good trade or a disaster depending on the application, and you cannot tell which without the fourth number."}]},{"t":"h2","id":"a-harness-small-enough-to-actually-use","runs":[{"text":"A harness small enough to actually use"}]},{"t":"code","lang":"python","label":"Python","text":"import time, torch\n\ndef measure(pipe, prompts, max_new_tokens=128):\n    _ = pipe(prompts[0], max_new_tokens=8)           # warm-up\n    if torch.cuda.is_available():\n        torch.cuda.synchronize(); torch.cuda.reset_peak_memory_stats()\n\n    t0 = time.perf_counter()\n    outs = [pipe(p, max_new_tokens=max_new_tokens) for p in prompts]\n    if torch.cuda.is_available():\n        torch.cuda.synchronize()\n    elapsed = time.perf_counter() - t0\n\n    peak = torch.cuda.max_memory_allocated()/1e9 if torch.cuda.is_available() else None\n    return {\"seconds\": round(elapsed, 2),\n            \"per_prompt\": round(elapsed/len(prompts), 3),\n            \"peak_gb\": round(peak, 2) if peak else None}"},{"t":"p","runs":[{"text":"Three things this gets right that ad-hoc timing gets wrong:"}]},{"t":"ul","items":[[{"text":"Warm-up excluded.","b":true},{"text":" The first call allocates, compiles kernels and pages weights in. It is two to ten times slower and it is not representative."}],[{"text":"Synchronised.","b":true},{"text":" CUDA work is queued asynchronously; without "},{"text":"synchronize()","code":true},{"text":" you are timing Python."}],[{"text":"Peak, not current.","b":true},{"text":" Out-of-memory happens at the peak, which may last milliseconds."}]]},{"t":"p","runs":[{"text":"Measure time to first token separately, because it is a different quantity:"}]},{"t":"code","lang":"python","label":"Python","text":"t0 = time.perf_counter()\nfor chunk in streamer:\n    ttft = time.perf_counter() - t0\n    break"},{"t":"h2","id":"report-the-tail","runs":[{"text":"Report the tail"}]},{"t":"p","runs":[{"text":"A mean of 300 ms with a 99th percentile of 4 seconds describes a system that one user in a hundred experiences as broken. Collect at least fifty runs and report the median and the 95th percentile. On shared free hardware the spread is wide and the mean is close to meaningless."}]},{"t":"h2","id":"measuring-the-quality-side","runs":[{"text":"Measuring the quality side"}]},{"t":"p","runs":[{"text":"This is twenty minutes of work and it is the part that gets skipped."}]},{"t":"ol","start":1,"items":[[{"text":"Fix "},{"text":"thirty examples","b":true},{"text":" from your real inputs, with expected outputs where an expected output exists."}],[{"text":"Run them through configuration A and configuration B."}],[{"text":"Score whatever is objective: exact match, valid JSON, correct number, correct label."}],[{"text":"Read the disagreements yourself. Thirty is few enough to read."}]]},{"t":"p","runs":[{"text":"Thirty examples will not detect a two-point difference: it is too small a sample for that, and pretending otherwise is how people convince themselves of improvements that are noise. It will reliably detect the failure that matters here: a configuration that has stopped producing valid structured output, or lost a language, or started truncating. That is a large, visible effect and thirty items find it."}]},{"t":"p","runs":[{"text":"Keep the examples in a file in the repository, not in a notebook cell. They will outlive the notebook."}]},{"t":"h2","id":"change-one-thing","runs":[{"text":"Change one thing"}]},{"t":"p","runs":[{"text":"The universal discipline, and the one people abandon when they are in a hurry. If you switch from bfloat16 on a T4 to 4-bit AWQ on an A10 with a new batch size, and it is faster, you have learned nothing transferable."}]},{"t":"p","runs":[{"text":"Write the configurations down before you run them:"}]},{"t":"code","lang":"","label":"Text","text":"A: bf16, batch 1, T4        → 8.1 tok/s, 14.2 GB, 30/30 valid JSON\nB: 4-bit nf4, batch 1, T4   → 6.4 tok/s,  5.1 GB, 27/30 valid JSON\nC: Q4_K_M GGUF, CPU laptop  → 7.9 tok/s,  4.4 GB, 29/30 valid JSON"},{"t":"p","runs":[{"text":"That table (three rows, four columns, an afternoon) is a decision. Note the result in row B that surprises people: 4-bit bitsandbytes is "},{"text":"slower","i":true},{"text":" than bfloat16, because it buys memory rather than speed."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Run your current setup through the harness and write down the four numbers. Even with nothing to compare against, you now have a baseline, and every later claim that something got faster becomes checkable rather than remembered."}]}],"check":{"question":[{"text":"Why does timing generation on a GPU without calling "},{"text":"torch.cuda.synchronize()","code":true},{"text":" typically produce a figure that is far too fast?"}],"options":[{"text":[{"text":"CUDA calls return once the work is queued, so the timer stops before the GPU has finished executing it."}],"correct":true,"tempting":null},{"text":[{"text":"The first call includes kernel compilation, which distorts the average unless it is excluded by a warm-up."}],"correct":false,"tempting":[{"text":"Describes a real distortion that pushes timings the wrong way, slower, not faster, and is a separate issue from synchronisation."}]},{"text":[{"text":"Without synchronisation the framework batches several generate calls together, so per-call time is divided across them."}],"correct":false,"tempting":[{"text":"Invents implicit batching across independent calls."}]},{"text":[{"text":"Peak memory statistics are only flushed on synchronisation, so timings taken before it read from a stale counter."}],"correct":false,"tempting":[{"text":"Conflates memory reporting with wall-clock timing; the counter and the clock are unrelated."}]}],"order":[2,0,1,3],"why":null},"prev":{"slug":"small-models-and-old-machines","title":"What actually runs on eight gigabytes","minutes":9},"next":{"slug":"spaces-in-practice","title":"A Space is someone else’s computer, already configured","minutes":9}}}