{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"measuring-your-machine","title":"Measuring what you actually have","url":"https://addaly.com/learn/running-models-yourself/measuring-your-machine","position":17,"of":82,"minutes":8,"answer":"Ten minutes of measurement on your own machine (prefill and decode separately, memory used, temperature over time) replaces every published benchmark and diagnoses most local performance problems immediately.","keyIdea":"Ten minutes of measurement on your own machine (prefill and decode separately, memory used, temperature over time) replaces every published benchmark and diagnoses most local performance problems immediately."},"body":{"format":"markdown","text":"## Published figures are not your figures\n\nEvery number in a review was measured on a different machine with a different build, a different driver, a different quant and a different context length. Your machine has its own answer and it takes ten minutes to get. Do this once, write the numbers down, and you will never again have to guess whether something is working properly.\n\n## The four measurements worth having\n\n**1. Prefill and decode, separately.**\n\n```bash\nllama-bench -m ~/models/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -ngl 99\n```\n\nThis prints `pp512` (prompt processing) and `tg128` (token generation) with a standard deviation over several runs. Run it for each model and each quant you are considering; it is far more informative than any table you can read.\n\n**2. Your usable memory bandwidth.** Compare the measured decode rate against the theoretical ceiling. If a 4.9 GB model generates 40 tokens per second, you are achieving about 196 GB/s of effective bandwidth. If your card is rated at 360, you are at 54%, low, and worth investigating. Good implementations reach 60 to 75%.\n\n**3. What is actually loaded and where.**\n\n```bash\nnvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,power.draw,temperature.gpu \\\n  --format=csv -l 1\n```\n\nOn AMD, `rocm-smi`. On a Mac, `sudo powermetrics --samplers gpu_power -i 1000`. Watch during a real generation. Two diagnoses come straight out of it: GPU utilisation sitting low during decode means you are memory-bound (expected) or waiting on the CPU (not expected), and memory used far below memory total means you have room for more context or a better quant.\n\n**4. Whether it is throttling.** Run a two-minute generation and watch temperature and clock speed. A laptop that starts at 12 tokens per second and settles at 7 is thermally limited. That is a cooling problem, not a model problem, and no configuration change will fix it.\n\n## Reading the tell-tale symptoms\n\n**Speed halves after the first few hundred tokens.** The KV cache is growing and attention over a longer sequence costs more. Normal. It becomes abnormal if the fall is sudden and large, which usually means the cache has spilled and something is being recomputed or swapped.\n\n**First token takes many seconds, then output is fast.** Prefill-bound. Shorten the prompt or enable prefix caching.\n\n**Everything is slow and the GPU shows 0% used.** The model is not on the GPU at all. Check the build has the right backend compiled in, and check the offload flag. This is the single most common local-inference bug, and the diagnosis is one `nvidia-smi` away.\n\n**Loads instantly, then runs at a token every few seconds.** Memory-mapped file being paged from disk because it does not fit in RAM. Confirm with `--no-mmap`, which will fail honestly.\n\n**Fine alone, terrible with a second process running.** You are sharing memory or bandwidth with something else. A browser with hardware acceleration and forty tabs takes real VRAM.\n\n## Build a habit, not a spreadsheet\n\nKeep one file with: your machine, the model, the quant, the context length, prefill rate, decode rate, and memory used. Five lines per configuration. When you later ask \"is 14 tokens per second reasonable here?\", the answer is in the file rather than in an argument.\n\nRe-measure after driver updates and engine upgrades. Both regress occasionally, and a written baseline is the only way you will notice rather than vaguely suspecting.\n\n## Derive your effective bandwidth\n\nThere is one derived figure worth computing, because it tells you whether the machine is performing as it should rather than merely how fast it is. Decode reads every weight once per token, so:\n\n```\neffective bandwidth (GB/s) = model file size in GB x tokens per second\n```\n\nA 4.9 GB model generating 38 tokens per second is moving about 186 GB/s. Compare that with the card's published memory bandwidth. A ratio between roughly 60 and 80% is normal and healthy. Much below that and something else is the constraint (layers spilling to host memory, a thermal cap, a power limit, or a build without the right kernels) and it is worth finding before you buy anything.\n\nThe same arithmetic run backwards is how you predict a model you have not downloaded. A 19 GB file on a card with 400 GB/s of usable bandwidth will not exceed about 21 tokens per second for one stream, whatever the engine. If that is too slow for the use, no configuration will rescue it and the answer is a smaller model.\n\n## One warning about comparing engines\n\nA benchmark comparison between two engines is only fair if the model, the quantisation, the context length, the batch size and the sampler are the same. They usually are not: comparing llama.cpp at Q4_K_M against vLLM at AWQ-4bit is comparing two different models. If you must compare, put both behind their OpenAI-compatible endpoints and drive them with the same HTTP load generator and the same requests. That is the only comparison that answers a question you actually have."},"check":{"question":"A user reports that their new model \"loads instantly but produces about one token every three seconds\" on a machine with 16 GB of RAM and no GPU, running a 13 GB quantised file. What is the most likely cause?","options":[{"text":"The context window is set too high, so the KV cache is consuming the memory the weights need.","correct":false,"tempting":"Picks a real failure with the wrong magnitude; a cache at ordinary context lengths is under a gigabyte against a 13 GB file."},{"text":"The instant load indicates memory mapping succeeded but the file does not fit alongside the operating system, so pages are being read from disk continuously.","correct":true},{"text":"Thread count defaults are wrong, and setting threads to the physical core count will restore normal speed.","correct":false,"tempting":"Would explain a two-fold difference, not a hundred-fold one; threading cannot take a model from disk speed to memory speed."},{"text":"The quantisation format is unsupported by the build, so the engine is falling back to a slow reference implementation.","correct":false,"tempting":"Plausible in principle but does not explain the instant load, which is the specific clue that the file was mapped rather than read."}]},"prev":{"slug":"more-than-one-gpu","title":"More than one GPU","url":"https://addaly.com/learn/running-models-yourself/more-than-one-gpu"},"next":{"slug":"when-it-does-not-fit","title":"When it does not fit: the ordered remedies","url":"https://addaly.com/learn/running-models-yourself/when-it-does-not-fit"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Measuring what you actually have', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/measuring-your-machine (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Measuring what you actually have","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"Ten minutes of measurement on your own machine (prefill and decode separately, memory used, temperature over time) replaces every published benchmark and diagnoses most local performance problems immediately.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"published-figures-are-not-your-figures","runs":[{"text":"Published figures are not your figures"}]},{"t":"p","runs":[{"text":"Every number in a review was measured on a different machine with a different build, a different driver, a different quant and a different context length. Your machine has its own answer and it takes ten minutes to get. Do this once, write the numbers down, and you will never again have to guess whether something is working properly."}]},{"t":"h2","id":"the-four-measurements-worth-having","runs":[{"text":"The four measurements worth having"}]},{"t":"p","runs":[{"text":"1. Prefill and decode, separately.","b":true}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-bench -m ~/models/Qwen3-8B-Q4_K_M.gguf -p 512 -n 128 -ngl 99"},{"t":"p","runs":[{"text":"This prints "},{"text":"pp512","code":true},{"text":" (prompt processing) and "},{"text":"tg128","code":true},{"text":" (token generation) with a standard deviation over several runs. Run it for each model and each quant you are considering; it is far more informative than any table you can read."}]},{"t":"p","runs":[{"text":"2. Your usable memory bandwidth.","b":true},{"text":" Compare the measured decode rate against the theoretical ceiling. If a 4.9 GB model generates 40 tokens per second, you are achieving about 196 GB/s of effective bandwidth. If your card is rated at 360, you are at 54%, low, and worth investigating. Good implementations reach 60 to 75%."}]},{"t":"p","runs":[{"text":"3. What is actually loaded and where.","b":true}]},{"t":"code","lang":"bash","label":"Bash","text":"nvidia-smi --query-gpu=memory.used,memory.total,utilization.gpu,power.draw,temperature.gpu \\\n  --format=csv -l 1"},{"t":"p","runs":[{"text":"On AMD, "},{"text":"rocm-smi","code":true},{"text":". On a Mac, "},{"text":"sudo powermetrics --samplers gpu_power -i 1000","code":true},{"text":". Watch during a real generation. Two diagnoses come straight out of it: GPU utilisation sitting low during decode means you are memory-bound (expected) or waiting on the CPU (not expected), and memory used far below memory total means you have room for more context or a better quant."}]},{"t":"p","runs":[{"text":"4. Whether it is throttling.","b":true},{"text":" Run a two-minute generation and watch temperature and clock speed. A laptop that starts at 12 tokens per second and settles at 7 is thermally limited. That is a cooling problem, not a model problem, and no configuration change will fix it."}]},{"t":"h2","id":"reading-the-tell-tale-symptoms","runs":[{"text":"Reading the tell-tale symptoms"}]},{"t":"p","runs":[{"text":"Speed halves after the first few hundred tokens.","b":true},{"text":" The KV cache is growing and attention over a longer sequence costs more. Normal. It becomes abnormal if the fall is sudden and large, which usually means the cache has spilled and something is being recomputed or swapped."}]},{"t":"p","runs":[{"text":"First token takes many seconds, then output is fast.","b":true},{"text":" Prefill-bound. Shorten the prompt or enable prefix caching."}]},{"t":"p","runs":[{"text":"Everything is slow and the GPU shows 0% used.","b":true},{"text":" The model is not on the GPU at all. Check the build has the right backend compiled in, and check the offload flag. This is the single most common local-inference bug, and the diagnosis is one "},{"text":"nvidia-smi","code":true},{"text":" away."}]},{"t":"p","runs":[{"text":"Loads instantly, then runs at a token every few seconds.","b":true},{"text":" Memory-mapped file being paged from disk because it does not fit in RAM. Confirm with "},{"text":"--no-mmap","code":true},{"text":", which will fail honestly."}]},{"t":"p","runs":[{"text":"Fine alone, terrible with a second process running.","b":true},{"text":" You are sharing memory or bandwidth with something else. A browser with hardware acceleration and forty tabs takes real VRAM."}]},{"t":"h2","id":"build-a-habit-not-a-spreadsheet","runs":[{"text":"Build a habit, not a spreadsheet"}]},{"t":"p","runs":[{"text":"Keep one file with: your machine, the model, the quant, the context length, prefill rate, decode rate, and memory used. Five lines per configuration. When you later ask “is 14 tokens per second reasonable here?”, the answer is in the file rather than in an argument."}]},{"t":"p","runs":[{"text":"Re-measure after driver updates and engine upgrades. Both regress occasionally, and a written baseline is the only way you will notice rather than vaguely suspecting."}]},{"t":"h2","id":"derive-your-effective-bandwidth","runs":[{"text":"Derive your effective bandwidth"}]},{"t":"p","runs":[{"text":"There is one derived figure worth computing, because it tells you whether the machine is performing as it should rather than merely how fast it is. Decode reads every weight once per token, so:"}]},{"t":"code","lang":"","label":"Text","text":"effective bandwidth (GB/s) = model file size in GB x tokens per second"},{"t":"p","runs":[{"text":"A 4.9 GB model generating 38 tokens per second is moving about 186 GB/s. Compare that with the card’s published memory bandwidth. A ratio between roughly 60 and 80% is normal and healthy. Much below that and something else is the constraint (layers spilling to host memory, a thermal cap, a power limit, or a build without the right kernels) and it is worth finding before you buy anything."}]},{"t":"p","runs":[{"text":"The same arithmetic run backwards is how you predict a model you have not downloaded. A 19 GB file on a card with 400 GB/s of usable bandwidth will not exceed about 21 tokens per second for one stream, whatever the engine. If that is too slow for the use, no configuration will rescue it and the answer is a smaller model."}]},{"t":"h2","id":"one-warning-about-comparing-engines","runs":[{"text":"One warning about comparing engines"}]},{"t":"p","runs":[{"text":"A benchmark comparison between two engines is only fair if the model, the quantisation, the context length, the batch size and the sampler are the same. They usually are not: comparing llama.cpp at Q4_K_M against vLLM at AWQ-4bit is comparing two different models. If you must compare, put both behind their OpenAI-compatible endpoints and drive them with the same HTTP load generator and the same requests. That is the only comparison that answers a question you actually have."}]}],"check":{"question":[{"text":"A user reports that their new model “loads instantly but produces about one token every three seconds” on a machine with 16 GB of RAM and no GPU, running a 13 GB quantised file. What is the most likely cause?"}],"options":[{"text":[{"text":"The context window is set too high, so the KV cache is consuming the memory the weights need."}],"correct":false,"tempting":[{"text":"Picks a real failure with the wrong magnitude; a cache at ordinary context lengths is under a gigabyte against a 13 GB file."}]},{"text":[{"text":"The instant load indicates memory mapping succeeded but the file does not fit alongside the operating system, so pages are being read from disk continuously."}],"correct":true,"tempting":null},{"text":[{"text":"Thread count defaults are wrong, and setting threads to the physical core count will restore normal speed."}],"correct":false,"tempting":[{"text":"Would explain a two-fold difference, not a hundred-fold one; threading cannot take a model from disk speed to memory speed."}]},{"text":[{"text":"The quantisation format is unsupported by the build, so the engine is falling back to a slow reference implementation."}],"correct":false,"tempting":[{"text":"Plausible in principle but does not explain the instant load, which is the specific clue that the file was mapped rather than read."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"more-than-one-gpu","title":"More than one GPU","minutes":9},"next":{"slug":"when-it-does-not-fit","title":"When it does not fit: the ordered remedies","minutes":9}}}