{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"cpu-inference-honestly","title":"CPU inference, honestly","url":"https://addaly.com/learn/running-models-yourself/cpu-inference-honestly","position":15,"of":82,"minutes":9,"answer":"CPU generation speed is set by memory channels and memory speed rather than cores: populate every channel, thread to physical cores, and expect cores to help prompt processing while doing almost nothing for generation.","keyIdea":"CPU generation speed is set by memory channels and memory speed rather than cores: populate every channel, thread to physical cores, and expect cores to help prompt processing while doing almost nothing for generation."},"body":{"format":"markdown","text":"## The component that matters is not the one on the box\n\nFor generation, a CPU's core count is nearly irrelevant and its memory configuration is everything. Decode reads the whole model per token, so the ceiling is memory bandwidth, and bandwidth on a CPU is set by memory channels and memory speed, not by cores.\n\nThe arithmetic for channels:\n\n```\nGB/s = channels x transfers_per_second x 8 bytes\n```\n\nSo dual-channel DDR4-3200 gives 2 x 3.2 x 8 = 51.2 GB/s theoretical, roughly 30 usable. Dual-channel DDR5-5600 gives 89.6, roughly 55 usable. Those two numbers describe almost every laptop and desktop, and they set a hard ceiling of about 6 and 11 tokens per second respectively on a 4.9 GB model.\n\nThis is why a sixteen-core desktop and an eight-core laptop with the same memory generate at nearly the same speed, to the great confusion of people who bought the sixteen cores for this.\n\n## Where cores do help\n\nPrefill is compute-bound, so cores matter there, up to a point. Prompt processing scales with threads until it saturates memory or hits diminishing returns, usually somewhere around the number of physical cores. Two rules that are worth more than they sound:\n\n- Set threads to **physical** cores, not logical ones. Hyper-threading usually costs a few per cent here because the two threads on a core contend for the same vector units.\n- On chips with performance and efficiency cores, restrict to the performance cores. Letting work land on efficiency cores slows the whole batch, because every thread waits for the slowest.\n\n```bash\nllama-server -m model.gguf -t 8 -tb 8 -c 8192\n```\n\n`-t` sets generation threads, `-tb` the batch (prefill) threads. On many machines the best generation setting is lower than the best prefill setting, because generation saturates bandwidth with a handful of threads and extra threads only add contention.\n\nInstruction sets matter for prefill too. AVX2 is the baseline; AVX-512 with VNNI, and AMX on recent server parts, give large gains for integer matrix work. A build compiled for your machine can beat a generic binary by a noticeable margin. This is one of the few cases where compiling llama.cpp yourself is worth the twenty minutes.\n\n## Where the interesting CPU builds are\n\nThe machines that make CPU inference attractive are the ones with many memory channels.\n\n- **Server and workstation platforms**: an eight-channel DDR4 board gives about 200 GB/s theoretical, and current twelve-channel DDR5 platforms reach 400 to 600. At that level a 70B at Q4 generates at a usable speed with no GPU at all, and a very large MoE model becomes comfortable.\n- **Used server hardware** is cheap for a reason worth knowing: idle power draw of 150 to 250 W, noise, and RAM that costs more than the board. Run the electricity arithmetic before buying.\n- **NUMA** is the trap on dual-socket machines. Memory attached to the other socket is reached over an interconnect, and a model spread across both sockets can be slower than one socket alone. llama.cpp has `--numa distribute` and `--numa isolate`; test both, and be prepared for one socket to win.\n\n## What CPU-only does well\n\nDo not read the above as \"CPU inference is pointless\". On an ordinary laptop with 16 GB and DDR5, a 4B model at Q4 runs at 10 to 15 tokens per second, which is faster than most people read. That covers classification, extraction, tagging, rewriting, summarising short documents and RAG answers over a few retrieved passages: a large fraction of real work, done privately on hardware you already own.\n\nThe honest boundary is prompt length. At 40 to 60 tokens per second of prefill, a 3,000-token prompt is a minute of silence. Design around it: keep prompts short, cache the system prompt, retrieve less, and stream the output so something appears early.\n\n## Getting the most from what you have\n\n1. **Populate all memory channels.** A single 32 GB stick in a dual-channel laptop halves your bandwidth against two 16 GB sticks. This is a free doubling and people miss it constantly.\n2. **Check the memory is running at its rated speed.** XMP or EXPO profiles are frequently off by default, leaving DDR5-6000 running at 4800.\n3. **Use every backend you have.** A Vulkan build reaches Intel and AMD integrated graphics and roughly triples prefill.\n4. **Pick the smallest model that does the job.** On a bandwidth-limited machine, a 3B is not slightly faster than an 8B, it is about twice as fast, because the bytes read per token halved.\n5. **Watch thermals.** A thin laptop that starts at 12 tokens per second and settles at 7 is throttling, not degrading, and a cooling pad is a cheaper upgrade than RAM."},"check":{"question":"Someone upgrades a desktop from 8 to 16 CPU cores, keeping the same dual-channel DDR5-5600 memory, and finds generation speed on an 8B model unchanged at about 10 tokens per second while prompt processing improves substantially. Why?","options":[{"text":"The model is not compiled with AVX-512 support, so the extra cores cannot be used for generation kernels.","correct":false,"tempting":"Points at an instruction set that affects both phases; if it were the cause, prefill would not have improved either."},{"text":"Generation is limited by how fast the weights can be read from memory, which the core count does not change, while prompt processing is arithmetic and does scale with cores.","correct":true},{"text":"Generation is inherently single-threaded because each token depends on the previous one, so extra cores can never help it.","correct":false,"tempting":"Confuses sequential dependency between tokens with parallelism within one token, which is real and is used; the limit is bandwidth, not serialisation."},{"text":"The thread count was left at its default, so only the original eight cores are being used for both phases.","correct":false,"tempting":"Would predict no improvement anywhere, and cannot explain a prefill gain alongside flat generation."}]},"prev":{"slug":"where-the-model-lives","title":"Where the model physically lives","url":"https://addaly.com/learn/running-models-yourself/where-the-model-lives"},"next":{"slug":"more-than-one-gpu","title":"More than one GPU","url":"https://addaly.com/learn/running-models-yourself/more-than-one-gpu"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'CPU inference, honestly', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/cpu-inference-honestly (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"CPU inference, honestly","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"CPU generation speed is set by memory channels and memory speed rather than cores: populate every channel, thread to physical cores, and expect cores to help prompt processing while doing almost nothing for generation.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"the-component-that-matters-is-not-the-one-on-the-box","runs":[{"text":"The component that matters is not the one on the box"}]},{"t":"p","runs":[{"text":"For generation, a CPU’s core count is nearly irrelevant and its memory configuration is everything. Decode reads the whole model per token, so the ceiling is memory bandwidth, and bandwidth on a CPU is set by memory channels and memory speed, not by cores."}]},{"t":"p","runs":[{"text":"The arithmetic for channels:"}]},{"t":"code","lang":"","label":"Text","text":"GB/s = channels x transfers_per_second x 8 bytes"},{"t":"p","runs":[{"text":"So dual-channel DDR4-3200 gives 2 x 3.2 x 8 = 51.2 GB/s theoretical, roughly 30 usable. Dual-channel DDR5-5600 gives 89.6, roughly 55 usable. Those two numbers describe almost every laptop and desktop, and they set a hard ceiling of about 6 and 11 tokens per second respectively on a 4.9 GB model."}]},{"t":"p","runs":[{"text":"This is why a sixteen-core desktop and an eight-core laptop with the same memory generate at nearly the same speed, to the great confusion of people who bought the sixteen cores for this."}]},{"t":"h2","id":"where-cores-do-help","runs":[{"text":"Where cores do help"}]},{"t":"p","runs":[{"text":"Prefill is compute-bound, so cores matter there, up to a point. Prompt processing scales with threads until it saturates memory or hits diminishing returns, usually somewhere around the number of physical cores. Two rules that are worth more than they sound:"}]},{"t":"ul","items":[[{"text":"Set threads to "},{"text":"physical","b":true},{"text":" cores, not logical ones. Hyper-threading usually costs a few per cent here because the two threads on a core contend for the same vector units."}],[{"text":"On chips with performance and efficiency cores, restrict to the performance cores. Letting work land on efficiency cores slows the whole batch, because every thread waits for the slowest."}]]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -t 8 -tb 8 -c 8192"},{"t":"p","runs":[{"text":"-t","code":true},{"text":" sets generation threads, "},{"text":"-tb","code":true},{"text":" the batch (prefill) threads. On many machines the best generation setting is lower than the best prefill setting, because generation saturates bandwidth with a handful of threads and extra threads only add contention."}]},{"t":"p","runs":[{"text":"Instruction sets matter for prefill too. AVX2 is the baseline; AVX-512 with VNNI, and AMX on recent server parts, give large gains for integer matrix work. A build compiled for your machine can beat a generic binary by a noticeable margin. This is one of the few cases where compiling llama.cpp yourself is worth the twenty minutes."}]},{"t":"h2","id":"where-the-interesting-cpu-builds-are","runs":[{"text":"Where the interesting CPU builds are"}]},{"t":"p","runs":[{"text":"The machines that make CPU inference attractive are the ones with many memory channels."}]},{"t":"ul","items":[[{"text":"Server and workstation platforms","b":true},{"text":": an eight-channel DDR4 board gives about 200 GB/s theoretical, and current twelve-channel DDR5 platforms reach 400 to 600. At that level a 70B at Q4 generates at a usable speed with no GPU at all, and a very large MoE model becomes comfortable."}],[{"text":"Used server hardware","b":true},{"text":" is cheap for a reason worth knowing: idle power draw of 150 to 250 W, noise, and RAM that costs more than the board. Run the electricity arithmetic before buying."}],[{"text":"NUMA","b":true},{"text":" is the trap on dual-socket machines. Memory attached to the other socket is reached over an interconnect, and a model spread across both sockets can be slower than one socket alone. llama.cpp has "},{"text":"--numa distribute","code":true},{"text":" and "},{"text":"--numa isolate","code":true},{"text":"; test both, and be prepared for one socket to win."}]]},{"t":"h2","id":"what-cpu-only-does-well","runs":[{"text":"What CPU-only does well"}]},{"t":"p","runs":[{"text":"Do not read the above as “CPU inference is pointless”. On an ordinary laptop with 16 GB and DDR5, a 4B model at Q4 runs at 10 to 15 tokens per second, which is faster than most people read. That covers classification, extraction, tagging, rewriting, summarising short documents and RAG answers over a few retrieved passages: a large fraction of real work, done privately on hardware you already own."}]},{"t":"p","runs":[{"text":"The honest boundary is prompt length. At 40 to 60 tokens per second of prefill, a 3,000-token prompt is a minute of silence. Design around it: keep prompts short, cache the system prompt, retrieve less, and stream the output so something appears early."}]},{"t":"h2","id":"getting-the-most-from-what-you-have","runs":[{"text":"Getting the most from what you have"}]},{"t":"ol","start":1,"items":[[{"text":"Populate all memory channels.","b":true},{"text":" A single 32 GB stick in a dual-channel laptop halves your bandwidth against two 16 GB sticks. This is a free doubling and people miss it constantly."}],[{"text":"Check the memory is running at its rated speed.","b":true},{"text":" XMP or EXPO profiles are frequently off by default, leaving DDR5-6000 running at 4800."}],[{"text":"Use every backend you have.","b":true},{"text":" A Vulkan build reaches Intel and AMD integrated graphics and roughly triples prefill."}],[{"text":"Pick the smallest model that does the job.","b":true},{"text":" On a bandwidth-limited machine, a 3B is not slightly faster than an 8B, it is about twice as fast, because the bytes read per token halved."}],[{"text":"Watch thermals.","b":true},{"text":" A thin laptop that starts at 12 tokens per second and settles at 7 is throttling, not degrading, and a cooling pad is a cheaper upgrade than RAM."}]]}],"check":{"question":[{"text":"Someone upgrades a desktop from 8 to 16 CPU cores, keeping the same dual-channel DDR5-5600 memory, and finds generation speed on an 8B model unchanged at about 10 tokens per second while prompt processing improves substantially. Why?"}],"options":[{"text":[{"text":"The model is not compiled with AVX-512 support, so the extra cores cannot be used for generation kernels."}],"correct":false,"tempting":[{"text":"Points at an instruction set that affects both phases; if it were the cause, prefill would not have improved either."}]},{"text":[{"text":"Generation is limited by how fast the weights can be read from memory, which the core count does not change, while prompt processing is arithmetic and does scale with cores."}],"correct":true,"tempting":null},{"text":[{"text":"Generation is inherently single-threaded because each token depends on the previous one, so extra cores can never help it."}],"correct":false,"tempting":[{"text":"Confuses sequential dependency between tokens with parallelism within one token, which is real and is used; the limit is bandwidth, not serialisation."}]},{"text":[{"text":"The thread count was left at its default, so only the original eight cores are being used for both phases."}],"correct":false,"tempting":[{"text":"Would predict no improvement anywhere, and cannot explain a prefill gain alongside flat generation."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"where-the-model-lives","title":"Where the model physically lives","minutes":9},"next":{"slug":"more-than-one-gpu","title":"More than one GPU","minutes":9}}}