CPU inference, honestly

The component that matters is not the one on the box

For generation, a CPU’s core count is nearly irrelevant and its memory configuration is everything. Decode reads the whole model per token, so the ceiling is memory bandwidth, and bandwidth on a CPU is set by memory channels and memory speed, not by cores.

The arithmetic for channels:

GB/s = channels x transfers_per_second x 8 bytes

So dual-channel DDR4-3200 gives 2 x 3.2 x 8 = 51.2 GB/s theoretical, roughly 30 usable. Dual-channel DDR5-5600 gives 89.6, roughly 55 usable. Those two numbers describe almost every laptop and desktop, and they set a hard ceiling of about 6 and 11 tokens per second respectively on a 4.9 GB model.

This is why a sixteen-core desktop and an eight-core laptop with the same memory generate at nearly the same speed, to the great confusion of people who bought the sixteen cores for this.

Where cores do help

Prefill is compute-bound, so cores matter there, up to a point. Prompt processing scales with threads until it saturates memory or hits diminishing returns, usually somewhere around the number of physical cores. Two rules that are worth more than they sound:

• Set threads to physical cores, not logical ones. Hyper-threading usually costs a few per cent here because the two threads on a core contend for the same vector units.
• On chips with performance and efficiency cores, restrict to the performance cores. Letting work land on efficiency cores slows the whole batch, because every thread waits for the slowest.

llama-server -m model.gguf -t 8 -tb 8 -c 8192

-t sets generation threads, -tb the batch (prefill) threads. On many machines the best generation setting is lower than the best prefill setting, because generation saturates bandwidth with a handful of threads and extra threads only add contention.

Instruction sets matter for prefill too. AVX2 is the baseline; AVX-512 with VNNI, and AMX on recent server parts, give large gains for integer matrix work. A build compiled for your machine can beat a generic binary by a noticeable margin. This is one of the few cases where compiling llama.cpp yourself is worth the twenty minutes.

Where the interesting CPU builds are

The machines that make CPU inference attractive are the ones with many memory channels.

• Server and workstation platforms: an eight-channel DDR4 board gives about 200 GB/s theoretical, and current twelve-channel DDR5 platforms reach 400 to 600. At that level a 70B at Q4 generates at a usable speed with no GPU at all, and a very large MoE model becomes comfortable.
• Used server hardware is cheap for a reason worth knowing: idle power draw of 150 to 250 W, noise, and RAM that costs more than the board. Run the electricity arithmetic before buying.
• NUMA is the trap on dual-socket machines. Memory attached to the other socket is reached over an interconnect, and a model spread across both sockets can be slower than one socket alone. llama.cpp has --numa distribute and --numa isolate; test both, and be prepared for one socket to win.

What CPU-only does well

Do not read the above as “CPU inference is pointless”. On an ordinary laptop with 16 GB and DDR5, a 4B model at Q4 runs at 10 to 15 tokens per second, which is faster than most people read. That covers classification, extraction, tagging, rewriting, summarising short documents and RAG answers over a few retrieved passages: a large fraction of real work, done privately on hardware you already own.

The honest boundary is prompt length. At 40 to 60 tokens per second of prefill, a 3,000-token prompt is a minute of silence. Design around it: keep prompts short, cache the system prompt, retrieve less, and stream the output so something appears early.

Getting the most from what you have

1. Populate all memory channels. A single 32 GB stick in a dual-channel laptop halves your bandwidth against two 16 GB sticks. This is a free doubling and people miss it constantly.
2. Check the memory is running at its rated speed. XMP or EXPO profiles are frequently off by default, leaving DDR5-6000 running at 4800.
3. Use every backend you have. A Vulkan build reaches Intel and AMD integrated graphics and roughly triples prefill.
4. Pick the smallest model that does the job. On a bandwidth-limited machine, a 3B is not slightly faster than an 8B, it is about twice as fast, because the bytes read per token halved.
5. Watch thermals. A thin laptop that starts at 12 tokens per second and settles at 7 is throttling, not degrading, and a cooling pad is a cheaper upgrade than RAM.