Running Models YourselfLesson 14 of 82

Contents

Running Models Yourself

Lesson 14 of 82. Sign in to keep your place.

Module 2 · The memory arithmetic

Where the model physically lives

9 min read

Where a 4.9 GB model sits decides how fast it runsLaptop system RAM, about30 GB/s6Base Apple Silicon, about100 GB/s2028 of 32 layers on amid-range card25All 32 layers on that samecard60A card with 900 GB/s180tokens per second, one streamThe third row and the fourth are the same graphics card. Moving four layers of thirty-two onto the CPUcosts fifty-eight per cent of the speed, because every token has to pass through all of them, which iswhy a smaller quant that fits entirely beats a better one that does not.
The third row and the fourth are the same graphics card. Moving four layers of thirty-two onto the CPU costs fifty-eight per cent of the speed, because every token has to pass through all of them, which is why a smaller quant that fits entirely beats a better one that does not.

Figure

Four places, four speeds

The weights are bytes that have to reach the processor. Where they sit decides everything about speed, and the numbers span three orders of magnitude.

  • GPU memory (VRAM). Roughly 300 GB/s on an older mid-range card, 900 to 1,000 on a current 24 GB consumer card, 2,000 to 3,000 on datacentre parts with high-bandwidth memory.
  • Unified memory on Apple Silicon. One pool shared by CPU and GPU. Roughly 68 to 120 GB/s on base chips, 200 to 275 on Pro, 400 to 550 on Max, around 800 on Ultra.
  • System RAM. Dual-channel DDR4-3200 is 51.2 GB/s theoretical, about 30 usable. Dual-channel DDR5-5600 is 89.6 theoretical, about 55 usable.
  • NVMe SSD. 3 to 7 GB/s sequential. An order of magnitude below system RAM and two below a GPU.

Decode speed is bandwidth divided by model size, so those numbers translate directly into tokens per second. A 4.9 GB model gets about 6 tokens/second from laptop RAM, about 20 from a base Mac, about 60 from a mid-range card, about 180 from a fast one.

The link between them

A discrete GPU reaches system RAM over PCIe: about 16 GB/s each way on PCIe 4.0 x16, half that on 3.0, and often far less on a laptop with an x4 or x8 link. That is twenty to fifty times slower than the card’s own memory, and it is why “just stream the weights from RAM” is not a strategy: you would be limited to PCIe bandwidth divided by model size, which is worse than running on the CPU directly.

Apple Silicon has no such link, because there is nothing to cross. That single architectural fact is why a Mac with 64 GB runs models that a PC with 64 GB of system RAM and a 12 GB card cannot run pleasantly.

Partial offload, with the arithmetic

llama.cpp will place some layers on the GPU and leave the rest on the CPU. The result is close to a harmonic mean, and it is brutal.

Take a 32-layer model where the GPU would give 60 tokens/second and the CPU 5. Put 28 layers on the GPU and 4 on the CPU. Every token must pass through all 32. Time per token is the sum of the parts:

Text
gpu part: 28/32 x (1/60) = 0.0146 s
cpu part:  4/32 x (1/5)  = 0.0250 s
total    = 0.0396 s  ->  25 tokens/second

Twelve per cent of the layers on the CPU cost fifty-eight per cent of the speed. This is the single most surprising number in local inference and it explains why “it nearly fits” is such an unhappy place to be. The corollary: getting the last two layers onto the card is worth far more than it looks, and a slightly smaller quant that fits entirely often beats a better quant that does not.

The exception is mixture-of-experts models, where the right split is by tensor type rather than by layer, because the expert tensors are large and rarely touched.

Memory mapping, and the mistake it causes

llama.cpp maps the model file into memory rather than reading it. The operating system pages parts in on demand. Two useful consequences: startup is instant, and several processes sharing a model file share the physical pages.

The trap is that on a machine with too little RAM, the model appears to load fine and then runs at disk speed, because pages are being evicted and re-read continuously. It does not report an error. It reports two tokens per minute. If a model “loads but is impossibly slow”, check whether it actually fits in RAM; --no-mmap will force a real allocation and fail honestly instead.

Practical placements

Everything in VRAM. The only configuration that gives the card’s full speed. Aim for it; give up context or bit rate to reach it.

Everything in unified memory (Mac). Same idea, and note macOS caps GPU-accessible memory at roughly 65 to 75%. On a 32 GB machine you can raise it, leaving several gigabytes for the system:

Bash
sudo sysctl iogpu.wired_limit_mb=24576

It resets on reboot.

Everything in system RAM, no GPU. Slow but predictable, and the right home for a large MoE model.

Split. Acceptable when the CPU share is small, or when the model is MoE and the split is by tensor. Otherwise a last resort.

Streaming from disk. Not a serious option for interactive use. Useful only to confirm a very large model runs at all before buying memory for it.

The habit

Before downloading, decide where the file will live and compute the speed that implies. Two minutes of arithmetic beats forty gigabytes of download followed by disappointment, and it is the difference between choosing hardware and being surprised by it.

Every lesson here is open. Your progress is recorded once you pass the Open Model Ecosystem exam.

Check your understanding

A 32-layer model runs at 60 tokens/second fully on the GPU. With 4 of the 32 layers left on the CPU, where they would run at 5 tokens/second, what should you expect?

Pick the one you would defend. Your answer stays private.

Next lesson · 9 minCPU inference, honestly

Stuck on something?Ask Tri, your AI tutor

Previous lesson: Prefill and decode are two different machines

How to cite: Addaly (2026) ‘Where the model physically lives’, in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/where-the-model-lives (Accessed: 13 October 2026).