{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"where-the-model-lives","title":"Where the model physically lives","url":"https://addaly.com/learn/running-models-yourself/where-the-model-lives","position":14,"of":82,"minutes":9,"answer":"Speed is decided by which memory the weights sit in, and a partial offload behaves like a harmonic mean: a tenth of the layers left on the CPU can cost half the speed.","keyIdea":"Speed is decided by which memory the weights sit in, and a partial offload behaves like a harmonic mean: a tenth of the layers left on the CPU can cost half the speed, so fitting entirely usually beats fitting better."},"body":{"format":"markdown","text":"## Four places, four speeds\n\nThe weights are bytes that have to reach the processor. Where they sit decides everything about speed, and the numbers span three orders of magnitude.\n\n- **GPU memory (VRAM).** Roughly 300 GB/s on an older mid-range card, 900 to 1,000 on a current 24 GB consumer card, 2,000 to 3,000 on datacentre parts with high-bandwidth memory.\n- **Unified memory on Apple Silicon.** One pool shared by CPU and GPU. Roughly 68 to 120 GB/s on base chips, 200 to 275 on Pro, 400 to 550 on Max, around 800 on Ultra.\n- **System RAM.** Dual-channel DDR4-3200 is 51.2 GB/s theoretical, about 30 usable. Dual-channel DDR5-5600 is 89.6 theoretical, about 55 usable.\n- **NVMe SSD.** 3 to 7 GB/s sequential. An order of magnitude below system RAM and two below a GPU.\n\nDecode speed is bandwidth divided by model size, so those numbers translate directly into tokens per second. A 4.9 GB model gets about 6 tokens/second from laptop RAM, about 20 from a base Mac, about 60 from a mid-range card, about 180 from a fast one.\n\n## The link between them\n\nA discrete GPU reaches system RAM over PCIe: about 16 GB/s each way on PCIe 4.0 x16, half that on 3.0, and often far less on a laptop with an x4 or x8 link. That is twenty to fifty times slower than the card's own memory, and it is why \"just stream the weights from RAM\" is not a strategy: you would be limited to PCIe bandwidth divided by model size, which is worse than running on the CPU directly.\n\nApple Silicon has no such link, because there is nothing to cross. That single architectural fact is why a Mac with 64 GB runs models that a PC with 64 GB of system RAM and a 12 GB card cannot run pleasantly.\n\n## Partial offload, with the arithmetic\n\nllama.cpp will place some layers on the GPU and leave the rest on the CPU. The result is close to a harmonic mean, and it is brutal.\n\nTake a 32-layer model where the GPU would give 60 tokens/second and the CPU 5. Put 28 layers on the GPU and 4 on the CPU. Every token must pass through all 32. Time per token is the sum of the parts:\n\n```\ngpu part: 28/32 x (1/60) = 0.0146 s\ncpu part:  4/32 x (1/5)  = 0.0250 s\ntotal    = 0.0396 s  ->  25 tokens/second\n```\n\nTwelve per cent of the layers on the CPU cost fifty-eight per cent of the speed. This is the single most surprising number in local inference and it explains why \"it nearly fits\" is such an unhappy place to be. The corollary: getting the last two layers onto the card is worth far more than it looks, and a slightly smaller quant that fits entirely often beats a better quant that does not.\n\n> Figure: Where a 4.9 GB model sits decides how fast it runs. Bars (tokens per second, one stream): Laptop system RAM, about 30 GB/s 6; Base Apple Silicon, about 100 GB/s 20; 28 of 32 layers on a mid-range card 25; All 32 layers on that same card 60; A card with 900 GB/s 180. The third row and the fourth are the same graphics card. Moving four layers of thirty-two onto the CPU costs fifty-eight per cent of the speed, because every token has to pass through all of them, which is why a smaller quant that fits entirely beats a better one that does not.\n\nThe exception is mixture-of-experts models, where the right split is by tensor type rather than by layer, because the expert tensors are large and rarely touched.\n\n## Memory mapping, and the mistake it causes\n\nllama.cpp maps the model file into memory rather than reading it. The operating system pages parts in on demand. Two useful consequences: startup is instant, and several processes sharing a model file share the physical pages.\n\nThe trap is that on a machine with too little RAM, the model appears to load fine and then runs at disk speed, because pages are being evicted and re-read continuously. It does not report an error. It reports two tokens per minute. If a model \"loads but is impossibly slow\", check whether it actually fits in RAM; `--no-mmap` will force a real allocation and fail honestly instead.\n\n## Practical placements\n\n**Everything in VRAM.** The only configuration that gives the card's full speed. Aim for it; give up context or bit rate to reach it.\n\n**Everything in unified memory (Mac).** Same idea, and note macOS caps GPU-accessible memory at roughly 65 to 75%. On a 32 GB machine you can raise it, leaving several gigabytes for the system:\n\n```bash\nsudo sysctl iogpu.wired_limit_mb=24576\n```\n\nIt resets on reboot.\n\n**Everything in system RAM, no GPU.** Slow but predictable, and the right home for a large MoE model.\n\n**Split.** Acceptable when the CPU share is small, or when the model is MoE and the split is by tensor. Otherwise a last resort.\n\n**Streaming from disk.** Not a serious option for interactive use. Useful only to confirm a very large model runs at all before buying memory for it.\n\n## The habit\n\nBefore downloading, decide where the file will live and compute the speed that implies. Two minutes of arithmetic beats forty gigabytes of download followed by disappointment, and it is the difference between choosing hardware and being surprised by it."},"check":{"question":"A 32-layer model runs at 60 tokens/second fully on the GPU. With 4 of the 32 layers left on the CPU, where they would run at 5 tokens/second, what should you expect?","options":[{"text":"Roughly 53 tokens/second, since only an eighth of the work moved to the slower device.","correct":false,"tempting":"Averages the speeds by layer share, which is the intuitive and wrong operation; speeds do not average, times add."},{"text":"About 25 tokens/second, because each token's time is the sum of the GPU and CPU portions and the slow portion dominates.","correct":true},{"text":"About 5 tokens/second, because any CPU involvement makes the whole model run at CPU speed.","correct":false,"tempting":"Overcorrects into a rule that would make partial offload useless, when 28 GPU layers do contribute real speed."},{"text":"Close to 60 tokens/second, because the CPU layers are computed in parallel with the GPU layers.","correct":false,"tempting":"Assumes parallelism across layers, which is impossible: layer n+1 needs the output of layer n."}]},"prev":{"slug":"prefill-and-decode","title":"Prefill and decode are two different machines","url":"https://addaly.com/learn/running-models-yourself/prefill-and-decode"},"next":{"slug":"cpu-inference-honestly","title":"CPU inference, honestly","url":"https://addaly.com/learn/running-models-yourself/cpu-inference-honestly"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Where the model physically lives', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/where-the-model-lives (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Where the model physically lives","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"Speed is decided by which memory the weights sit in, and a partial offload behaves like a harmonic mean: a tenth of the layers left on the CPU can cost half the speed, so fitting entirely usually beats fitting better.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"bars","unit":"tokens per second, one stream","title":"Where a 4.9 GB model sits decides how fast it runs","series":[{"label":"Laptop system RAM, about 30 GB/s","value":6},{"label":"Base Apple Silicon, about 100 GB/s","value":20},{"label":"28 of 32 layers on a mid-range card","value":25},{"label":"All 32 layers on that same card","value":60,"highlight":true},{"label":"A card with 900 GB/s","value":180}],"caption":"The third row and the fourth are the same graphics card. Moving four layers of thirty-two onto the CPU costs fifty-eight per cent of the speed, because every token has to pass through all of them, which is why a smaller quant that fits entirely beats a better one that does not."},"alt":"Figure: Where a 4.9 GB model sits decides how fast it runs. Bars (tokens per second, one stream): Laptop system RAM, about 30 GB/s 6; Base Apple Silicon, about 100 GB/s 20; 28 of 32 layers on a mid-range card 25; All 32 layers on that same card 60; A card with 900 GB/s 180. The third row and the fourth are the same graphics card. Moving four layers of thirty-two onto the CPU costs fifty-eight per cent of the speed, because every token has to pass through all of them, which is why a smaller quant that fits entirely beats a better one that does not."}},"blocks":[{"t":"h2","id":"four-places-four-speeds","runs":[{"text":"Four places, four speeds"}]},{"t":"p","runs":[{"text":"The weights are bytes that have to reach the processor. Where they sit decides everything about speed, and the numbers span three orders of magnitude."}]},{"t":"ul","items":[[{"text":"GPU memory (VRAM).","b":true},{"text":" Roughly 300 GB/s on an older mid-range card, 900 to 1,000 on a current 24 GB consumer card, 2,000 to 3,000 on datacentre parts with high-bandwidth memory."}],[{"text":"Unified memory on Apple Silicon.","b":true},{"text":" One pool shared by CPU and GPU. Roughly 68 to 120 GB/s on base chips, 200 to 275 on Pro, 400 to 550 on Max, around 800 on Ultra."}],[{"text":"System RAM.","b":true},{"text":" Dual-channel DDR4-3200 is 51.2 GB/s theoretical, about 30 usable. Dual-channel DDR5-5600 is 89.6 theoretical, about 55 usable."}],[{"text":"NVMe SSD.","b":true},{"text":" 3 to 7 GB/s sequential. An order of magnitude below system RAM and two below a GPU."}]]},{"t":"p","runs":[{"text":"Decode speed is bandwidth divided by model size, so those numbers translate directly into tokens per second. A 4.9 GB model gets about 6 tokens/second from laptop RAM, about 20 from a base Mac, about 60 from a mid-range card, about 180 from a fast one."}]},{"t":"h2","id":"the-link-between-them","runs":[{"text":"The link between them"}]},{"t":"p","runs":[{"text":"A discrete GPU reaches system RAM over PCIe: about 16 GB/s each way on PCIe 4.0 x16, half that on 3.0, and often far less on a laptop with an x4 or x8 link. That is twenty to fifty times slower than the card’s own memory, and it is why “just stream the weights from RAM” is not a strategy: you would be limited to PCIe bandwidth divided by model size, which is worse than running on the CPU directly."}]},{"t":"p","runs":[{"text":"Apple Silicon has no such link, because there is nothing to cross. That single architectural fact is why a Mac with 64 GB runs models that a PC with 64 GB of system RAM and a 12 GB card cannot run pleasantly."}]},{"t":"h2","id":"partial-offload-with-the-arithmetic","runs":[{"text":"Partial offload, with the arithmetic"}]},{"t":"p","runs":[{"text":"llama.cpp will place some layers on the GPU and leave the rest on the CPU. The result is close to a harmonic mean, and it is brutal."}]},{"t":"p","runs":[{"text":"Take a 32-layer model where the GPU would give 60 tokens/second and the CPU 5. Put 28 layers on the GPU and 4 on the CPU. Every token must pass through all 32. Time per token is the sum of the parts:"}]},{"t":"code","lang":"","label":"Text","text":"gpu part: 28/32 x (1/60) = 0.0146 s\ncpu part:  4/32 x (1/5)  = 0.0250 s\ntotal    = 0.0396 s  ->  25 tokens/second"},{"t":"p","runs":[{"text":"Twelve per cent of the layers on the CPU cost fifty-eight per cent of the speed. This is the single most surprising number in local inference and it explains why “it nearly fits” is such an unhappy place to be. The corollary: getting the last two layers onto the card is worth far more than it looks, and a slightly smaller quant that fits entirely often beats a better quant that does not."}]},{"t":"p","runs":[{"text":"The exception is mixture-of-experts models, where the right split is by tensor type rather than by layer, because the expert tensors are large and rarely touched."}]},{"t":"h2","id":"memory-mapping-and-the-mistake-it-causes","runs":[{"text":"Memory mapping, and the mistake it causes"}]},{"t":"p","runs":[{"text":"llama.cpp maps the model file into memory rather than reading it. The operating system pages parts in on demand. Two useful consequences: startup is instant, and several processes sharing a model file share the physical pages."}]},{"t":"p","runs":[{"text":"The trap is that on a machine with too little RAM, the model appears to load fine and then runs at disk speed, because pages are being evicted and re-read continuously. It does not report an error. It reports two tokens per minute. If a model “loads but is impossibly slow”, check whether it actually fits in RAM; "},{"text":"--no-mmap","code":true},{"text":" will force a real allocation and fail honestly instead."}]},{"t":"h2","id":"practical-placements","runs":[{"text":"Practical placements"}]},{"t":"p","runs":[{"text":"Everything in VRAM.","b":true},{"text":" The only configuration that gives the card’s full speed. Aim for it; give up context or bit rate to reach it."}]},{"t":"p","runs":[{"text":"Everything in unified memory (Mac).","b":true},{"text":" Same idea, and note macOS caps GPU-accessible memory at roughly 65 to 75%. On a 32 GB machine you can raise it, leaving several gigabytes for the system:"}]},{"t":"code","lang":"bash","label":"Bash","text":"sudo sysctl iogpu.wired_limit_mb=24576"},{"t":"p","runs":[{"text":"It resets on reboot."}]},{"t":"p","runs":[{"text":"Everything in system RAM, no GPU.","b":true},{"text":" Slow but predictable, and the right home for a large MoE model."}]},{"t":"p","runs":[{"text":"Split.","b":true},{"text":" Acceptable when the CPU share is small, or when the model is MoE and the split is by tensor. Otherwise a last resort."}]},{"t":"p","runs":[{"text":"Streaming from disk.","b":true},{"text":" Not a serious option for interactive use. Useful only to confirm a very large model runs at all before buying memory for it."}]},{"t":"h2","id":"the-habit","runs":[{"text":"The habit"}]},{"t":"p","runs":[{"text":"Before downloading, decide where the file will live and compute the speed that implies. Two minutes of arithmetic beats forty gigabytes of download followed by disappointment, and it is the difference between choosing hardware and being surprised by it."}]}],"check":{"question":[{"text":"A 32-layer model runs at 60 tokens/second fully on the GPU. With 4 of the 32 layers left on the CPU, where they would run at 5 tokens/second, what should you expect?"}],"options":[{"text":[{"text":"Roughly 53 tokens/second, since only an eighth of the work moved to the slower device."}],"correct":false,"tempting":[{"text":"Averages the speeds by layer share, which is the intuitive and wrong operation; speeds do not average, times add."}]},{"text":[{"text":"About 25 tokens/second, because each token’s time is the sum of the GPU and CPU portions and the slow portion dominates."}],"correct":true,"tempting":null},{"text":[{"text":"About 5 tokens/second, because any CPU involvement makes the whole model run at CPU speed."}],"correct":false,"tempting":[{"text":"Overcorrects into a rule that would make partial offload useless, when 28 GPU layers do contribute real speed."}]},{"text":[{"text":"Close to 60 tokens/second, because the CPU layers are computed in parallel with the GPU layers."}],"correct":false,"tempting":[{"text":"Assumes parallelism across layers, which is impossible: layer n+1 needs the output of layer n."}]}],"order":[2,1,3,0],"why":null},"prev":{"slug":"prefill-and-decode","title":"Prefill and decode are two different machines","minutes":9},"next":{"slug":"cpu-inference-honestly","title":"CPU inference, honestly","minutes":9}}}