{"course":{"slug":"hugging-face","title":"Hugging Face, End to End","url":"https://addaly.com/learn/hugging-face","level":"intermediate","track":"create"},"lesson":{"slug":"device-map-and-offloading","title":"Spreading a model across whatever you have","url":"https://addaly.com/learn/hugging-face/device-map-and-offloading","position":21,"of":75,"minutes":8,"answer":"device_map=\"auto\" places layers across GPU, CPU and disk and streams weights shard by shard, which makes oversized models run correctly and extremely slowly.","keyIdea":"`device_map=\"auto\"` places layers across GPU, CPU and disk and streams weights shard by shard, which makes oversized models run correctly and extremely slowly, so offloading suits one-off jobs while quantization is the right answer for anything interactive."},"body":{"format":"markdown","text":"## One line that does a surprising amount\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    name, dtype=torch.bfloat16, device_map=\"auto\"\n)\n```\n\n`device_map=\"auto\"` comes from the accelerate library and does four things: inspect available devices and their free memory, load the model skeleton without weights, decide which layers go where, then stream the weights in one shard at a time so the full model never has to exist in system RAM at once.\n\nThat last part is why it also helps on a single GPU. Without it, the naive path is: read 14 GB into CPU memory, then copy to the GPU. On a machine with 16 GB of RAM that fails before the GPU is involved at all. With it, shards move through and are released.\n\n## What the map looks like\n\n```python\nprint(model.hf_device_map)\n# {'model.embed_tokens': 0, 'model.layers.0': 0, ...,\n#  'model.layers.24': 'cpu', ..., 'lm_head': 'disk'}\n```\n\nThree kinds of destination, in descending order of speed:\n\n- **A GPU index**: fast.\n- **`'cpu'`**: the layer lives in system RAM and its weights are copied to the GPU when that layer runs, then evicted. Correct results, considerably slower.\n- **`'disk'`**: the layer is read from an offload folder each time. Correct results, dramatically slower.\n\nThis is the honest mechanism behind \"I ran a 70B model on my laptop and it took four minutes per token\". It ran because every layer was fetched from disk for every single token. Nothing was broken; you were reading gigabytes from an SSD thousands of times.\n\n> Figure: Where device_map auto puts a layer. Layers, top to bottom: A GPU index (The layer stays on the card. Full speed.); cpu (Copied to the GPU when that layer runs, then evicted. Correct, several times slower.); disk (Read from an offload folder for every token of every request. Correct, hundreds of times slower.). Print hf_device_map and read it. Every entry a zero means the model fits and the speed is the hardware's. A cpu or a disk entry is the whole explanation for four minutes a token, and the fix is quantization rather than patience.\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(\n    name, device_map=\"auto\",\n    max_memory={0: \"14GiB\", \"cpu\": \"24GiB\"},\n    offload_folder=\"./offload\",\n)\n```\n\n`max_memory` is worth setting explicitly. Left to itself, accelerate fills the GPU and leaves no headroom for the KV cache, so the model loads and then runs out of memory on the first long generation, which reads as a mysterious intermittent failure.\n\n## When offloading is the right answer\n\n- **Yes:** a one-off batch job overnight; checking whether a large model is even worth pursuing; running something once to produce a dataset.\n- **No:** anything interactive; anything you will run more than a few dozen times.\n\nFor the second case, quantization almost always beats offloading. A 70B model at 4-bit entirely in 40 GB of GPU memory is faster by orders of magnitude than the same model at bfloat16 with half its layers on disk, and usually better than dropping to a 13B model, because a heavily quantized large model tends to outperform a small model at full precision on the same memory budget. That comparison is worth running yourself on your own task rather than taking on trust.\n\n## Apple silicon\n\nOn a Mac, PyTorch's Metal backend gives you `mps`:\n\n```python\nmodel = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float16).to(\"mps\")\n```\n\nUnified memory means the GPU can address system RAM, so a 32 GB MacBook runs models that would need a 32 GB discrete card. The limits are real: some operations still fall back to CPU, bfloat16 support has been uneven across versions, and throughput is well below a comparable NVIDIA card. For most Mac users, `llama.cpp` through GGUF is the faster and more reliable path, and the next lesson covers it.\n\n## Multiple GPUs, briefly\n\n`device_map=\"auto\"` across two cards gives you **pipeline parallelism**: layers 0 to 15 on card 0, 16 to 31 on card 1. Card 1 waits while card 0 works, so you get the memory of both and roughly the speed of one. It is a memory solution, not a speed solution, and people are regularly disappointed by that. Tensor parallelism (splitting each layer across cards so both compute at once) is what serving frameworks do, and it is a reason to reach for vLLM rather than tuning transformers further.\n\n## Do this now\n\nLoad any model with `device_map=\"auto\"` and print `hf_device_map`. If every entry is `0`, the model fits and you are fine. If you see `'cpu'` or `'disk'`, you now know exactly why generation is slow, and the fix is quantization rather than patience."},"check":{"question":"A large model loaded with `device_map=\"auto\"` produces correct output at roughly one token every few seconds on a machine with one GPU and a fast SSD. What is the explanation?","options":[{"text":"Pipeline parallelism is serialising the layers, so only one part of the model computes at a time.","correct":false,"tempting":"Describes multi-GPU behaviour; with a single GPU there is no pipeline to serialise."},{"text":"Automatic placement defaults to float32 when it cannot determine the dtype, quadrupling the arithmetic per token.","correct":false,"tempting":"Would slow things by a factor of a few, not thousands, and dtype comes from the load call rather than the map."},{"text":"Some layers were placed on CPU or disk, so their weights are fetched and evicted for every token generated.","correct":true},{"text":"The KV cache has been offloaded to disk, so each new token requires re-reading the whole conversation history from storage.","correct":false,"tempting":"Plausible in shape, but the cache stays with its layer's device rather than being offloaded independently by default."}]},"prev":{"slug":"dtype-and-the-memory-bill","title":"The memory arithmetic you can do in your head","url":"https://addaly.com/learn/hugging-face/dtype-and-the-memory-bill"},"next":{"slug":"quantisation-four-ways","title":"Quantization: what you buy and what you pay","url":"https://addaly.com/learn/hugging-face/quantisation-four-ways"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Spreading a model across whatever you have', in Hugging Face, End to End. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/hugging-face/device-map-and-offloading (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Spreading a model across whatever you have","courseTitle":"Hugging Face, End to End","module":{"index":3,"slug":"running-it-on-what-you-have","title":"Running it on the hardware you actually have"},"answer":null,"keyIdea":"`device_map=\"auto\"` places layers across GPU, CPU and disk and streams weights shard by shard, which makes oversized models run correctly and extremely slowly, so offloading suits one-off jobs while quantization is the right answer for anything interactive.","hero":{"kind":"figure","figure":{"t":"figure","label":"Figure","figure":{"type":"stack","title":"Where device_map auto puts a layer","layers":[{"label":"A GPU index","detail":"The layer stays on the card. Full speed."},{"label":"cpu","detail":"Copied to the GPU when that layer runs, then evicted. Correct, several times slower."},{"label":"disk","detail":"Read from an offload folder for every token of every request. Correct, hundreds of times slower."}],"caption":"Print hf_device_map and read it. Every entry a zero means the model fits and the speed is the hardware’s. A cpu or a disk entry is the whole explanation for four minutes a token, and the fix is quantization rather than patience."},"alt":"Figure: Where device_map auto puts a layer. Layers, top to bottom: A GPU index (The layer stays on the card. Full speed.); cpu (Copied to the GPU when that layer runs, then evicted. Correct, several times slower.); disk (Read from an offload folder for every token of every request. Correct, hundreds of times slower.). Print hf_device_map and read it. Every entry a zero means the model fits and the speed is the hardware’s. A cpu or a disk entry is the whole explanation for four minutes a token, and the fix is quantization rather than patience."}},"blocks":[{"t":"h2","id":"one-line-that-does-a-surprising-amount","runs":[{"text":"One line that does a surprising amount"}]},{"t":"code","lang":"python","label":"Python","text":"model = AutoModelForCausalLM.from_pretrained(\n    name, dtype=torch.bfloat16, device_map=\"auto\"\n)"},{"t":"p","runs":[{"text":"device_map=\"auto\"","code":true},{"text":" comes from the accelerate library and does four things: inspect available devices and their free memory, load the model skeleton without weights, decide which layers go where, then stream the weights in one shard at a time so the full model never has to exist in system RAM at once."}]},{"t":"p","runs":[{"text":"That last part is why it also helps on a single GPU. Without it, the naive path is: read 14 GB into CPU memory, then copy to the GPU. On a machine with 16 GB of RAM that fails before the GPU is involved at all. With it, shards move through and are released."}]},{"t":"h2","id":"what-the-map-looks-like","runs":[{"text":"What the map looks like"}]},{"t":"code","lang":"python","label":"Python","text":"print(model.hf_device_map)\n# {'model.embed_tokens': 0, 'model.layers.0': 0, ...,\n#  'model.layers.24': 'cpu', ..., 'lm_head': 'disk'}"},{"t":"p","runs":[{"text":"Three kinds of destination, in descending order of speed:"}]},{"t":"ul","items":[[{"text":"A GPU index","b":true},{"text":": fast."}],[{"text":"'cpu'","b":true,"code":true},{"text":": the layer lives in system RAM and its weights are copied to the GPU when that layer runs, then evicted. Correct results, considerably slower."}],[{"text":"'disk'","b":true,"code":true},{"text":": the layer is read from an offload folder each time. Correct results, dramatically slower."}]]},{"t":"p","runs":[{"text":"This is the honest mechanism behind “I ran a 70B model on my laptop and it took four minutes per token”. It ran because every layer was fetched from disk for every single token. Nothing was broken; you were reading gigabytes from an SSD thousands of times."}]},{"t":"code","lang":"python","label":"Python","text":"model = AutoModelForCausalLM.from_pretrained(\n    name, device_map=\"auto\",\n    max_memory={0: \"14GiB\", \"cpu\": \"24GiB\"},\n    offload_folder=\"./offload\",\n)"},{"t":"p","runs":[{"text":"max_memory","code":true},{"text":" is worth setting explicitly. Left to itself, accelerate fills the GPU and leaves no headroom for the KV cache, so the model loads and then runs out of memory on the first long generation, which reads as a mysterious intermittent failure."}]},{"t":"h2","id":"when-offloading-is-the-right-answer","runs":[{"text":"When offloading is the right answer"}]},{"t":"ul","items":[[{"text":"Yes:","b":true},{"text":" a one-off batch job overnight; checking whether a large model is even worth pursuing; running something once to produce a dataset."}],[{"text":"No:","b":true},{"text":" anything interactive; anything you will run more than a few dozen times."}]]},{"t":"p","runs":[{"text":"For the second case, quantization almost always beats offloading. A 70B model at 4-bit entirely in 40 GB of GPU memory is faster by orders of magnitude than the same model at bfloat16 with half its layers on disk, and usually better than dropping to a 13B model, because a heavily quantized large model tends to outperform a small model at full precision on the same memory budget. That comparison is worth running yourself on your own task rather than taking on trust."}]},{"t":"h2","id":"apple-silicon","runs":[{"text":"Apple silicon"}]},{"t":"p","runs":[{"text":"On a Mac, PyTorch’s Metal backend gives you "},{"text":"mps","code":true},{"text":":"}]},{"t":"code","lang":"python","label":"Python","text":"model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float16).to(\"mps\")"},{"t":"p","runs":[{"text":"Unified memory means the GPU can address system RAM, so a 32 GB MacBook runs models that would need a 32 GB discrete card. The limits are real: some operations still fall back to CPU, bfloat16 support has been uneven across versions, and throughput is well below a comparable NVIDIA card. For most Mac users, "},{"text":"llama.cpp","code":true},{"text":" through GGUF is the faster and more reliable path, and the next lesson covers it."}]},{"t":"h2","id":"multiple-gpus-briefly","runs":[{"text":"Multiple GPUs, briefly"}]},{"t":"p","runs":[{"text":"device_map=\"auto\"","code":true},{"text":" across two cards gives you "},{"text":"pipeline parallelism","b":true},{"text":": layers 0 to 15 on card 0, 16 to 31 on card 1. Card 1 waits while card 0 works, so you get the memory of both and roughly the speed of one. It is a memory solution, not a speed solution, and people are regularly disappointed by that. Tensor parallelism (splitting each layer across cards so both compute at once) is what serving frameworks do, and it is a reason to reach for vLLM rather than tuning transformers further."}]},{"t":"h2","id":"do-this-now","runs":[{"text":"Do this now"}],"role":"try"},{"t":"p","runs":[{"text":"Load any model with "},{"text":"device_map=\"auto\"","code":true},{"text":" and print "},{"text":"hf_device_map","code":true},{"text":". If every entry is "},{"text":"0","code":true},{"text":", the model fits and you are fine. If you see "},{"text":"'cpu'","code":true},{"text":" or "},{"text":"'disk'","code":true},{"text":", you now know exactly why generation is slow, and the fix is quantization rather than patience."}]}],"check":{"question":[{"text":"A large model loaded with "},{"text":"device_map=\"auto\"","code":true},{"text":" produces correct output at roughly one token every few seconds on a machine with one GPU and a fast SSD. What is the explanation?"}],"options":[{"text":[{"text":"Pipeline parallelism is serialising the layers, so only one part of the model computes at a time."}],"correct":false,"tempting":[{"text":"Describes multi-GPU behaviour; with a single GPU there is no pipeline to serialise."}]},{"text":[{"text":"Automatic placement defaults to float32 when it cannot determine the dtype, quadrupling the arithmetic per token."}],"correct":false,"tempting":[{"text":"Would slow things by a factor of a few, not thousands, and dtype comes from the load call rather than the map."}]},{"text":[{"text":"Some layers were placed on CPU or disk, so their weights are fetched and evicted for every token generated."}],"correct":true,"tempting":null},{"text":[{"text":"The KV cache has been offloaded to disk, so each new token requires re-reading the whole conversation history from storage."}],"correct":false,"tempting":[{"text":"Plausible in shape, but the cache stays with its layer’s device rather than being offloaded independently by default."}]}],"order":[0,1,2,3],"why":null},"prev":{"slug":"dtype-and-the-memory-bill","title":"The memory arithmetic you can do in your head","minutes":9},"next":{"slug":"quantisation-four-ways","title":"Quantization: what you buy and what you pay","minutes":10}}}