Spreading a model across whatever you have

One line that does a surprising amount

model = AutoModelForCausalLM.from_pretrained(
    name, dtype=torch.bfloat16, device_map="auto"
)

device_map="auto" comes from the accelerate library and does four things: inspect available devices and their free memory, load the model skeleton without weights, decide which layers go where, then stream the weights in one shard at a time so the full model never has to exist in system RAM at once.

That last part is why it also helps on a single GPU. Without it, the naive path is: read 14 GB into CPU memory, then copy to the GPU. On a machine with 16 GB of RAM that fails before the GPU is involved at all. With it, shards move through and are released.

What the map looks like

print(model.hf_device_map)
# {'model.embed_tokens': 0, 'model.layers.0': 0, ...,
#  'model.layers.24': 'cpu', ..., 'lm_head': 'disk'}

Three kinds of destination, in descending order of speed:

• A GPU index: fast.
• 'cpu': the layer lives in system RAM and its weights are copied to the GPU when that layer runs, then evicted. Correct results, considerably slower.
• 'disk': the layer is read from an offload folder each time. Correct results, dramatically slower.

This is the honest mechanism behind “I ran a 70B model on my laptop and it took four minutes per token”. It ran because every layer was fetched from disk for every single token. Nothing was broken; you were reading gigabytes from an SSD thousands of times.

model = AutoModelForCausalLM.from_pretrained(
    name, device_map="auto",
    max_memory={0: "14GiB", "cpu": "24GiB"},
    offload_folder="./offload",
)

max_memory is worth setting explicitly. Left to itself, accelerate fills the GPU and leaves no headroom for the KV cache, so the model loads and then runs out of memory on the first long generation, which reads as a mysterious intermittent failure.

When offloading is the right answer

• Yes: a one-off batch job overnight; checking whether a large model is even worth pursuing; running something once to produce a dataset.
• No: anything interactive; anything you will run more than a few dozen times.

For the second case, quantization almost always beats offloading. A 70B model at 4-bit entirely in 40 GB of GPU memory is faster by orders of magnitude than the same model at bfloat16 with half its layers on disk, and usually better than dropping to a 13B model, because a heavily quantized large model tends to outperform a small model at full precision on the same memory budget. That comparison is worth running yourself on your own task rather than taking on trust.

Apple silicon

On a Mac, PyTorch’s Metal backend gives you mps:

model = AutoModelForCausalLM.from_pretrained(name, dtype=torch.float16).to("mps")

Unified memory means the GPU can address system RAM, so a 32 GB MacBook runs models that would need a 32 GB discrete card. The limits are real: some operations still fall back to CPU, bfloat16 support has been uneven across versions, and throughput is well below a comparable NVIDIA card. For most Mac users, llama.cpp through GGUF is the faster and more reliable path, and the next lesson covers it.

Multiple GPUs, briefly

device_map="auto" across two cards gives you pipeline parallelism: layers 0 to 15 on card 0, 16 to 31 on card 1. Card 1 waits while card 0 works, so you get the memory of both and roughly the speed of one. It is a memory solution, not a speed solution, and people are regularly disappointed by that. Tensor parallelism (splitting each layer across cards so both compute at once) is what serving frameworks do, and it is a reason to reach for vLLM rather than tuning transformers further.

Do this now

Load any model with device_map="auto" and print hf_device_map. If every entry is 0, the model fits and you are fine. If you see 'cpu' or 'disk', you now know exactly why generation is slow, and the fix is quantization rather than patience.