{"course":{"slug":"fine-tuning","title":"Fine-Tuning, and When Not To","url":"https://addaly.com/learn/fine-tuning","level":"advanced","track":"deep"},"lesson":{"slug":"multi-gpu-without-tears","title":"Multi-GPU without tears","url":"https://addaly.com/learn/fine-tuning/multi-gpu-without-tears","position":57,"of":84,"minutes":9,"answer":"Adding GPUs multiplies the effective batch size by the device count, so gradient accumulation must be divided or the learning rate raised.","keyIdea":"Adding GPUs multiplies the effective batch size by the device count, so gradient accumulation must be divided or the learning rate raised; and sharded strategies over PCIe often give back in communication what they gain in parallelism, which is why a single card is the faster path for any run under about twelve hours."},"body":{"format":"markdown","text":"## Two different reasons to use more than one card\n\nThey need different tools and confusing them wastes days.\n\n**The model fits on one GPU and you want it faster.** Use data parallelism: a full copy of the model on each card, different batches on each, gradients averaged across them every step. In PyTorch this is DDP.\n\n**The model does not fit on one GPU.** Use sharding: FSDP or DeepSpeed ZeRO, splitting the optimiser state, the gradients and optionally the parameters across cards. This is the subject of the full-fine-tuning lesson.\n\nFor almost all LoRA and QLoRA work, the answer is the first. A quantised 8B model fits on one card, so more cards means more throughput, not more capacity.\n\n## Launching\n\n```bash\naccelerate launch --num_processes 4 train.py\n# or\ntorchrun --nproc_per_node 4 train.py\n```\n\nYour training script does not change. Both launchers start one process per GPU and set the environment variables the libraries read.\n\n## The silent batch-size multiplication\n\nThis is the error that catches everybody once.\n\n```\neffective = per_device_batch_size × gradient_accumulation_steps × num_gpus\n```\n\nA config tuned on one card with `per_device=2, accum=8` has an effective batch of 16. Launch it on four GPUs, unchanged, and the effective batch is **64**. Four times the tokens per step, a quarter of the optimiser steps for the same data, and a learning rate now sized for a batch a quarter as large.\n\nThe run completes. The loss curve is plausible. The result is worse and nobody knows why.\n\nWhen you add GPUs, divide `gradient_accumulation_steps` by the same factor (`accum=2` on four cards keeps the effective batch at 16) or deliberately keep the larger batch and raise the learning rate to match. Either is fine. Doing neither is the bug.\n\n## When two GPUs are slower than one\n\nSharded strategies move tensors between cards constantly. On a machine where the GPUs are connected by PCIe rather than NVLink (which includes most consumer multi-GPU builds and many cheaper rented instances) that communication can dominate.\n\nRules of thumb:\n\n- **DDP over PCIe**: fine. One gradient all-reduce per step, and with gradient accumulation you can make that per *several* forward passes.\n- **FSDP or ZeRO-3 over PCIe**: often disappointing. Parameters are gathered and released every layer, every step.\n- **ZeRO-3 with CPU offload over PCIe**: expect a large slowdown. It is for making things possible, not fast.\n\nMeasure rather than assume. Run 50 steps on one card and 50 on four, and compare seconds per step against tokens per step. If four cards are not giving you close to three times the throughput, the communication is eating it.\n\n## Two practical traps\n\n**Every process runs the whole script.** Without guards, four processes will each download your dataset, each write to the same log file, and each try to push to the Hub. Guard side effects:\n\n```python\nif accelerator.is_main_process:\n    dataset.save_to_disk(path)\n```\n\nThe Hugging Face `Trainer` handles logging and saving correctly on its own; it is your own code around it that needs the guard.\n\n**Data loader workers.** `dataloader_num_workers=4` with four GPUs is sixteen worker processes. On a small dataset this is pure overhead and a common source of hangs at startup. Start at 2 and raise it only if the GPUs are visibly waiting.\n\n## When not to bother\n\nA two-hour LoRA run does not need multi-GPU. The complexity (a launcher, a batch-size recalculation, the debugging of a hang that only occurs with four processes) costs more than the ninety minutes it saves.\n\nThe threshold in practice is around a day. If a run would take more than about twelve hours on one card, the extra cards are worth the setup. Below that, a single card and a config you trust is the faster path to an answer, and one GPU has the additional merit that every bug is your bug rather than a distributed one.\n\nThere is also a better use for a second card that nobody suggests: run two *different* configurations on the two GPUs simultaneously rather than one configuration across both. An ablation finishes in the same wall-clock time as a single run, the setup is nothing more than two terminals, and you end the day with a comparison rather than with one number that arrived faster."},"check":{"question":"A config tuned on one GPU (per_device=2, accum=8, lr=2e-4) is launched unchanged on four GPUs with DDP. The run finishes and the model is worse. What happened?","options":[{"text":"DDP averages gradients across devices, which shrinks the update by a factor of four and undertrains the adapter.","correct":false,"tempting":"Misreads averaging as attenuation; averaging is what makes four devices equivalent to one larger batch, not a division of the step size."},{"text":"Each GPU trained its own adapter copy, and only rank zero's was saved, discarding three quarters of the learning.","correct":false,"tempting":"Describes a synchronisation failure that DDP specifically prevents; gradients are all-reduced so every replica stays identical."},{"text":"The effective batch became 64 rather than 16, so there were a quarter as many optimiser steps.","correct":true},{"text":"Four processes each shuffled the data with the same seed, so every device trained on the identical batches.","correct":false,"tempting":"Assumes sharding is broken; the distributed sampler assigns disjoint slices per rank, which is exactly what makes the batch larger."}]},"prev":{"slug":"reproducibility-and-seeds","title":"Reproducibility, and the noise floor","url":"https://addaly.com/learn/fine-tuning/reproducibility-and-seeds"},"next":{"slug":"fine-tuning-an-embedding-model","title":"Fine-tuning an embedding model","url":"https://addaly.com/learn/fine-tuning/fine-tuning-an-embedding-model"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Multi-GPU without tears', in Fine-Tuning, and When Not To. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/fine-tuning/multi-gpu-without-tears (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Multi-GPU without tears","courseTitle":"Fine-Tuning, and When Not To","module":{"index":6,"slug":"running-the-job","title":"The mechanics of a run"},"answer":null,"keyIdea":"Adding GPUs multiplies the effective batch size by the device count, so gradient accumulation must be divided or the learning rate raised; and sharded strategies over PCIe often give back in communication what they gain in parallelism, which is why a single card is the faster path for any run under about twelve hours.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-different-reasons-to-use-more-than-one-card","runs":[{"text":"Two different reasons to use more than one card"}]},{"t":"p","runs":[{"text":"They need different tools and confusing them wastes days."}]},{"t":"p","runs":[{"text":"The model fits on one GPU and you want it faster.","b":true},{"text":" Use data parallelism: a full copy of the model on each card, different batches on each, gradients averaged across them every step. In PyTorch this is DDP."}]},{"t":"p","runs":[{"text":"The model does not fit on one GPU.","b":true},{"text":" Use sharding: FSDP or DeepSpeed ZeRO, splitting the optimiser state, the gradients and optionally the parameters across cards. This is the subject of the full-fine-tuning lesson."}]},{"t":"p","runs":[{"text":"For almost all LoRA and QLoRA work, the answer is the first. A quantised 8B model fits on one card, so more cards means more throughput, not more capacity."}]},{"t":"h2","id":"launching","runs":[{"text":"Launching"}]},{"t":"code","lang":"bash","label":"Bash","text":"accelerate launch --num_processes 4 train.py\n# or\ntorchrun --nproc_per_node 4 train.py"},{"t":"p","runs":[{"text":"Your training script does not change. Both launchers start one process per GPU and set the environment variables the libraries read."}]},{"t":"h2","id":"the-silent-batch-size-multiplication","runs":[{"text":"The silent batch-size multiplication"}]},{"t":"p","runs":[{"text":"This is the error that catches everybody once."}]},{"t":"code","lang":"","label":"Text","text":"effective = per_device_batch_size × gradient_accumulation_steps × num_gpus"},{"t":"p","runs":[{"text":"A config tuned on one card with "},{"text":"per_device=2, accum=8","code":true},{"text":" has an effective batch of 16. Launch it on four GPUs, unchanged, and the effective batch is "},{"text":"64","b":true},{"text":". Four times the tokens per step, a quarter of the optimiser steps for the same data, and a learning rate now sized for a batch a quarter as large."}]},{"t":"p","runs":[{"text":"The run completes. The loss curve is plausible. The result is worse and nobody knows why."}]},{"t":"p","runs":[{"text":"When you add GPUs, divide "},{"text":"gradient_accumulation_steps","code":true},{"text":" by the same factor ("},{"text":"accum=2","code":true},{"text":" on four cards keeps the effective batch at 16) or deliberately keep the larger batch and raise the learning rate to match. Either is fine. Doing neither is the bug."}]},{"t":"h2","id":"when-two-gpus-are-slower-than-one","runs":[{"text":"When two GPUs are slower than one"}]},{"t":"p","runs":[{"text":"Sharded strategies move tensors between cards constantly. On a machine where the GPUs are connected by PCIe rather than NVLink (which includes most consumer multi-GPU builds and many cheaper rented instances) that communication can dominate."}]},{"t":"p","runs":[{"text":"Rules of thumb:"}]},{"t":"ul","items":[[{"text":"DDP over PCIe","b":true},{"text":": fine. One gradient all-reduce per step, and with gradient accumulation you can make that per "},{"text":"several","i":true},{"text":" forward passes."}],[{"text":"FSDP or ZeRO-3 over PCIe","b":true},{"text":": often disappointing. Parameters are gathered and released every layer, every step."}],[{"text":"ZeRO-3 with CPU offload over PCIe","b":true},{"text":": expect a large slowdown. It is for making things possible, not fast."}]]},{"t":"p","runs":[{"text":"Measure rather than assume. Run 50 steps on one card and 50 on four, and compare seconds per step against tokens per step. If four cards are not giving you close to three times the throughput, the communication is eating it."}]},{"t":"h2","id":"two-practical-traps","runs":[{"text":"Two practical traps"}]},{"t":"p","runs":[{"text":"Every process runs the whole script.","b":true},{"text":" Without guards, four processes will each download your dataset, each write to the same log file, and each try to push to the Hub. Guard side effects:"}]},{"t":"code","lang":"python","label":"Python","text":"if accelerator.is_main_process:\n    dataset.save_to_disk(path)"},{"t":"p","runs":[{"text":"The Hugging Face "},{"text":"Trainer","code":true},{"text":" handles logging and saving correctly on its own; it is your own code around it that needs the guard."}]},{"t":"p","runs":[{"text":"Data loader workers.","b":true},{"text":" "},{"text":"dataloader_num_workers=4","code":true},{"text":" with four GPUs is sixteen worker processes. On a small dataset this is pure overhead and a common source of hangs at startup. Start at 2 and raise it only if the GPUs are visibly waiting."}]},{"t":"h2","id":"when-not-to-bother","runs":[{"text":"When not to bother"}]},{"t":"p","runs":[{"text":"A two-hour LoRA run does not need multi-GPU. The complexity (a launcher, a batch-size recalculation, the debugging of a hang that only occurs with four processes) costs more than the ninety minutes it saves."}]},{"t":"p","runs":[{"text":"The threshold in practice is around a day. If a run would take more than about twelve hours on one card, the extra cards are worth the setup. Below that, a single card and a config you trust is the faster path to an answer, and one GPU has the additional merit that every bug is your bug rather than a distributed one."}]},{"t":"p","runs":[{"text":"There is also a better use for a second card that nobody suggests: run two "},{"text":"different","i":true},{"text":" configurations on the two GPUs simultaneously rather than one configuration across both. An ablation finishes in the same wall-clock time as a single run, the setup is nothing more than two terminals, and you end the day with a comparison rather than with one number that arrived faster."}]}],"check":{"question":[{"text":"A config tuned on one GPU (per_device=2, accum=8, lr=2e-4) is launched unchanged on four GPUs with DDP. The run finishes and the model is worse. What happened?"}],"options":[{"text":[{"text":"DDP averages gradients across devices, which shrinks the update by a factor of four and undertrains the adapter."}],"correct":false,"tempting":[{"text":"Misreads averaging as attenuation; averaging is what makes four devices equivalent to one larger batch, not a division of the step size."}]},{"text":[{"text":"Each GPU trained its own adapter copy, and only rank zero’s was saved, discarding three quarters of the learning."}],"correct":false,"tempting":[{"text":"Describes a synchronisation failure that DDP specifically prevents; gradients are all-reduced so every replica stays identical."}]},{"text":[{"text":"The effective batch became 64 rather than 16, so there were a quarter as many optimiser steps."}],"correct":true,"tempting":null},{"text":[{"text":"Four processes each shuffled the data with the same seed, so every device trained on the identical batches."}],"correct":false,"tempting":[{"text":"Assumes sharding is broken; the distributed sampler assigns disjoint slices per rank, which is exactly what makes the batch larger."}]}],"order":[3,1,2,0],"why":null},"prev":{"slug":"reproducibility-and-seeds","title":"Reproducibility, and the noise floor","minutes":9},"next":{"slug":"fine-tuning-an-embedding-model","title":"Fine-tuning an embedding model","minutes":10}}}