Multi-GPU without tears

Two different reasons to use more than one card

They need different tools and confusing them wastes days.

The model fits on one GPU and you want it faster. Use data parallelism: a full copy of the model on each card, different batches on each, gradients averaged across them every step. In PyTorch this is DDP.

The model does not fit on one GPU. Use sharding: FSDP or DeepSpeed ZeRO, splitting the optimiser state, the gradients and optionally the parameters across cards. This is the subject of the full-fine-tuning lesson.

For almost all LoRA and QLoRA work, the answer is the first. A quantised 8B model fits on one card, so more cards means more throughput, not more capacity.

Launching

accelerate launch --num_processes 4 train.py
# or
torchrun --nproc_per_node 4 train.py

Your training script does not change. Both launchers start one process per GPU and set the environment variables the libraries read.

The silent batch-size multiplication

This is the error that catches everybody once.

effective = per_device_batch_size × gradient_accumulation_steps × num_gpus

A config tuned on one card with per_device=2, accum=8 has an effective batch of 16. Launch it on four GPUs, unchanged, and the effective batch is 64. Four times the tokens per step, a quarter of the optimiser steps for the same data, and a learning rate now sized for a batch a quarter as large.

The run completes. The loss curve is plausible. The result is worse and nobody knows why.

When you add GPUs, divide gradient_accumulation_steps by the same factor (accum=2 on four cards keeps the effective batch at 16) or deliberately keep the larger batch and raise the learning rate to match. Either is fine. Doing neither is the bug.

When two GPUs are slower than one

Sharded strategies move tensors between cards constantly. On a machine where the GPUs are connected by PCIe rather than NVLink (which includes most consumer multi-GPU builds and many cheaper rented instances) that communication can dominate.

Rules of thumb:

• DDP over PCIe: fine. One gradient all-reduce per step, and with gradient accumulation you can make that per several forward passes.
• FSDP or ZeRO-3 over PCIe: often disappointing. Parameters are gathered and released every layer, every step.
• ZeRO-3 with CPU offload over PCIe: expect a large slowdown. It is for making things possible, not fast.

Measure rather than assume. Run 50 steps on one card and 50 on four, and compare seconds per step against tokens per step. If four cards are not giving you close to three times the throughput, the communication is eating it.

Two practical traps

Every process runs the whole script. Without guards, four processes will each download your dataset, each write to the same log file, and each try to push to the Hub. Guard side effects:

if accelerator.is_main_process:
    dataset.save_to_disk(path)

The Hugging Face Trainer handles logging and saving correctly on its own; it is your own code around it that needs the guard.

Data loader workers. dataloader_num_workers=4 with four GPUs is sixteen worker processes. On a small dataset this is pure overhead and a common source of hangs at startup. Start at 2 and raise it only if the GPUs are visibly waiting.

When not to bother

A two-hour LoRA run does not need multi-GPU. The complexity (a launcher, a batch-size recalculation, the debugging of a hang that only occurs with four processes) costs more than the ninety minutes it saves.

The threshold in practice is around a day. If a run would take more than about twelve hours on one card, the extra cards are worth the setup. Below that, a single card and a config you trust is the faster path to an answer, and one GPU has the additional merit that every bug is your bug rather than a distributed one.

There is also a better use for a second card that nobody suggests: run two different configurations on the two GPUs simultaneously rather than one configuration across both. An ablation finishes in the same wall-clock time as a single run, the setup is nothing more than two terminals, and you end the day with a comparison rather than with one number that arrived faster.