Splitting a model across GPUs

When one card is not enough

A 70-billion-parameter model in bf16 is 140 gigabytes. A 405-billion model is 810. The large sparse models exceed a single card even at four bits. At some size the question stops being “which card” and becomes “how do I cut the model up”, and there are three answers, distinguished by what has to travel between the cards.

Pipeline parallelism: split by layer

Put layers one to forty on the first GPU and forty-one to eighty on the second. For each token, the first card runs its half and passes the residual-stream vector (a few kilobytes) to the second, which finishes. Almost nothing crosses the wire, so this works over any connection, including the slow PCIe bus between consumer cards, and it is what llama.cpp does by default when it sees more than one GPU.

The cost is that for a single user, only one card is working at any moment. The bandwidth wall applies per card: the first card reads its 70 gigabytes, then the second reads its 70, and the token takes as long as it would on one card with enough memory. Pipeline parallelism buys capacity, not speed. With many users, the cards can work on different sequences at once, and the idle gaps (pipeline bubbles) shrink, which is why servers use it across machines where nothing faster is possible.

Tensor parallelism: split every matrix

Instead of splitting the model by layer, split each weight matrix across the cards: the attention heads shared out, the feed-forward width divided. Each card computes its slice of every layer at the same time, and after each attention block and each feed-forward block the partial results are combined with an all-reduce operation.

Now the weights are read in parallel. Two cards read 70 gigabytes each at the same moment, and the effective bandwidth is the sum. Single-stream decode speeds up, nearly in proportion to the number of cards, which is the only way to make a very large model fast rather than merely runnable.

The price is communication: two all-reduces per layer, around 160 per token for an 80-layer model, each of which every card must wait for. Over NVLink, at close to a terabyte a second between cards in one machine, the waits are small and the scaling is good up to eight cards. Over PCIe, at a few tens of gigabytes a second, the waits dominate and the speed-up largely evaporates. The rule is that tensor parallelism stays inside one machine on a fast interconnect, and the number of cards must divide the number of attention heads. Serving engines expose it as a single setting: --tensor-parallel-size in vLLM.

Expert parallelism: split by expert

For a mixture-of-experts model, module two’s architecture suggests its own cut: put different experts on different cards. A token’s hidden state is sent to whichever card holds the expert it was routed to, and the result comes back: an all-to-all exchange rather than an all-reduce. This is the natural fit for sparse models and how the largest of them are served. Its characteristic problem is balance: if the router favours a few experts, the cards holding them are busy while the rest wait, and a good deal of engineering in these systems goes into keeping the load even.

Replicas: do not split at all

If the model fits on one card, the highest throughput per card comes from not splitting. Load a full copy on each, put a load balancer in front, and let each card run its own continuous batch. There is no communication, no waiting and no divisibility constraint. Tensor parallelism is for when the model does not fit or when single-stream speed matters more than throughput; for everything else, replicas win.

The decision, in one pass

• Fits on one card: replicas.
• Does not fit, fast interconnect available: tensor parallelism within the machine.
• Does not fit in one machine: tensor parallelism inside each machine, pipeline parallelism between them. Machine-to-machine links, even the fast ones at around fifty gigabytes a second, are closer to PCIe than to NVLink, so the thing that crosses them should be the small activation vector, not the all-reduce.
• Mixture of experts: expert parallelism, with attention to balance.

The consumer version

Two 24-gigabyte cards over PCIe hold a four-bit 70-billion model (40 gigabytes of weights, 8 left for cache) with a layer split, at the speed of one card. Tensor parallelism over PCIe is possible with some engines and gains a little. A single machine with 128 gigabytes of unified memory avoids the question entirely, at a lower bandwidth per byte, and that trade is why such machines became popular for local large models: no split, no interconnect, one memory pool at 400 to 800 gigabytes a second.

A note on training

Training has its own parallelism: sharding the optimiser states and gradients that module five said cost sixteen bytes per parameter across every card in a cluster, under names like FSDP and ZeRO. The principle is the same, the volumes are far larger, and it is a different course. For inference, the three cuts above and the choice not to cut are the whole menu.