{"course":{"slug":"how-llms-work","title":"How a Language Model Actually Works","url":"https://addaly.com/learn/how-llms-work","level":"intermediate","track":"start"},"lesson":{"slug":"splitting-a-model-across-gpus","title":"Splitting a model across GPUs","url":"https://addaly.com/learn/how-llms-work/splitting-a-model-across-gpus","position":64,"of":78,"minutes":8,"answer":"The three ways to cut a model across cards differ in what crosses the wire (a small activation vector for a layer split, an all-reduce per block for a tensor split.","keyIdea":"The three ways to cut a model across cards differ in what crosses the wire (a small activation vector for a layer split, an all-reduce per block for a tensor split, routed tokens for an expert split) and only the tensor split adds bandwidth and therefore speed."},"body":{"format":"markdown","text":"## When one card is not enough\n\nA 70-billion-parameter model in bf16 is 140 gigabytes. A 405-billion model is 810. The large sparse models exceed a single card even at four bits. At some size the question stops being \"which card\" and becomes \"how do I cut the model up\", and there are three answers, distinguished by **what has to travel between the cards**.\n\n## Pipeline parallelism: split by layer\n\nPut layers one to forty on the first GPU and forty-one to eighty on the second. For each token, the first card runs its half and passes the residual-stream vector (a few kilobytes) to the second, which finishes. Almost nothing crosses the wire, so this works over any connection, including the slow PCIe bus between consumer cards, and it is what `llama.cpp` does by default when it sees more than one GPU.\n\nThe cost is that for a single user, only one card is working at any moment. The bandwidth wall applies per card: the first card reads its 70 gigabytes, then the second reads its 70, and the token takes as long as it would on one card with enough memory. Pipeline parallelism buys **capacity**, not speed. With many users, the cards can work on different sequences at once, and the idle gaps (pipeline bubbles) shrink, which is why servers use it across machines where nothing faster is possible.\n\n## Tensor parallelism: split every matrix\n\nInstead of splitting the model by layer, split each weight matrix across the cards: the attention heads shared out, the feed-forward width divided. Each card computes its slice of every layer at the same time, and after each attention block and each feed-forward block the partial results are combined with an all-reduce operation.\n\nNow the weights are read in parallel. Two cards read 70 gigabytes each at the same moment, and the effective bandwidth is the sum. Single-stream decode speeds up, nearly in proportion to the number of cards, which is the only way to make a very large model *fast* rather than merely runnable.\n\nThe price is communication: two all-reduces per layer, around 160 per token for an 80-layer model, each of which every card must wait for. Over NVLink, at close to a terabyte a second between cards in one machine, the waits are small and the scaling is good up to eight cards. Over PCIe, at a few tens of gigabytes a second, the waits dominate and the speed-up largely evaporates. The rule is that tensor parallelism stays inside one machine on a fast interconnect, and the number of cards must divide the number of attention heads. Serving engines expose it as a single setting: `--tensor-parallel-size` in vLLM.\n\n## Expert parallelism: split by expert\n\nFor a mixture-of-experts model, module two's architecture suggests its own cut: put different experts on different cards. A token's hidden state is sent to whichever card holds the expert it was routed to, and the result comes back: an all-to-all exchange rather than an all-reduce. This is the natural fit for sparse models and how the largest of them are served. Its characteristic problem is balance: if the router favours a few experts, the cards holding them are busy while the rest wait, and a good deal of engineering in these systems goes into keeping the load even.\n\n## Replicas: do not split at all\n\nIf the model fits on one card, the highest throughput per card comes from not splitting. Load a full copy on each, put a load balancer in front, and let each card run its own continuous batch. There is no communication, no waiting and no divisibility constraint. Tensor parallelism is for when the model does not fit or when single-stream speed matters more than throughput; for everything else, replicas win.\n\n## The decision, in one pass\n\n- Fits on one card: replicas.\n- Does not fit, fast interconnect available: tensor parallelism within the machine.\n- Does not fit in one machine: tensor parallelism inside each machine, pipeline parallelism between them. Machine-to-machine links, even the fast ones at around fifty gigabytes a second, are closer to PCIe than to NVLink, so the thing that crosses them should be the small activation vector, not the all-reduce.\n- Mixture of experts: expert parallelism, with attention to balance.\n\n## The consumer version\n\nTwo 24-gigabyte cards over PCIe hold a four-bit 70-billion model (40 gigabytes of weights, 8 left for cache) with a layer split, at the speed of one card. Tensor parallelism over PCIe is possible with some engines and gains a little. A single machine with 128 gigabytes of unified memory avoids the question entirely, at a lower bandwidth per byte, and that trade is why such machines became popular for local large models: no split, no interconnect, one memory pool at 400 to 800 gigabytes a second.\n\n## A note on training\n\nTraining has its own parallelism: sharding the optimiser states and gradients that module five said cost sixteen bytes per parameter across every card in a cluster, under names like FSDP and ZeRO. The principle is the same, the volumes are far larger, and it is a different course. For inference, the three cuts above and the choice not to cut are the whole menu."},"check":{"question":"A team spreads a 70-billion-parameter model over four GPUs with a layer split and is surprised that a single user's tokens per second is no higher than it was on one larger card. Why?","options":[{"text":"In a layer split only one card works at a time for each token, so bandwidth is not aggregated.","correct":true},{"text":"The interconnect is the bottleneck, because the whole key-value cache must be copied between cards at every layer boundary.","correct":false,"tempting":"Interconnects do limit tensor parallelism, but a layer split transfers only the activation vector; the cache for each layer stays on the card that holds that layer."},{"text":"Four cards need roughly four times the batch size before any speed-up can appear, and a single user cannot supply it.","correct":false,"tempting":"Mixes in the batching crossover; batch size affects throughput for many users, but the single-user speed under a layer split is fixed by one card's bandwidth regardless of batch."},{"text":"Layer-split parallelism is only implemented for training, and inference engines silently fall back to running the whole model on the first card while the others hold copies of the weights.","correct":false,"tempting":"Invents a fallback; the layer split is real and is the default multi-GPU mode in common local engines, and the other cards genuinely hold their layers and do their share of the work, just not at the same time."}]},"prev":{"slug":"serving-many-users","title":"Serving a hundred people from one card","url":"https://addaly.com/learn/how-llms-work/serving-many-users"},"next":{"slug":"the-price-of-a-token","title":"The price of a token, from both sides","url":"https://addaly.com/learn/how-llms-work/the-price-of-a-token"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'Splitting a model across GPUs', in How a Language Model Actually Works. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/how-llms-work/splitting-a-model-across-gpus (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"Splitting a model across GPUs","courseTitle":"How a Language Model Actually Works","module":{"index":7,"slug":"running-it","title":"Running it: memory, speed and money"},"answer":null,"keyIdea":"The three ways to cut a model across cards differ in what crosses the wire (a small activation vector for a layer split, an all-reduce per block for a tensor split, routed tokens for an expert split) and only the tensor split adds bandwidth and therefore speed.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"when-one-card-is-not-enough","runs":[{"text":"When one card is not enough"}]},{"t":"p","runs":[{"text":"A 70-billion-parameter model in bf16 is 140 gigabytes. A 405-billion model is 810. The large sparse models exceed a single card even at four bits. At some size the question stops being “which card” and becomes “how do I cut the model up”, and there are three answers, distinguished by "},{"text":"what has to travel between the cards","b":true},{"text":"."}]},{"t":"h2","id":"pipeline-parallelism-split-by-layer","runs":[{"text":"Pipeline parallelism: split by layer"}]},{"t":"p","runs":[{"text":"Put layers one to forty on the first GPU and forty-one to eighty on the second. For each token, the first card runs its half and passes the residual-stream vector (a few kilobytes) to the second, which finishes. Almost nothing crosses the wire, so this works over any connection, including the slow PCIe bus between consumer cards, and it is what "},{"text":"llama.cpp","code":true},{"text":" does by default when it sees more than one GPU."}]},{"t":"p","runs":[{"text":"The cost is that for a single user, only one card is working at any moment. The bandwidth wall applies per card: the first card reads its 70 gigabytes, then the second reads its 70, and the token takes as long as it would on one card with enough memory. Pipeline parallelism buys "},{"text":"capacity","b":true},{"text":", not speed. With many users, the cards can work on different sequences at once, and the idle gaps (pipeline bubbles) shrink, which is why servers use it across machines where nothing faster is possible."}]},{"t":"h2","id":"tensor-parallelism-split-every-matrix","runs":[{"text":"Tensor parallelism: split every matrix"}]},{"t":"p","runs":[{"text":"Instead of splitting the model by layer, split each weight matrix across the cards: the attention heads shared out, the feed-forward width divided. Each card computes its slice of every layer at the same time, and after each attention block and each feed-forward block the partial results are combined with an all-reduce operation."}]},{"t":"p","runs":[{"text":"Now the weights are read in parallel. Two cards read 70 gigabytes each at the same moment, and the effective bandwidth is the sum. Single-stream decode speeds up, nearly in proportion to the number of cards, which is the only way to make a very large model "},{"text":"fast","i":true},{"text":" rather than merely runnable."}]},{"t":"p","runs":[{"text":"The price is communication: two all-reduces per layer, around 160 per token for an 80-layer model, each of which every card must wait for. Over NVLink, at close to a terabyte a second between cards in one machine, the waits are small and the scaling is good up to eight cards. Over PCIe, at a few tens of gigabytes a second, the waits dominate and the speed-up largely evaporates. The rule is that tensor parallelism stays inside one machine on a fast interconnect, and the number of cards must divide the number of attention heads. Serving engines expose it as a single setting: "},{"text":"--tensor-parallel-size","code":true},{"text":" in vLLM."}]},{"t":"h2","id":"expert-parallelism-split-by-expert","runs":[{"text":"Expert parallelism: split by expert"}]},{"t":"p","runs":[{"text":"For a mixture-of-experts model, module two’s architecture suggests its own cut: put different experts on different cards. A token’s hidden state is sent to whichever card holds the expert it was routed to, and the result comes back: an all-to-all exchange rather than an all-reduce. This is the natural fit for sparse models and how the largest of them are served. Its characteristic problem is balance: if the router favours a few experts, the cards holding them are busy while the rest wait, and a good deal of engineering in these systems goes into keeping the load even."}]},{"t":"h2","id":"replicas-do-not-split-at-all","runs":[{"text":"Replicas: do not split at all"}]},{"t":"p","runs":[{"text":"If the model fits on one card, the highest throughput per card comes from not splitting. Load a full copy on each, put a load balancer in front, and let each card run its own continuous batch. There is no communication, no waiting and no divisibility constraint. Tensor parallelism is for when the model does not fit or when single-stream speed matters more than throughput; for everything else, replicas win."}]},{"t":"h2","id":"the-decision-in-one-pass","runs":[{"text":"The decision, in one pass"}]},{"t":"ul","items":[[{"text":"Fits on one card: replicas."}],[{"text":"Does not fit, fast interconnect available: tensor parallelism within the machine."}],[{"text":"Does not fit in one machine: tensor parallelism inside each machine, pipeline parallelism between them. Machine-to-machine links, even the fast ones at around fifty gigabytes a second, are closer to PCIe than to NVLink, so the thing that crosses them should be the small activation vector, not the all-reduce."}],[{"text":"Mixture of experts: expert parallelism, with attention to balance."}]]},{"t":"h2","id":"the-consumer-version","runs":[{"text":"The consumer version"}]},{"t":"p","runs":[{"text":"Two 24-gigabyte cards over PCIe hold a four-bit 70-billion model (40 gigabytes of weights, 8 left for cache) with a layer split, at the speed of one card. Tensor parallelism over PCIe is possible with some engines and gains a little. A single machine with 128 gigabytes of unified memory avoids the question entirely, at a lower bandwidth per byte, and that trade is why such machines became popular for local large models: no split, no interconnect, one memory pool at 400 to 800 gigabytes a second."}]},{"t":"h2","id":"a-note-on-training","runs":[{"text":"A note on training"}]},{"t":"p","runs":[{"text":"Training has its own parallelism: sharding the optimiser states and gradients that module five said cost sixteen bytes per parameter across every card in a cluster, under names like FSDP and ZeRO. The principle is the same, the volumes are far larger, and it is a different course. For inference, the three cuts above and the choice not to cut are the whole menu."}]}],"check":{"question":[{"text":"A team spreads a 70-billion-parameter model over four GPUs with a layer split and is surprised that a single user’s tokens per second is no higher than it was on one larger card. Why?"}],"options":[{"text":[{"text":"In a layer split only one card works at a time for each token, so bandwidth is not aggregated."}],"correct":true,"tempting":null},{"text":[{"text":"The interconnect is the bottleneck, because the whole key-value cache must be copied between cards at every layer boundary."}],"correct":false,"tempting":[{"text":"Interconnects do limit tensor parallelism, but a layer split transfers only the activation vector; the cache for each layer stays on the card that holds that layer."}]},{"text":[{"text":"Four cards need roughly four times the batch size before any speed-up can appear, and a single user cannot supply it."}],"correct":false,"tempting":[{"text":"Mixes in the batching crossover; batch size affects throughput for many users, but the single-user speed under a layer split is fixed by one card’s bandwidth regardless of batch."}]},{"text":[{"text":"Layer-split parallelism is only implemented for training, and inference engines silently fall back to running the whole model on the first card while the others hold copies of the weights."}],"correct":false,"tempting":[{"text":"Invents a fallback; the layer split is real and is the default multi-GPU mode in common local engines, and the other cards genuinely hold their layers and do their share of the work, just not at the same time."}]}],"order":[3,2,0,1],"why":null},"prev":{"slug":"serving-many-users","title":"Serving a hundred people from one card","minutes":9},"next":{"slug":"the-price-of-a-token","title":"The price of a token, from both sides","minutes":9}}}