{"course":{"slug":"running-models-yourself","title":"Running Models Yourself","url":"https://addaly.com/learn/running-models-yourself","level":"intermediate","track":"deep"},"lesson":{"slug":"more-than-one-gpu","title":"More than one GPU","url":"https://addaly.com/learn/running-models-yourself/more-than-one-gpu","position":16,"of":82,"minutes":9,"answer":"A second card doubles memory almost for free with a layer split, speeds one user only under tensor parallelism and only over a fast link.","keyIdea":"A second card doubles memory almost for free with a layer split, speeds one user only under tensor parallelism and only over a fast link, and often serves concurrent users best as two independent copies of the model."},"body":{"format":"markdown","text":"## Two ways to use two cards\n\nAdding a second card can double your memory, double your throughput, or do neither, depending entirely on which strategy the engine uses and how the cards are connected.\n\n**Layer split (pipeline).** Layers 0 to 19 on card A, 20 to 39 on card B. A token flows through A, its activation crosses to B, and B finishes. This is llama.cpp's default and vLLM's `--pipeline-parallel-size`.\n\n- What crosses between cards is one small activation tensor per layer boundary: kilobytes. PCIe handles it comfortably.\n- Memory is the sum of both cards, so a 70B fits across two 24 GB cards.\n- Speed for a single user is **not** doubled. Card A works while B waits, then B works while A waits. Only one card is busy at a time. Expect the speed of one card, sometimes slightly less.\n- Throughput does improve with several concurrent requests, because the pipeline can be kept full.\n\n**Tensor parallel.** Every layer is split across both cards; each computes part of every matrix multiply and they combine results.\n\n- Both cards work simultaneously, so a single user does get faster: commonly 1.5 to 1.8x on two cards, not 2x.\n- Every layer requires an all-reduce between cards. That is a real synchronisation, several times per layer, at every token.\n- Therefore it is extremely sensitive to the interconnect. With NVLink between cards it works beautifully. Over PCIe 4.0 x16 it still works and gives most of the benefit. Over PCIe 3.0 x4 (which is what a second slot on a consumer board often is) the communication can cost more than the parallelism gains, and one card alone may be faster.\n\n```bash\nvllm serve <model> --tensor-parallel-size 2\n```\n\nA practical constraint people meet late: tensor parallelism generally requires the number of attention heads to divide by the number of cards. A model with 14 KV heads does not split cleanly across four.\n\n## Mixed and mismatched cards\n\nllama.cpp will happily use two different cards and lets you set the ratio:\n\n```bash\nllama-server -m model.gguf -ngl 99 --split-mode layer --tensor-split 24,8\n```\n\nThat puts three quarters of the layers on the first card. Useful, and note the consequence: the slower card sets the pace for the layers it holds, so a fast card paired with a slow one lands somewhere in between rather than at the average of their specifications. vLLM with tensor parallelism is far less tolerant: it assumes identical cards and runs everything at the pace of the weakest.\n\n## The costs nobody mentions in the forum post\n\n**Power and the supply.** Two 300 W cards plus a system is 800 W under load. A 650 W supply will shut down mid-generation, which looks like a driver crash and is not one. Transient spikes on modern cards are well above their rated draw.\n\n**Physical space and airflow.** Two large cards in adjacent slots starve the top one of air; it will throttle, and then your matched pair is no longer matched.\n\n**Lanes.** Most consumer boards give x16 to the first slot and x4 to the second, often shared with an NVMe drive. Check the manual before assuming x8/x8.\n\n**Software.** Two cards of the same vendor and generation is a well-trodden path. Mixing vendors (an NVIDIA card and an AMD card in one machine) is a Vulkan-backend project, not a configuration.\n\n## When a second card is the right purchase\n\n- **You need the memory.** This is the strong case. A 70B or a large MoE that does not fit on one card fits on two, and layer split does that with almost no penalty.\n- **You serve concurrent users.** Two cards running two copies of the same model behind a load balancer is often better than one model split across both: no interconnect traffic, no synchronisation, and linear throughput. This is the configuration people underuse.\n- **You want single-user speed.** Weakest case. Tensor parallel over consumer PCIe gives perhaps 1.5x for a great deal of complexity, and a single faster card usually costs less than the trouble.\n\n## Measure before you commit\n\nIf you already have two cards, test all three arrangements on your own workload: layer split, tensor parallel, and two independent servers. On consumer hardware the third frequently wins on throughput and always wins on simplicity, and it is the one nobody tries first."},"check":{"question":"Two identical 24 GB cards, the second in a PCIe 3.0 x4 slot, run a 32B model that fits on one card with short context. The goal is maximum total throughput for twenty concurrent users. What is the most promising arrangement?","options":[{"text":"Tensor parallelism across both cards, since it is the only mode where both cards compute simultaneously.","correct":false,"tempting":"Chooses the mode most dependent on interconnect speed and puts it on the slowest possible link, where all-reduce traffic can exceed the gain."},{"text":"A layer split across both cards, freeing memory for a much larger KV cache and therefore more concurrent sequences.","correct":false,"tempting":"Buys capacity that is not the constraint (the model already fits) and leaves one card idle at any moment for a single request."},{"text":"Two independent servers, one model copy per card, behind a load balancer.","correct":true},{"text":"One card serving the model with the second dedicated to KV cache storage over PCIe.","correct":false,"tempting":"Describes an arrangement no mainstream engine implements, and would move cache traffic onto the slowest link in the machine."}]},"prev":{"slug":"cpu-inference-honestly","title":"CPU inference, honestly","url":"https://addaly.com/learn/running-models-yourself/cpu-inference-honestly"},"next":{"slug":"measuring-your-machine","title":"Measuring what you actually have","url":"https://addaly.com/learn/running-models-yourself/measuring-your-machine"},"authorship":"Founder and series editor: Dr Mustafa Mun. Written by AI models to a written standard. Exam keys checked blind by a second model: 3,665 of 3,666 agreed. Academic review invited.","review":{"status":"Draft, open for academic review"},"cite":"Addaly (2026) 'More than one GPU', in Running Models Yourself. Edited by M. Mun. Addaly. Available at: https://addaly.com/learn/running-models-yourself/more-than-one-gpu (Accessed: 13 October 2026).","licence":{"name":"CC BY-NC-SA 4.0","url":"https://creativecommons.org/licenses/by-nc-sa/4.0/"},"reader":{"schema":1,"title":"More than one GPU","courseTitle":"Running Models Yourself","module":{"index":2,"slug":"the-memory-arithmetic","title":"The memory arithmetic"},"answer":null,"keyIdea":"A second card doubles memory almost for free with a layer split, speeds one user only under tensor parallelism and only over a fast link, and often serves concurrent users best as two independent copies of the model.","hero":{"kind":"plate","image":null,"alt":null,"caption":null},"blocks":[{"t":"h2","id":"two-ways-to-use-two-cards","runs":[{"text":"Two ways to use two cards"}]},{"t":"p","runs":[{"text":"Adding a second card can double your memory, double your throughput, or do neither, depending entirely on which strategy the engine uses and how the cards are connected."}]},{"t":"p","runs":[{"text":"Layer split (pipeline).","b":true},{"text":" Layers 0 to 19 on card A, 20 to 39 on card B. A token flows through A, its activation crosses to B, and B finishes. This is llama.cpp’s default and vLLM’s "},{"text":"--pipeline-parallel-size","code":true},{"text":"."}]},{"t":"ul","items":[[{"text":"What crosses between cards is one small activation tensor per layer boundary: kilobytes. PCIe handles it comfortably."}],[{"text":"Memory is the sum of both cards, so a 70B fits across two 24 GB cards."}],[{"text":"Speed for a single user is "},{"text":"not","b":true},{"text":" doubled. Card A works while B waits, then B works while A waits. Only one card is busy at a time. Expect the speed of one card, sometimes slightly less."}],[{"text":"Throughput does improve with several concurrent requests, because the pipeline can be kept full."}]]},{"t":"p","runs":[{"text":"Tensor parallel.","b":true},{"text":" Every layer is split across both cards; each computes part of every matrix multiply and they combine results."}]},{"t":"ul","items":[[{"text":"Both cards work simultaneously, so a single user does get faster: commonly 1.5 to 1.8x on two cards, not 2x."}],[{"text":"Every layer requires an all-reduce between cards. That is a real synchronisation, several times per layer, at every token."}],[{"text":"Therefore it is extremely sensitive to the interconnect. With NVLink between cards it works beautifully. Over PCIe 4.0 x16 it still works and gives most of the benefit. Over PCIe 3.0 x4 (which is what a second slot on a consumer board often is) the communication can cost more than the parallelism gains, and one card alone may be faster."}]]},{"t":"code","lang":"bash","label":"Bash","text":"vllm serve <model> --tensor-parallel-size 2"},{"t":"p","runs":[{"text":"A practical constraint people meet late: tensor parallelism generally requires the number of attention heads to divide by the number of cards. A model with 14 KV heads does not split cleanly across four."}]},{"t":"h2","id":"mixed-and-mismatched-cards","runs":[{"text":"Mixed and mismatched cards"}]},{"t":"p","runs":[{"text":"llama.cpp will happily use two different cards and lets you set the ratio:"}]},{"t":"code","lang":"bash","label":"Bash","text":"llama-server -m model.gguf -ngl 99 --split-mode layer --tensor-split 24,8"},{"t":"p","runs":[{"text":"That puts three quarters of the layers on the first card. Useful, and note the consequence: the slower card sets the pace for the layers it holds, so a fast card paired with a slow one lands somewhere in between rather than at the average of their specifications. vLLM with tensor parallelism is far less tolerant: it assumes identical cards and runs everything at the pace of the weakest."}]},{"t":"h2","id":"the-costs-nobody-mentions-in-the-forum-post","runs":[{"text":"The costs nobody mentions in the forum post"}]},{"t":"p","runs":[{"text":"Power and the supply.","b":true},{"text":" Two 300 W cards plus a system is 800 W under load. A 650 W supply will shut down mid-generation, which looks like a driver crash and is not one. Transient spikes on modern cards are well above their rated draw."}]},{"t":"p","runs":[{"text":"Physical space and airflow.","b":true},{"text":" Two large cards in adjacent slots starve the top one of air; it will throttle, and then your matched pair is no longer matched."}]},{"t":"p","runs":[{"text":"Lanes.","b":true},{"text":" Most consumer boards give x16 to the first slot and x4 to the second, often shared with an NVMe drive. Check the manual before assuming x8/x8."}]},{"t":"p","runs":[{"text":"Software.","b":true},{"text":" Two cards of the same vendor and generation is a well-trodden path. Mixing vendors (an NVIDIA card and an AMD card in one machine) is a Vulkan-backend project, not a configuration."}]},{"t":"h2","id":"when-a-second-card-is-the-right-purchase","runs":[{"text":"When a second card is the right purchase"}]},{"t":"ul","items":[[{"text":"You need the memory.","b":true},{"text":" This is the strong case. A 70B or a large MoE that does not fit on one card fits on two, and layer split does that with almost no penalty."}],[{"text":"You serve concurrent users.","b":true},{"text":" Two cards running two copies of the same model behind a load balancer is often better than one model split across both: no interconnect traffic, no synchronisation, and linear throughput. This is the configuration people underuse."}],[{"text":"You want single-user speed.","b":true},{"text":" Weakest case. Tensor parallel over consumer PCIe gives perhaps 1.5x for a great deal of complexity, and a single faster card usually costs less than the trouble."}]]},{"t":"h2","id":"measure-before-you-commit","runs":[{"text":"Measure before you commit"}]},{"t":"p","runs":[{"text":"If you already have two cards, test all three arrangements on your own workload: layer split, tensor parallel, and two independent servers. On consumer hardware the third frequently wins on throughput and always wins on simplicity, and it is the one nobody tries first."}]}],"check":{"question":[{"text":"Two identical 24 GB cards, the second in a PCIe 3.0 x4 slot, run a 32B model that fits on one card with short context. The goal is maximum total throughput for twenty concurrent users. What is the most promising arrangement?"}],"options":[{"text":[{"text":"Tensor parallelism across both cards, since it is the only mode where both cards compute simultaneously."}],"correct":false,"tempting":[{"text":"Chooses the mode most dependent on interconnect speed and puts it on the slowest possible link, where all-reduce traffic can exceed the gain."}]},{"text":[{"text":"A layer split across both cards, freeing memory for a much larger KV cache and therefore more concurrent sequences."}],"correct":false,"tempting":[{"text":"Buys capacity that is not the constraint (the model already fits) and leaves one card idle at any moment for a single request."}]},{"text":[{"text":"Two independent servers, one model copy per card, behind a load balancer."}],"correct":true,"tempting":null},{"text":[{"text":"One card serving the model with the second dedicated to KV cache storage over PCIe."}],"correct":false,"tempting":[{"text":"Describes an arrangement no mainstream engine implements, and would move cache traffic onto the slowest link in the machine."}]}],"order":[1,2,0,3],"why":null},"prev":{"slug":"cpu-inference-honestly","title":"CPU inference, honestly","minutes":9},"next":{"slug":"measuring-your-machine","title":"Measuring what you actually have","minutes":8}}}