More than one GPU

Two ways to use two cards

Adding a second card can double your memory, double your throughput, or do neither, depending entirely on which strategy the engine uses and how the cards are connected.

Layer split (pipeline). Layers 0 to 19 on card A, 20 to 39 on card B. A token flows through A, its activation crosses to B, and B finishes. This is llama.cpp’s default and vLLM’s --pipeline-parallel-size.

• What crosses between cards is one small activation tensor per layer boundary: kilobytes. PCIe handles it comfortably.
• Memory is the sum of both cards, so a 70B fits across two 24 GB cards.
• Speed for a single user is not doubled. Card A works while B waits, then B works while A waits. Only one card is busy at a time. Expect the speed of one card, sometimes slightly less.
• Throughput does improve with several concurrent requests, because the pipeline can be kept full.

Tensor parallel. Every layer is split across both cards; each computes part of every matrix multiply and they combine results.

• Both cards work simultaneously, so a single user does get faster: commonly 1.5 to 1.8x on two cards, not 2x.
• Every layer requires an all-reduce between cards. That is a real synchronisation, several times per layer, at every token.
• Therefore it is extremely sensitive to the interconnect. With NVLink between cards it works beautifully. Over PCIe 4.0 x16 it still works and gives most of the benefit. Over PCIe 3.0 x4 (which is what a second slot on a consumer board often is) the communication can cost more than the parallelism gains, and one card alone may be faster.

vllm serve <model> --tensor-parallel-size 2

A practical constraint people meet late: tensor parallelism generally requires the number of attention heads to divide by the number of cards. A model with 14 KV heads does not split cleanly across four.

Mixed and mismatched cards

llama.cpp will happily use two different cards and lets you set the ratio:

llama-server -m model.gguf -ngl 99 --split-mode layer --tensor-split 24,8

That puts three quarters of the layers on the first card. Useful, and note the consequence: the slower card sets the pace for the layers it holds, so a fast card paired with a slow one lands somewhere in between rather than at the average of their specifications. vLLM with tensor parallelism is far less tolerant: it assumes identical cards and runs everything at the pace of the weakest.

The costs nobody mentions in the forum post

Power and the supply. Two 300 W cards plus a system is 800 W under load. A 650 W supply will shut down mid-generation, which looks like a driver crash and is not one. Transient spikes on modern cards are well above their rated draw.

Physical space and airflow. Two large cards in adjacent slots starve the top one of air; it will throttle, and then your matched pair is no longer matched.

Lanes. Most consumer boards give x16 to the first slot and x4 to the second, often shared with an NVMe drive. Check the manual before assuming x8/x8.

Software. Two cards of the same vendor and generation is a well-trodden path. Mixing vendors (an NVIDIA card and an AMD card in one machine) is a Vulkan-backend project, not a configuration.

When a second card is the right purchase

• You need the memory. This is the strong case. A 70B or a large MoE that does not fit on one card fits on two, and layer split does that with almost no penalty.
• You serve concurrent users. Two cards running two copies of the same model behind a load balancer is often better than one model split across both: no interconnect traffic, no synchronisation, and linear throughput. This is the configuration people underuse.
• You want single-user speed. Weakest case. Tensor parallel over consumer PCIe gives perhaps 1.5x for a great deal of complexity, and a single faster card usually costs less than the trouble.

Measure before you commit

If you already have two cards, test all three arrangements on your own workload: layer split, tensor parallel, and two independent servers. On consumer hardware the third frequently wins on throughput and always wins on simplicity, and it is the one nobody tries first.