Multi-GPU Setup for Local LLMs: NVLink, PCIe & Layer Splitting

updated 2026-09-10 · verified 2026-08-04

One GPU tops out at 32GB. The models you actually want to run need 48GB, 74GB, 96GB. Here’s how multi-GPU inference actually works — and where the marketing lies.

Start with the one intuition that matters: during decoding, every token requires reading the entire model’s weights once. Memory bandwidth sets your tok/s; memory capacity decides whether the model fits at all. Every multi-GPU trade-off reduces to those two numbers.

The two ways to split a model

Layer splitting (pipeline parallelism) is what llama.cpp and Ollama do by default: layers 1–40 on GPU 1, layers 41–80 on GPU 2. Each token flows through GPU 1, then its activations cross PCIe to GPU 2. The key is how small that transfer is: Llama-3.3-70B has a hidden width of 8,192, so an fp16 activation crossing the boundary is 16KB. Even with KV handles attached, inter-GPU traffic per token is in the tens-of-KB range — against PCIe 3.0 x16’s ~16 GB/s, that’s microseconds, while reading 43GB of weights takes tens of milliseconds. The interconnect is simply not the bottleneck.

Tensor parallelism splits every layer across all cards (vLLM’s --tensor-parallel-size, some llama.cpp modes). Every layer’s computation ends in a cross-card all-reduce — at least two full rounds of communication per layer per token, orders of magnitude more traffic than layer splitting. It’s faster for throughput at scale, but demands serious interconnect bandwidth. This is what NVLink is for.

For single-user inference, layer splitting over PCIe is all you need. NVLink’s 900 GB/s is solving a problem you don’t have.

What NVLink is actually for

Facts first:

• The RTX 3090 was the last consumer card with an NVLink connector; the RTX 4090 and 5090 removed it physically.
• On datacenter cards NVLink is genuinely impressive — 600 GB/s on A100, 900 GB/s on H100 — but that bandwidth exists for training and tensor parallelism.
• In llama.cpp’s layer-split mode, bridging two 3090s with NVLink barely changes decode tok/s — each token crosses the boundary once, and the bottleneck stays at VRAM bandwidth, not the inter-GPU link.

When NVLink is actually worth paying for:

• Training and fine-tuning: gradient sync moves full model state every step and saturates PCIe.
• High-concurrency tensor-parallel serving: with dozens of concurrent sessions sharing a card pool, all-reduce traffic multiplies by session count and the interconnect starts to bind.
• Multi-user API serving in general: once you care about aggregate throughput rather than single-stream tok/s, vLLM + tensor parallelism + a real interconnect is the right architecture.

One person chatting with a 70B model? Skip the NVLink bridge and the exotic motherboard; spend it on cards with more bandwidth.

VRAM math: it just adds up

Two 24GB cards give you 48GB of usable capacity. Four give you 96GB. We don’t discount for tensor-parallel overhead in capacity — it’s small, and the safety margins in our verdict tiers already absorb it. Worked example, Llama-3.3-70B at Q4_K_M with 8K context:

weights 43.2GB (70.6B × 4.9 bpw ÷ 8) + KV cache 2.7GB + runtime overhead 1.5GB = 47.4GB

Against the verdict tiers: need ≤ 80% of usable is “comfortable”, ≤ 100% is “tight”, and the 100–120% band is the classic “needs-multi” zone. 47.4GB is hard-infeasible on one 24GB card (197% over) and “tight” on 2×24GB — it runs, but don’t expect longer contexts: the 70B KV cache costs ~327KB per token at fp16, so 32K context adds another 10.7GB and blows straight through a 48GB rig. For long context, either use q8 KV (halves it) or add capacity.

Setup	Usable VRAM	Runs (Q4_K_M, 8K context)
2× RTX 3090/4090	48 GB	Llama-3.3-70B (tight, needs 47.4GB)
2× RTX 5090	64 GB	70B comfortable, generous context headroom
2× RTX A6000	96 GB	gpt-oss-120b (73.6GB, comfortable), 70B at Q8 (79.3GB, tight)
4× RTX 3090	96 GB	same capacity, cheaper, more power
4× RTX 4090	96 GB	everything above, faster

Know the boundary too: DeepSeek-R1 671B needs ~411GB of Q4 weights — eight consumer cards won’t fit it. Multi-GPU is not infinite-capacity magic; its sweet spot is the 48–96GB tier.

Speed math: 0.85 penalty, not double

Multi-GPU does not double your bandwidth. Cross-card synchronization and imperfect splits cost about 15%:

Effective bandwidth = Σ card bandwidth × 0.85 (same-model cards) Theoretical tok/s = effective bandwidth × 0.75 efficiency ÷ per-token weight size

A 70B Q4_K_M model reads 43.2GB of weights per token. Plugging in each setup (theoretical estimates — frameworks, drivers, and CPU overhead move real numbers ±30%, often toward the lower half of the band):

Setup	Effective bandwidth	70B Q4 theoretical speed
2× RTX 3090	1,591 GB/s	~28 tok/s
2× RTX 4090	1,714 GB/s	~30 tok/s
2× RTX 5090	3,046 GB/s	~53 tok/s
2× RTX A6000	1,306 GB/s	~23 tok/s
4× RTX 3090	3,182 GB/s	~55 tok/s

Two corollaries worth remembering. First, 2×3090 is only ~7% slower than 2×4090 — 8% less bandwidth, while used prices in 2026 differ more than twofold (~$1,100 vs ~$2,400 per card). For pure LLM inference the 3090’s value is unmatched. Second, four cards run about twice as fast as two: the 0.85 penalty doesn’t deepen with card count — layer splitting scales well. The real costs are the PSU, cooling, and motherboard.

Mixed cards: the slowest card dominates. Our model takes min(bandwidth) as the bottleneck: pair a 4090 with a 3060 and effective bandwidth collapses to 360 GB/s — a theoretical ~6 tok/s on 70B, over 4× slower than two 3090s. You gained capacity and lost speed to nearly CPU-offload territory. Don’t.

The MoE exception: gpt-oss-120b has 116.8B total parameters but activates only 5.1B per token — about 3.1GB of weights read per token. The same 2×A6000 that theoretically manages 23 tok/s on a dense 70B does 300+ tok/s on gpt-oss-120b. High-capacity multi-GPU plus MoE models is an underrated 2026 combination.

Setup in practice

llama.cpp (layer split is the multi-GPU default):

llama-cli -m model.gguf -ngl 99 --split-mode layer --tensor-split 24,24

--tensor-split distributes layers by VRAM ratio — 1,1 for identical cards; --main-gpu picks which card holds the output layer. Ollama is simpler: it uses all visible GPUs automatically, and CUDA_VISIBLE_DEVICES hides the ones you don’t want. After loading, check nvidia-smi: VRAM should be distributed across cards in the expected ratio. If one card is full and the other idle, your split ratio or model placement is off.

vLLM users, note: --tensor-parallel-size 2 is tensor parallelism — built for multi-user throughput. It doesn’t necessarily beat layer splitting on single-stream latency, and all-reduce over consumer PCIe isn’t free.

Hardware checklist

• PSU: TDPs are 350W (3090), 450W (4090), 575W (5090). Two 450W cards plus a CPU want a quality 1,200W+ supply; transient spikes trip cheap units, and this is not the component to cheap out on.
• Electricity sense-check: 4×3090 at full draw is ~1,400W — roughly 1,000 kWh per month running 24/7. Do that math before scheduling batch inference.
• Motherboard: you need physical x16 slots, but PCIe 3.0 x8 electrical is fine for layer-split decode. Even x1 risers work — the cost is slower model loading.
• Cooling: blower-style or liquid-cooled cards stack better; two open-air 4090s in adjacent slots will thermal-throttle under sustained load.
• Riser cables work. Many local-LLM rigs run cards on PCIe risers in open frames — layer splitting’s bandwidth requirements are that low.
• The unified-memory alternative: before buying 4 GPUs, price out a 96GB Mac Studio. The M3 Ultra’s 819 GB/s bandwidth means a theoretical ~14 tok/s on 70B Q4 — about half of 2×3090 — but it’s silent, draws ~100W-class power, and needs zero assembly. Your tok/s requirements and your ears can’t both win.

Common misconceptions

• “VRAM adds up, so bandwidth adds up too”: capacity does add directly; bandwidth takes a 0.85 haircut, and with mixed cards it follows the slowest one. Three quantities, three rules.
• “No NVLink, no multi-GPU”: the opposite — layer splitting exists precisely to avoid depending on fast interconnects. The 4090/5090 dropping NVLink is a non-event for inference.
• “An NVLink bridge speeds up my two 3090s”: decode is bottlenecked on VRAM bandwidth; you’d be widening a road that isn’t congested.
• “Multi-GPU is good for fine-tuning”: multi-card fine-tuning on consumer cards is still a patchwork; summed VRAM does nothing for gradient sync’s bandwidth hunger. For serious training, rent cloud.
• “Splitting a model that fits on one card makes it faster”: if it fits, don’t split. A 32B Q4 needs 23.7GB — tight but viable on a single 24GB card; splitting across 2×12GB adds the 0.85 penalty and doubles your failure points. Multi-GPU is a capacity tool, not a speed tool.

Bottom line: multi-GPU buys capacity, bandwidth scales at 0.85, and NVLink is for training. Figure out how many GB your target model needs first, then count cards — run your model through the VRAM calculator before trusting any rule of thumb, including this one.

FAQ

Sources

• llama.cpp multi-GPU documentation
• vLLM documentation (tensor parallelism)
• NVIDIA NVLink product page
• NVIDIA GeForce RTX 5090 specifications (no NVLink)