Multi-GPU Setup for Local LLMs: NVLink, PCIe & Layer Splitting

updated 2026-08-04 · verified 2026-08-04

One GPU tops out at 32GB. The models you actually want to run need 48GB, 74GB, 96GB. Here’s how multi-GPU inference actually works — and where the marketing lies.

The two ways to split a model

Layer splitting (pipeline parallelism) is what llama.cpp and Ollama do by default: layers 1–40 on GPU 1, layers 41–80 on GPU 2. Each token flows through GPU 1, then its activations cross PCIe to GPU 2. Simple, works on any cards, and — crucially for inference — the inter-GPU traffic per token is tiny (just the activations, a few MB).

Tensor parallelism splits every layer across all cards (vLLM, some llama.cpp modes). Each token’s computation needs all cards talking constantly. This is faster at scale but demands serious interconnect bandwidth — this is what NVLink is for.

For single-user inference, layer splitting over PCIe is all you need. NVLink’s 900 GB/s is solving a problem you don’t have.

VRAM math: it just adds up

Two 24GB cards give you 48GB of usable capacity. Four give you 96GB. We don’t discount for tensor-parallel overhead in capacity (it’s small and the safety margins in our verdict tiers already absorb it):

Setup	Usable VRAM	Runs (Q4_K_M)
2× RTX 3090/4090	48 GB	Llama-3.3-70B (tight, ~47.4GB)
2× RTX A6000	96 GB	gpt-oss-120b (tight), 70B at Q8
4× RTX 3090	96 GB	same capacity, cheaper, more power
4× RTX 4090	96 GB	everything above, faster

Speed math: 0.85 penalty, not double

Multi-GPU does not double your bandwidth. Cross-card synchronization and imperfect splits cost about 15%:

Effective bandwidth = Σ card bandwidth × 0.85 (same-model cards)

2× RTX 4090: 2 × 1,008 × 0.85 = 1,714 GB/s effective → 70B Q4 at ~30 tok/s
2× RTX 3090: 2 × 936 × 0.85 = 1,591 GB/s effective → 70B Q4 at ~19 tok/s
4× RTX 3090: 4 × 936 × 0.85 = 3,182 GB/s effective → 70B Q4 at ~38 tok/s (if you needed the capacity anyway)

Mixed cards: the slowest card dominates. Pairing a 4090 with a 3060 gets you capacity but 3060-class speed on its layers. Don’t.

Practical checklist

PSU: two 350–450W cards want a quality 1,200W+ supply; transient spikes trip cheap units.
Motherboard: you need physical x16 slots, but PCIe 3.0 x8 electrical is fine for layer splitting.
Cooling: blower-style or liquid-cooled cards stack better; two open-air 4090s in adjacent slots will thermal-throttle.
Riser cables work. Many local-LLM rigs run cards on PCIe risers in open frames — bandwidth requirements are that low.
Unified memory alternative: before buying 4 GPUs, price out a 96GB Mac Studio. Slower per token, but silent, 100W, and zero assembly.

FAQ

Do I need NVLink for multi-GPU inference?

No. For single-user inference, layer splitting over plain PCIe is enough — each layer's weights are read independently, so inter-GPU traffic is small. NVLink matters for training and high-concurrency serving, not for one user chatting with a 70B model.

How much slower is 2×3090 than 1×4090-class bandwidth?

Our model applies a 0.85 bandwidth penalty for same-model multi-GPU: 2×936 GB/s ≈ 1,591 GB/s effective. A 70B Q4 model generates at roughly 19 tok/s on 2×3090 vs 30 tok/s on 2×4090.

Can I mix different GPU models?

Technically yes (llama.cpp splits layers proportionally), but the slowest card's bandwidth becomes the bottleneck for its share of layers. Mixing a 3090 with a 4060 works; expect speed closer to the slower card. Same-model pairs are strongly recommended.