Best GPU for Local LLMs in 2026: Every Budget Tier

updated 2026-09-10 · verified 2026-08-04

Prices verified September 2026 against our GPU database snapshot. The used market moves fast — the 2026 DRAM price surge alone pushed used 3060s up ~40% in two months — so treat every number here as a dated snapshot.

The one rule that matters

VRAM first, bandwidth second, everything else last.

The reason is how inference works: to generate each token, the GPU reads the entire model weights once. In single-user chat the batch is tiny, the compute cores sit mostly idle, and the bottleneck is almost purely “how many GB per second can VRAM deliver.” So:

• If the model doesn’t fit, the card is out — or you fall off the CPU-offload speed cliff (exact numbers below).
• Once it fits, generation speed ≈ memory bandwidth ÷ weight bytes read per token. CUDA core counts and Tensor core generations barely move tok/s.

Corollary: when comparing two cards, compare VRAM, then bandwidth, then stop. The 4090 has more than double the 3090’s compute, but the same 24GB and only 8% more bandwidth — so it’s about 8% faster at LLM inference. The premium buys gaming and image-generation performance, not tokens.

Step 1: do the VRAM math before you shop

Total requirement = weights + KV cache + runtime overhead (~1.5GB).

Weights are set by quantization. From measured GGUF files (our data — embedding tables and metadata push these slightly above theoretical bit-rates):

Quantization	GB per 1B params
FP16	2.00
Q8_0	1.06
Q6_K	0.82
Q4_K_M (community default)	0.61
Q3_K_M	0.50

KV cache grows linearly with context and is the most commonly ignored term. Qwen3-32B: ~2.1GB at 8K context, but 8.6GB at its full 32K — larger than the VRAM gap between many adjacent GPU tiers. Quantizing the KV cache to q8 halves it, and is the cheapest VRAM saver for long-context work.

Worked examples (Q4_K_M, 8K context, fp16 KV):

Model	Weights	KV cache	Total	On a 24GB card
Llama-3.1-8B	4.9 GB	1.1 GB	≈7.5 GB	comfortable
Phi-4 (14.7B)	9.0 GB	1.7 GB	≈12.2 GB	comfortable
Mistral-Small-24B	14.7 GB	1.3 GB	≈17.5 GB	comfortable
Qwen3-32B	20.1 GB	2.1 GB	≈23.7 GB	tight
Llama-3.3-70B	43.2 GB	2.7 GB	≈47.4 GB	impossible on one card; tight on 2×24GB

Two traps people fall into:

• Long context breaks the conclusion. Qwen3-32B is “tight” on 24GB at 8K context and flat-out infeasible at 32K. The 32GB 5090 is what makes it comfortable at 8K and workable at 32K.
• MoE saves speed, not VRAM. Qwen3-30B-A3B activates only 3.3B parameters per token, but all 30.5B must be resident — it still needs a 24GB-class card.

If you’d rather not do this by hand, our VRAM checker runs the verdict for any model × GPU combination, KV cache and context included.

Step 2: estimating speed

Theoretical generation speed ≈ bandwidth × 0.75 ÷ per-token weight bytes. The 0.75 is an empirical efficiency factor; real results vary ±30% with framework, driver, and CPU — use it to compare tiers, not to predict exact numbers. At Q4_K_M:

GPU	Bandwidth	8B model	32B model
RTX 3060 12GB	360 GB/s	≈55 tok/s	doesn’t fit
RTX 3090 24GB	936 GB/s	≈143 tok/s	≈35 tok/s
RTX 4090 24GB	1,008 GB/s	≈154 tok/s	≈38 tok/s
RTX 5090 32GB	1,792 GB/s	≈273 tok/s	≈67 tok/s

For reference: above 20 tok/s chat feels instant, 10 tok/s is reading speed, below 5 gets painful.

The MoE dividend shows up here. On the same 3090, Qwen3-30B-A3B reads only its 3.3B active parameters per token — theoretical ~350 tok/s, ten times the 32B dense model. That’s the root cause of MoE dominating local deployment in 2026.

The offload cliff. Partially offload a 70B Q4 to DDR5 system memory (~80 GB/s) and the formula takes min(936, 80): theoretical speed ≈1.4 tok/s. Going from “fits” to “doesn’t fit” isn’t a 20% penalty — it’s a 25× collapse. This is why the capacity rule comes first.

Budget tiers (September 2026 prices)

~$260 — RTX 3060 12GB (used)

The honest entry point. 12GB runs every 7B–8B model at Q4–Q6 with room to spare and 14B-class models (Phi-4) at Q4; 360 GB/s means an 8B generates around 55 tok/s, fine for everyday chat. Note the market shift: the DRAM surge pushed it from $170–220 in July to $250–275, and NVIDIA relisted new cards at $339 this year — above ~$280 used, just buy new. Ideal if you’re starting with Ollama and don’t yet know whether you’re an “8B is enough” or “32B or nothing” user. Don’t overbuy before you know.

~$650–800 — RX 7900 XTX 24GB (used, ~$650) / RTX 4070 Ti Super 16GB (used, ~$760)

The AMD option is the VRAM dark horse at this price: ~$650 for 24GB and 960 GB/s, paper specs right against the 3090. The cost is ROCm — workable in llama.cpp, not seamless, and you’ll debug drivers yourself. If you’re comfortable troubleshooting, it’s the cheapest road to 24GB; if not, pay up for the 3090.

The NVIDIA alternative, the 4070 Ti Super, has only 16GB: comfortable for 14B at Q4, tight for gpt-oss-20b (~14.7GB total), and out of the question for 24B+. The memory price surge pushed used units to $720–800, squeezing it between the 7900 XTX and the 3090. It makes sense mainly if you care about power draw and only run models up to 14B (285W — noticeably thriftier than a 3090).

~$1,100 — RTX 3090 24GB (used) — the value king

Still the answer in September 2026. 24GB at ~$1,100 runs everything through Qwen3-32B/Q4, including 30B-class MoE models, with 936 GB/s delivering 35 tok/s on a 32B. Two of them ($2,200) run Llama-3.3-70B tight — modeled at ~19 tok/s after the 0.85 multi-GPU bandwidth penalty; see our multi-GPU guide for details.

The costs, in writing: a card launched in 2020, 350W, zero warranty, and GDDR6X memory with a reputation for heat. The inspection checklist below is not optional.

~$1,800 — Mac mini M4 Pro 48GB (new)

The underrated silent route. Unified memory at the 75% usability rule gives you ~36GB: Qwen3-32B/Q4 fits comfortably, but 273 GB/s bandwidth means ~10 tok/s on a 32B and ~40 tok/s on an 8B. It’s for people who need big models to fit, can tolerate slow, and hate fan noise and power bills — ~50W for the whole machine, brand new with warranty, a combination no used NVIDIA card offers.

~$2,400 — RTX 4090 24GB (used)

Discontinued, and its price now anchors to the 5090 rather than its launch MSRP — up another ~10% in the last three months. Same 24GB as the 3090 with only 8% more bandwidth: for pure LLM inference, the ~$1,300 premium over a 3090 buys you ~10% speed. Unless you also game or run SD/flux image work, that money has no justification.

~$2,800 — RTX A6000 48GB (used)

The only single card with 48GB: a 70B Q4 (~47.4GB total) fits tight, with no multi-GPU, PSU, or cooling-stack hassle. But bandwidth is only 768 GB/s — a 70B generates at ~13 tok/s, slower than 2×3090 and more expensive. You buy it for the single-slot blower, ECC memory, and zero assembly, not for value.

$1,999 MSRP (street ~$3,600+) — RTX 5090 32GB (new)

The best new card for LLMs, period. 32GB finally makes 32B-class models comfortable on one card — Qwen3-32B fits even at its full 32K context — and 1,792 GB/s is a 78% bandwidth jump over the 4090 that converts almost linearly into speed: ~67 tok/s on a 32B, ~273 on an 8B.

The 2026 reality is availability: new cards trade around $4,300 on secondary markets, used from $3,600 — roughly 2.1× MSRP. Worth it at MSRP. At double the price for a 32GB card, split the same money across two 3090s and take 48GB instead.

$2,899–5,299 — Mac Studio (M4 Max 64GB / M3 Ultra 96GB)

A different philosophy: enormous unified memory, silent, ~100W. At the 75% rule, the 96GB M3 Ultra gives you ~72GB usable — enough for gpt-oss-120b (native MXFP4 weights ≈62GB), which no single consumer NVIDIA card can hold. The trade is bandwidth: 819 GB/s runs a dense 70B at ~14 tok/s — fine for chat, frustrating for agent loops. MoE is the exception: gpt-oss-120b reads only 5.1B active parameters per token, good for a theoretical ~200+ tok/s. Big-memory MoE is where the Mac earns its price.

Note: Apple discontinued the 256GB/512GB M3 Ultra configs in 2026 (DRAM shortage) — 96GB is the ceiling, and the base price rose from $3,999 to $5,299.

Used-card checklist (mandatory for 3090/4090)

• Check seller history and ratings; only buy from accounts with real sales records. Use platform escrow, keep chat records.
• On arrival, run 30+ minutes at full load (running a 32B model is a ready-made stress test) while watching memory temps — the 3090’s backside GDDR6X is a known hotspot, and sustained 100°C+ means thermal pads need replacing. Price that into the deal.
• Run a VRAM stress test. Artifacting, errors, or driver crashes mean an immediate return.
• Ask whether the card was opened or mined on. Mining cards aren’t an automatic no — but the price has to drop another notch (see “what to skip”).
• For multi-GPU builds, also confirm: a quality 1,200W+ PSU (two 350W cards with transient spikes), physical x16 slots, and a case airflow plan that can handle stacked cards.

Common misconceptions

• “More compute means faster inference.” No. Single-user generation is bandwidth-bound; the 4090’s doubled compute converts to only its 8% bandwidth edge on LLMs.
• “Buy for the max context length.” KV cache inflates linearly with context. Pick your card on an 8K verdict first; pay the VRAM premium for 32K+ only once you’ve confirmed you actually need it.
• “MoE models need less VRAM.” They save per-token reads (speed), not capacity. A 30B MoE still needs a 24GB card.
• “Two cards double your speed.” Cross-card sync has a cost; we model ×0.85. Two 3090s deliver ~1,590 GB/s effective, not 1,872.
• “A 96GB Mac holds 96GB models.” macOS caps GPU memory allocation at ~75% of physical RAM — 96GB means ~72GB usable.
• “2GB short? Just offload to CPU.” Offloaded layers read from ~80 GB/s system memory and speed collapses by an order of magnitude. Fine for a one-off evaluation, wrong for daily chat.

What to skip

• 8GB cards (4060, 5060, 3070): fine for games, dead on arrival for modern LLMs — a single 8B model at Q4 plus KV cache already needs ~7.5GB.
• H100/A100 for personal use: $15k–30k buys NVLink, ECC, and virtualization — datacenter features. Single-user inference obeys the same bandwidth formula, and the H100’s 3.35 TB/s won’t make chat 30× nicer. Rent by the hour when you need one.
• Mining-grade used cards under $500: in 2026 that price doesn’t even buy 16GB, and the savings don’t survive one dead card.

Quick answers

I want to run…	Buy
7B–8B models	used RTX 3060 12GB
14B–24B models	used RTX 3090; RX 7900 XTX if you tolerate tinkering
32B models	used RTX 3090 (tight); RTX 5090 at MSRP (comfortable)
70B models	2× used 3090, or Mac Studio 96GB
120B MoE models	Mac Studio 96GB, or 4× 3090
DeepSeek-R1 671B	nothing — rent cloud GPU

Next steps

Once you’ve picked a target model, run the exact verdict — KV cache and context included — in our VRAM checker, then check the cost comparator for total power draw. A 350W used card versus a 100W Mac, left running all year, slowly changes the value math.

FAQ

Sources

• NVIDIA GeForce specifications
• Apple Mac Studio technical specifications
• llama.cpp documentation (multi-GPU, CPU offload, KV cache quantization)
• Measured GGUF file sizes (bartowski, Hugging Face)