GPUFits
How Much VRAM per Billion Parameters? (2026 Cheat Sheet)
updated 2026-09-10 · verified 2026-08-04
The single most useful number in local LLMs is VRAM per billion parameters. Learn this table once and you can size any model in your head in five seconds.
The cheat sheet (weights only, measured)
| Quantization | Bits per weight | GB per 1B params | vs FP16 |
|---|---|---|---|
| FP16 | 16.0 | 2.00 GB | — |
| Q8_0 | 8.5 | 1.06 GB | −47% |
| Q6_K | 6.6 | 0.82 GB | −59% |
| Q5_K_M | 5.7 | 0.71 GB | −64% |
| Q4_K_M | 4.9 | 0.61 GB | −69% |
| Q3_K_M | 4.0 | 0.50 GB | −75% |
| Q2_K | 3.2 | 0.40 GB | −80% |
These are not the theoretical llama.cpp bit-rates — they’re measured from real GGUF files (Llama-3.1-8B, verified 2026-08-04). Measured values run slightly above theory because embedding tables and metadata don’t compress as much. On larger models the gap shrinks.
Why VRAM scales linearly with parameters
Every parameter in a model is one number that has to live in VRAM. At FP16 each number takes 2 bytes, so 1B parameters is almost exactly 2GB — no compression, no tricks, this is the boring part. Quantization just represents each number with fewer bits: Q4 squeezes weights to ~4 bits, which in theory means 0.5GB per billion.
But the measured value is 0.61GB. Where does the extra 20% go? Three places: quantization works on blocks, and every block stores its own scaling factors; K-quants deliberately keep sensitive tensors (embedding tables, output heads) at higher precision like Q6 or Q8; and GGUF files carry metadata. So never treat “bits ÷ 8” as exact — the error is worst on small models, where embedding tables make up a much larger share of an 8B model than of a 70B one.
The full formula
Total VRAM = weights (table above × params) + KV cache + overhead
- KV cache: roughly 0.5–4GB at 8K context for common models, grows linearly with context. Qwen3-32B goes from 2.1GB at 8K to 8.6GB at 32K.
- Overhead: ~1.5GB for the runtime itself (CUDA context etc.).
Why KV cache isn’t in the table
Because KV cache has nothing to do with parameter count. It depends on architecture constants: layers × KV heads × head dimension × context length. This produces a counterintuitive result — Gemma-3-27B (62 layers, 16 KV heads) needs about 4.2GB of KV cache at 8K context, more than Llama-3.3-70B (80 layers but only 8 KV heads) at 2.7GB. Bigger model, smaller KV share.
That’s why “GB per billion parameters” only covers weights. KV cache has to be computed per model — the formula and per-model numbers are in our KV cache guide.
Worked examples (Q4_K_M, 8K context)
| Model | Weights | KV cache | Overhead | Total | Verdict |
|---|---|---|---|---|---|
| Llama 3.2 3B | 2.0 GB | 0.9 GB | 1.5 GB | 4.4 GB | even an 8GB card works |
| Llama 3.1 8B | 4.9 GB | 1.1 GB | 1.5 GB | 7.5 GB | comfy on 12GB |
| Mistral Small 24B | 14.7 GB | 1.3 GB | 1.5 GB | 17.5 GB | comfy on 24GB; needs Q3 on 16GB |
| Qwen3 32B | 20.1 GB | 2.1 GB | 1.5 GB | 23.7 GB | tight on 24GB |
| Llama 3.3 70B | 43.2 GB | 2.7 GB | 1.5 GB | 47.4 GB | needs 2×24GB or a 48GB card |
| gpt-oss-120b (MoE) | 71.5 GB | 0.6 GB¹ | 1.5 GB | 73.6 GB | 96GB Mac or 4×24GB |
| DeepSeek-R1 671B | 411.0 GB | 0.6 GB¹ | 1.5 GB | 413.1 GB | cloud only |
¹ DeepSeek-R1 uses MLA attention and gpt-oss-120b uses GQA with sliding-window attention — both compress KV cache dramatically. See our KV cache guide.
What each GPU tier actually runs
Read the table backwards and you get what every card can handle. Our thresholds: need ≤ 80% of usable VRAM is “comfortable”, ≤ 100% is “tight”.
- 12GB (RTX 3060): 8B at Q4 comfortably (7.5GB); it can even stretch to 8B at Q8 (~11.1GB, tight). Anything past ~16B means dropping quants or changing cards.
- 16GB (RTX 4070 Ti Super): 8B at Q8 comfortably; a 24B model like Mistral Small doesn’t fit at Q4 (17.5GB), but Q3_K_M brings it to ~14.8GB — tight but workable.
- 24GB (RTX 3090/4090): 24B at Q4 comfortably; Qwen3-32B at Q4 is tight (23.7GB); Gemma-3-27B fits at 8K context (22.4GB), but stretch context to 32K and the KV cache balloons to 16.6GB — ~35GB total, game over.
- 32GB (RTX 5090): Qwen3-32B at Q4 with its full 32K context (~30.2GB, tight). A 70B won’t fit even at Q2 — that’s still ~32.2GB.
- 48GB (RTX A6000 / 2×24GB): the entry ticket for 70B at Q4 — 47.4GB lands exactly at “tight”.
- Apple unified memory: usable VRAM is 75% of nominal. A 48GB Mac mini M4 Pro gives you ~36GB, comfortable for Qwen3-32B at Q4; a 96GB Mac Studio (~72GB usable) is the floor for gpt-oss-120b, right on the line.
VRAM decides if it runs; bandwidth decides how fast
Fitting is step one. The theoretical generation speed ceiling is roughly memory bandwidth × 0.75 ÷ weight bytes read per token (theoretical estimate — real-world results vary with framework and drivers, ±30%). Some comparisons:
- 8B at Q4: RTX 3060 (360 GB/s) ~55 tok/s, RTX 4090 (1008 GB/s) ~154 tok/s, Mac mini M4 Pro (273 GB/s) ~42 tok/s.
- Qwen3-32B at Q4 has plenty of VRAM headroom on a 48GB Mac mini, but only manages ~10 tok/s. “Runs” and “usable” are different things.
- MoE models only read active parameters per token: Qwen3-30B-A3B theoretically hits ~370 tok/s on an RTX 4090 — like a dense 3B model — at the cost of 21GB of VRAM.
The worst outcome is forcing a model that doesn’t fit: CPU offload caps effective bandwidth at system memory speeds, 60–100 GB/s. A 70B at Q4 drops from ~30 tok/s on dual 4090s to ~1.4 tok/s offloaded — 20× slower. If you’re 1GB over budget, drop one quant level instead of offloading.
Practical: five steps to size any model
- Find the parameter count: it’s on the Hugging Face model page or in the file name (7B, 32B, A3B…). For MoE, note both total and active parameters.
- Compute weights: parameters × the per-1B value from the cheat sheet. For exact per-quant numbers, use our VRAM calculator — it uses the same formulas as this article.
- Estimate KV cache: at 8K context, roughly 1GB for an 8B-class model, 2GB for 30B-class, 3GB for 70B-class; scale up proportionally for longer contexts. If you don’t know the architecture constants, this estimate is good enough for a buying decision.
- Add 1.5GB of runtime overhead.
- Compare against usable VRAM: full nominal for discrete GPUs, ×0.75 for Apple unified memory. Need ≤ 80% of usable is comfortable; anything over 100%, don’t force it.
When it doesn’t fit, the trade-off order is: drop one quant level > shorten context > enable q8 KV cache > add a GPU > CPU offload. Our recommendation logic follows the same order — a tight Q4 always beats a comfortable Q2, because quality loss hurts your experience more than VRAM headroom helps it.
Which quant for which situation
- Q8_0 / Q6_K: when you have VRAM to spare and the task is quality-sensitive (code, math, long-form writing). Worth it for 8B on 12GB or 13B on 24GB.
- Q4_K_M: the default answer. Chat, summarization, translation — imperceptible quality loss, 70% VRAM saved.
- Q3_K_M: the borderline rescue — the realistic way to run 24B on 16GB, or 32B with long context on 24GB. Quality loss becomes noticeable but usable.
- Q2_K: only for “I just want to hear what a 70B sounds like”. Quality loss is obvious; don’t use it for real work.
Common mistakes
“The GGUF file is 5GB, so an 8GB card is enough.” File size ≈ weights. After loading, add KV cache and ~1.5GB of runtime overhead. That 5GB file really needs about 7.5GB.
“MoE models only load active parameters.” Wrong. Which experts fire is decided per token at generation time, so every expert must stay in VRAM. Active parameters affect speed, not capacity.
“The model supports 128K context, so my card can use 128K.” KV cache grows linearly with context — Qwen3-32B’s KV cache goes from 2.1GB at 8K to 34GB at 128K, larger than the weights. Advertised context is an architecture capability, not your VRAM budget.
“Every 24GB card gives me 24GB.” Multi-card VRAM adds up directly, but your desktop OS and browser eat into it; Apple unified memory needs the 0.75 factor on top. Keeping 20% headroom is the baseline rule.
Mental math shortcut
Rule of thumb: GB ≈ 0.6 × billions at Q4_K_M. 8B → ~5GB. 30B → ~18GB. 70B → ~43GB. Add ~2.5GB for KV cache and overhead at 8K context, more for longer contexts.
MoE models (DeepSeek-R1, gpt-oss, Qwen3-A3B) still need total parameters in memory — but they generate at the speed of their much smaller active parameter count. That’s why gpt-oss-120b feels fast despite needing 74GB.
FAQ
How much VRAM does a 7B or 8B model need?
About 4.9GB for weights at Q4_K_M, plus roughly 1.1GB of KV cache at 8K context and 1.5GB of runtime overhead — call it 7.5GB total. Any 12GB card handles it comfortably, and can even stretch to Q8 (~11.1GB, tight).
How much VRAM does a 70B model need?
About 47GB at Q4_K_M including KV cache and overhead — no single consumer GPU fits it. Two 24GB cards (48GB) or one RTX A6000 run it tight; 64GB+ runs it comfortably.
Why does the cheat sheet use Q4_K_M as the default?
Q4_K_M is the community standard: it cuts VRAM by ~70% vs FP16 with a quality loss most users can't detect in chat. Every major runtime (llama.cpp, Ollama, LM Studio) ships it as the default download.
The GGUF file is only 5GB — why does it use 7.5GB of VRAM once loaded?
The file size is essentially just the weights. KV cache and runtime overhead (CUDA context, compute buffers) are allocated in VRAM after loading and don't show up in the file size. Always add roughly 2.5GB on top of the file size when estimating.
Does a MoE model only need its active parameters in VRAM?
No. Every expert must stay resident in VRAM, because which experts fire is decided per token at generation time. The active parameter count determines speed, not capacity — gpt-oss-120b needs ~74GB of VRAM but generates at the speed of its 5.1B active parameters.