LLM VRAM Calculator

How to use this calculator

1. Pick a model.The list covers mainstream open models (Llama, Qwen, Gemma, DeepSeek and others). Architecture parameters — layer count, KV heads, head dimension — come from each model's official config.json on Hugging Face.
2. Pick a quantization level and context length.Quantization levels are ordered by bits per weight (bpw), with bpw values reverse-engineered from real GGUF file sizes. Context length directly sets the KV cache size — don't max it out by default.
3. Pick a GPU and read the verdict.Four tiers: comfortable (requirement ≤ 80% of usable VRAM), tight (≤ 100%), needs multi-GPU (≤ 120%), infeasible (> 120%). For unified-memory machines like Macs, usable VRAM is 75% of the nominal figure.

How the VRAM requirement is calculated

Total = weights + KV cache + system overhead. Three independent terms:

• Weights scale linearly with quantization: 8.51 bpw for Q8_0, 4.9 for Q4_K_M, 3.17 for Q2_K. MoE models count total parameters — every expert must be loaded, not just the ones activated per token.
• KV cache: the leading 2 is for the K and V tensors. It grows strictly linearly with context length — double the context, double the KV. Quantizing KV from fp16 to q8 (1 byte) halves it. MLA architectures like DeepSeek-R1 use a different formula — layers × (kvLoraRank + qkRopeHeadDim) × bytes × context — and need far less KV than GQA models of similar size.
• Overhead is a fixed 1.5GB reserved for the CUDA context and inference runtime, independent of model and context.

Worked example: Llama 3.1 8B, Q4_K_M, 8K context

What to give up first when it doesn't fit

1. Drop one quantization level.Q4_K_M to Q3_K_M cuts weights by about 18% (4.9 → 4.0 bpw) with usually acceptable quality loss. Q2_K degrades noticeably — last resort only.
2. Shorten the context.KV is linear in context: going from 32K to 8K removes three quarters of it. Most chat and RAG workloads don't need the full window.
3. Quantize KV to q8.Halves the KV cache, and typically hurts quality less than dropping weights another level.
4. Consider multi-GPU.Usable VRAM of identical cards adds directly with no discount, but inference speed takes a 0.85 bandwidth penalty (theoretical estimate ±30%), and motherboard, PSU and cooling costs double.
5. CPU offload as the last resort.Once layers spill into system memory, effective bandwidth falls to 60–100 GB/s and speed falls off a cliff (theoretical estimate ±30%). Fine for offline batch jobs, not for interactive use.

FAQ

How much VRAM does a 70B model need?

About 47GB at Q4_K_M with 8K context — beyond any single consumer GPU. Two RTX 4090s (48GB) run it tight; 2×A6000 or 4×4090 run it comfortably.

Does longer context use more VRAM?

Yes. KV cache grows linearly with context length. For Qwen3-32B at Q4_K_M, going from 8K to 32K context adds about 6.5GB.

What is the system overhead term?

A fixed 1.5GB allowance for CUDA context and runtime memory, applied on top of weights and KV cache.

How much VRAM does Q4_K_M save over Q8_0?

Weights scale with bits per weight: Q4_K_M is 4.9 bpw versus 8.51 bpw for Q8_0, so weights shrink by about 42%. For Llama 3.1 8B that is 8.5GB down to 4.9GB. KV cache is unaffected by the weight quantization level.

Further reading

• How Much VRAM per Billion Parameters?2026 cheat sheet: look up the number by size and quant
• KV Cache ExplainedWhy long contexts eat your VRAM
• Quantization Explained: Q4 vs Q8The VRAM, quality and speed trade-off
• The VRAM Cost of Long ContextLinear growth, architecture differences, right-sizing