Quantization Explained: Q4 vs Q8 and What You Actually Lose

updated 2026-09-10 · verified 2026-08-04

Quantization is how a 70B model that “needs” 140GB runs on a $700 used graphics card. Here’s the mechanism, the numbers, and exactly what you’re trading.

What quantization does

A model’s weights are stored as floating-point numbers: FP16 uses 16 bits per weight. Quantization maps a group of float weights onto a smaller set of integer values, stores one scaling factor per group (block), and reconstructs approximate floats at inference time.

Two design ideas do most of the work:

• Block-wise scaling. Instead of one scale for the whole model, every 32 weights get their own block with its own factor. Errors stay local, and total distortion is far smaller than a single global scale.
• K-quant super-blocks. The “K” in Q4_K / Q6_K refers to llama.cpp’s K-quants. Eight blocks form a super-block, and even the scaling factors get quantized, which cuts overhead further. On top of that, precision is allocated by importance — sensitive spots like attention layers, embeddings, and the output head keep more bits.

So “Q4” is only a nominal tier: Q4_K_M measures about 4.9 bits per weight in practice, not 4.0. The 4 is the base bit-width; the rest is factor overhead plus deliberately protected high-precision tensors.

The measured numbers (Llama-3.1-8B GGUF)

These sizes come from real file byte counts in the bartowski repo (8.03B parameters, verified 2026-08-04), not theoretical bit rates:

Quant	Measured bits/weight	File size	% of FP16	Quality guidance
FP16	16.0	~16.1 GB	100%	reference
Q8_0	8.5	8.5 GB	53%	indistinguishable for almost all uses
Q6_K	6.6	6.6 GB	41%	the “if you have VRAM to spare” pick
Q5_K_M	5.7	5.7 GB	35%	excellent, minor losses on hard tasks
Q4_K_M	4.9	4.9 GB	30%	the default — best size/quality trade
Q3_K_M	4.0	4.0 GB	25%	noticeable degradation; emergency only
Q2_K	3.2	3.2 GB	20%	for curiosity, not for work

Two details worth remembering:

• Measured bits per weight run slightly above llama.cpp’s theoretical rates (Q4_K_M is ~4.83 in theory) because embedding tables and metadata compress poorly. The bigger the model, the smaller the gap — a 32B at Q4_K_M measures 19.8GB (Qwen3-32B), right on the formula.
• Q8 down to Q4 is a smooth slope; Q4 to Q3 is a cliff. At Q3_K_M the model starts dropping facts and contradicting itself; Q2_K is barely more than “produces words.”

Decoding the filenames

A GGUF download page lists a wall of files. Three rules decode all of them:

• The 0 in Q8_0 / Q4_0: legacy uniform quantization, every tensor treated equally. Superseded by K-quants; only kept for historical compatibility.
• _K (Q4_K, Q5_K, Q6_K): super-block mixed precision — the current default family.
• The _S / _M / _L suffix: small/medium/large — how much extra precision the sensitive tensors get. Q4_K_M’s “M” means key tensors are stored closer to Q6_K level. That’s why the community treats Q4_K_M, not Q4_K_S, as the default. Later suffix, slightly bigger file, slightly better quality.

Practical takeaway: facing a wall of options, grab Q4_K_M; one tier better is Q5_K_M or Q6_K; stay away from the 0-series and IQ2/Q2 files unless you know exactly why you want them.

File size is only the first line item: the full VRAM equation

File size ≈ weight VRAM, but a running model also needs a KV cache and runtime overhead. The formula our calculator uses:

total VRAM = params (billions) × bits per weight ÷ 8   ← weights
           + KV cache (grows linearly with context)
           + ~1.5 GB runtime overhead

Three complete worked examples at 8K context with fp16 KV cache (verdict rule: need ≤ 80% of usable VRAM is comfortable, ≤ 100% is tight):

Setup	Weights	KV cache	Overhead	Total	Where it lands
Llama-3.1-8B Q6_K	6.6 GB	1.1 GB	1.5 GB	9.2 GB	comfortable on a 12GB card (RTX 3060)
Llama-3.1-8B Q8_0	8.5 GB	1.1 GB	1.5 GB	11.1 GB	tight on 12GB, comfortable on 16GB
Qwen3-32B Q4_K_M	20.1 GB	2.1 GB	1.5 GB	23.7 GB	tight on a 24GB card (3090/4090)
Qwen3-32B Q8_0	34.9 GB	2.1 GB	1.5 GB	38.5 GB	no single consumer card; needs ~48GB
Llama-3.3-70B Q4_K_M	43.2 GB	2.7 GB	1.5 GB	47.4 GB	tight on 2×24GB or one 48GB card
Llama-3.3-70B Q8_0	75.1 GB	2.7 GB	1.5 GB	79.3 GB	80GB datacenter card

How to read this table: the quant tier mostly decides which card can run the model at all — not how different it feels once running. An 8B on a 12GB card gets Q6_K for free (Q8 would hit the ceiling); a 32B on 24GB has exactly one comfortable answer, Q4_K_M; moving a 70B from Q4 to Q8 jumps the hardware requirement from “two used 3090s” to “one A100” and multiplies the budget. This is the most common real decision quantization forces on you.

MoE models (Qwen3-30B-A3B, gpt-oss, DeepSeek) size by total parameters: 30B-A3B at Q4_K_M is about 18.7GB of weights. Every expert must sit in VRAM — there is no “only load the active experts” trick.

Speed: quantization is free acceleration

Lower precision isn’t just smaller — it’s faster. Token generation is bottlenecked by memory bandwidth: every emitted token requires reading the weights end to end. Our speed formula:

theoretical tok/s = bandwidth × 0.75 ÷ weight GB read per token

Plugging in real numbers (theoretical estimates — frameworks, drivers, and CPU overhead move reality by roughly ±30%):

Hardware	Model and quant	Bytes per token	Theoretical speed
RTX 4090 (1008 GB/s)	8B Q8_0	8.5 GB	~89 tok/s
RTX 4090	8B Q4_K_M	4.9 GB	~154 tok/s
RTX 3060 (360 GB/s)	8B Q4_K_M	4.9 GB	~55 tok/s
RTX 4090	70B Q4_K_M	43.2 GB	~17 tok/s
2× RTX 3090	70B Q4_K_M	43.2 GB	~28 tok/s
RTX 4090	Qwen3-30B-A3B Q4_K_M	2.0 GB (active params only)	~374 tok/s

Three takeaways:

• In theory the Q4-to-Q8 speed ratio equals the byte ratio (4.9 vs 8.5, about 1.7×). Real gains come in below that because decode isn’t purely bandwidth-bound — dequantization and sampling cost compute — but the direction is certain: on the same card, Q4 is faster than Q8, not “about the same.”
• MoE models are the bandwidth formula’s biggest winners: VRAM must hold all 30.5B parameters, but each token only reads the 3.3B active ones — so Qwen3-30B-A3B runs faster on a 4090 than a dense 8B.
• Prefill (reading your prompt) is compute-bound, so quantization doesn’t help it. If your workload is mostly long-document Q&A, the felt speed gap is smaller than the table; if it’s long-form output (code, essays), the table is what you feel.

Picking a tier by workload

Workload	Recommended tier	Why
Chat, writing, translation	Q4_K_M	indistinguishable in blind tests; save VRAM for context
Coding assistant	Q4_K_M, Q6_K if there’s headroom	small errors compound across long dependency chains
Math, complex reasoning	Q6_K or Q8_0	multi-step computation is most sensitive to weight noise
Agents / tool use	Q6_K and up	structured output (JSON calls) can’t tolerate drift
Long-document RAG	Q4_K_M + q8 KV	VRAM spent on context beats VRAM spent on weights
Research reproduction, fine-tune baselines	FP16 / BF16	don’t quantize when you need to match published numbers

The general heuristic: secure “comfortable” first, then upgrade the tier. A Q8 pinned at 100% VRAM that can only open 2K of context is a worse experience than a Q6 using 70% of VRAM with 32K of context. Headroom is itself quality — it buys context length.

A practical workflow

1. Do the math: weights = parameters × bits ÷ 8 (the table above has the coefficients), plus KV cache (~1–3GB at 8K), plus 1.5GB overhead. Or use our VRAM calculator — it runs the same formulas as this article.
2. Pick the highest tier that fits: comfortable (≤80%) preferred, tight (≤100%) acceptable. That’s exactly what our recommendation engine does — a tight Q4 beats a comfortable Q2.
3. Download: from the bartowski repos or the model’s official GGUF repo on Hugging Face, grabbing the file whose name ends in your tier. An 8B Q4_K_M is about 4.9GB — a few minutes on ordinary broadband.
4. Verify actual usage: check the VRAM your runtime reports after loading. If it disagrees with the formula, look at the KV cache first — many runtimes default to a 2048–4096 context, so real usage lands well below an 8K-based estimate.
5. When context-starved, enable q8 KV cache first (halves KV usage with minimal quality loss) before lowering the weight tier. Doing it backwards is the most common beginner mistake.

Common misconceptions

• “Q4_K_M stores every weight in 4 bits.” No. It’s a mixed-precision scheme measuring 4.9 bits on average; sensitive tensors get more. That’s also why it beats the old Q4_0 by a clear margin.
• “File size equals VRAM usage.” The file ≈ weight VRAM, but you still add the KV cache and ~1.5GB of runtime overhead. At long context the KV cache can exceed the weights — Qwen3-32B at 32K context needs ~8.6GB of KV alone.
• “Halving the bits doubles the speed.” Only the token-generation phase scales roughly inversely with bytes, and real gains are diluted by compute overhead; prefill barely benefits at all.
• “Ultra-low quants (IQ2, Q2) are a free lunch.” Below Q4, perplexity and quality both degrade and the model starts inventing facts. On a tight budget the right move is a smaller model, not a big model crushed to Q2.
• “Q8 is always worth it.” Only when Q8 doesn’t squeeze your context. Cutting context from 32K to 4K to make room for Q8 is a net loss on most tasks.

The KV cache wildcard

Quantizing weights is only half the story. The KV cache (the model’s working memory for your conversation) defaults to fp16 and can exceed the weights at long contexts: Llama-3.1-8B at 128K context needs ~17GB of KV — over three times its Q4 weights. llama.cpp, Ollama, and LM Studio all support q8 KV cache, halving that cost with minimal quality impact. If you’re context-starved, enabling q8 KV is usually a better move than dropping weight precision — the formulas and full comparison are in our KV cache guide.

The one-line summary

Start at Q4_K_M, upgrade to Q6_K when VRAM is free, consider Q8 for math and agent workloads, always protect context length first, and only touch sub-Q4 when nothing else fits. To run the numbers for your own GPU and model, open the VRAM calculator — every figure in this article comes from the same formulas behind it.

FAQ

Sources

• Measured GGUF file sizes (bartowski/Meta-Llama-3.1-8B-Instruct-GGUF, Hugging Face)
• llama.cpp quantization documentation