Step 3: Choosing a Model and Quant: B, GGUF, and Q4_K_M Explained

Updated 2026-09-11 · Verified 2026-09-11

In the previous step we learned that VRAM is the model’s “parking space.” Now the next question: with so many models in the lot, which one do you pick? Download pages are full of filenames like Llama-3.1-8B-Instruct-Q4_K_M.gguf — intimidating, until you split it into three parts, each explainable in one sentence.

The “B” in 8B: brain size

B stands for billion — it’s how many billions of parameters the model has. Parameters are the numbers inside the model; more of them generally means more learned knowledge. Think of it as rough brain size: 3B ≈ 3 billion, 8B ≈ 8 billion, 70B ≈ 70 billion.

As a beginner, you only need three gut feelings:

• ~3B: the starter tier. Runs on almost any computer, fine for casual chat.
• ~8B: the sweet spot. The best balance of quality and size — most people’s first serious model.
• 70B+: enthusiast tier. Much smarter, much hungrier for VRAM. Admire from a distance for now.

One intuition trap: bigger isn’t automatically better for you — bigger means more VRAM. A 70B you can’t fit might as well not exist.

GGUF: the model, packed into a file

GGUF is the standard file format for local models — think of it as the MP4 of local AI. One file (sometimes a few parts) contains the entire model, ready to run in Ollama or llama.cpp. It comes from the llama.cpp project and is now the de facto standard. You don’t need to know what’s inside; you just need to recognize that files ending in .gguf are what you download.

Q4_K_M vs Q8_0: the model’s “compression quality”

An original model stores every parameter in 16 bits, which takes a lot of space. Quantization re-stores each parameter in fewer bits — like compressing a RAW photo into a JPG: much smaller, slightly lower fidelity. The Q4_K_M and Q8_0 in filenames are just different compression levels:

Quant	File size (8B model)	Plain-English verdict
Q8_0	~8.5 GB	Nearly lossless; indistinguishable from the original
Q6_K	~6.6 GB	Pick this when you have VRAM to spare
Q4_K_M	~4.9 GB	The community default — beginners start here
Q2_K	~3.2 GB	Technically talks; don’t use it for real work

The key takeaway: when in doubt, choose Q4_K_M. It’s about 30% of the original size, and in blind tests most people can’t tell it from Q8 in conversation. Upgrade to Q6_K or Q8_0 only if you have headroom. Below Q4 is a cliff — a Q2 model will confidently make things up.

And the most common beginner trap: file size ≠ VRAM usage. Once running, the model also needs its “conversation memory” (the KV cache, which grows with chat length) plus about 1.5 GB of runtime overhead. Llama-3.1-8B’s Q4_K_M file is 4.9 GB, but actually occupies about 7.5 GB at 8K context (theoretical estimate). Rule of thumb for Q4: VRAM (GB) ≈ 0.6 × the B number, plus ~2.5 GB.

For the full mechanics and measured comparisons, read Quantization Explained: Q4 vs Q8; for the per-billion-parameter math, see How Much VRAM per Billion Parameters?.

Where to download: Hugging Face

Hugging Face (huggingface.co) is the world’s largest model hub — the app store of open models. Free account, free downloads, nearly everything lives there. For GGUF versions, stick to two kinds of sources: the model’s official account, or community quantizers like bartowski who publish high-quality GGUFs. Each model page lists a row of files — grab the one labeled Q4_K_M and you’re done.

To recap: the B picks the size, GGUF is the format, Q4_K_M is the level, Hugging Face is the place. You can now read any download page. One last thing before we run anything — do the math:

Open the calculator below, pick a model you’d like to run, and see what verdict your GPU gets.