Run GGUF Models Locally: llama.cpp and Ollama Walkthrough

updated 2026-09-10 · verified 2026-09-10

Download a GGUF file, run it, check the speed — three steps, in theory. What actually trips people up is the part nobody explains: which of a dozen quant filenames to grab, what to set -ngl to, and why the model “runs” but types like it’s 1995. This guide walks both paths — llama.cpp and Ollama — end to end, with the math behind every number and the verification steps at the end.

Before you download anything: do the VRAM math

The wrong order is “download first, find out it doesn’t fit later.” A 70B Q4 file is 43GB — discovering the shortfall halfway through costs you an hour of bandwidth. The right first step is arithmetic. Our VRAM calculator uses this formula:

Total VRAM = params (billions) × bits-per-weight ÷ 8    ← weights, ≈ GGUF file size
           + KV cache (grows linearly with context)
           + ~1.5GB runtime overhead

The bits-per-weight values come from real GGUF file sizes: Q4_K_M ≈ 4.9, Q6_K ≈ 6.6, Q8_0 ≈ 8.5. Worked examples at 8k context with fp16 KV cache:

Model × quant	Weights	KV @8k	Total	Fits on
Llama-3.1-8B Q4_K_M	4.9GB	1.1GB	7.5GB	RTX 3060 12GB, comfortable
Llama-3.1-8B Q8_0	8.5GB	1.1GB	11.1GB	12GB tight, 16GB comfortable
Qwen3-32B Q4_K_M	20.1GB	2.1GB	23.7GB	Single 24GB card (3090/4090), tight
Llama-3.3-70B Q4_K_M	43.2GB	2.7GB	47.4GB	2× 24GB or a 48GB card
Qwen3-30B-A3B (MoE) Q4_K_M	18.7GB	0.8GB	21.0GB	Single 24GB card

The verdict bands: total ≤ 80% of usable VRAM is comfortable, ≤100% is tight. To check your own card against every quant before downloading, use the GPU checker — it runs the same formulas as this article.

Picking a quant level

A bartowski-style Hugging Face repo lists a row of files for the same model: Q8_0, Q6_K, Q5_K_M, Q4_K_M, Q3_K_M… Three rules cover it:

• Default to Q4_K_M. Roughly 30% of FP16 size, and in chat and writing tasks most people can’t tell it from Q8 in a blind test. It’s the community default for a reason.
• Step up if you have headroom: Q6_K, then Q8_0. Math, long reasoning chains, and agent tool-calling are sensitive to weight noise and deserve the upgrade.
• Q4→Q3 is a cliff. Q3_K_M starts dropping facts and contradicting itself; Q2 is little more than “produces tokens.” If Q4 doesn’t fit, the right move is usually a smaller model, not a more crushed big one. Full comparison in our quantization guide.

Path A: llama.cpp

llama.cpp is the engine underneath everything — Ollama and LM Studio both run on it. Grab a prebuilt release from GitHub (or build from source) and you get llama-cli, llama-server, and friends.

Downloading the GGUF

Two options: download the .gguf file for your chosen quant from the Hugging Face page, or let llama.cpp pull it directly with the built-in -hf flag:

llama-cli -hf bartowski/Meta-Llama-3.1-8B-Instruct-GGUF:Q4_K_M

An 8B Q4_K_M is about 4.9GB — a few minutes on decent broadband. Note that larger models ship as multi-part GGUF splits; you need all of them in one directory.

The three flags that matter

-ngl (—n-gpu-layers): layers on the GPU. The single most important llama.cpp flag, bar none. -ngl 32 puts all 32 layers of Llama-3.1-8B on the card; -ngl 99 means “offload as much as fits,” and whatever doesn’t fit falls back to the CPU in system RAM — that’s partial offload, and it’s technically possible but drops generation to system-memory bandwidth: an 8B Q4 that does ~154 tok/s fully on a 4090 falls to single digits at the offload floor (theoretical estimates, ±30%). Fine for overnight batch jobs, wrong for chat. Mechanism and decision boundaries in our CPU offload guide.

--ctx-size: context length. This sets your KV cache size directly. Llama-3.1-8B at 8k context needs ~1.1GB of KV; at 32k that grows to 4.3GB, moving the total from 7.5GB to 10.7GB — a comfortable setup becomes a ceiling-scraper. When VRAM is tight, decide how much context you actually need instead of maxing it by default.

--cache-type-k q8_0 --cache-type-v q8_0: KV cache quantization. Halves KV cache footprint with negligible quality loss. When context is what’s squeezing you, this is the first escape hatch — ahead of dropping the weight quant. Details in our KV cache guide.

A typical complete command:

llama-cli -m Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf -ngl 99 --ctx-size 8192

Want an API instead of an interactive prompt? Swap llama-cli for llama-server and you get a local OpenAI-compatible endpoint.

The startup log is your acceptance report

llama.cpp prints where every layer landed (GPU/CPU) and the actual VRAM allocated for weights and KV cache. Make a habit of reading it: the allocations should match your formula estimate to within the same ballpark. If they don’t, check whether --ctx-size differs from what you assumed.

Path B: Ollama

Ollama = llama.cpp + a model registry + service management + auto-updates. After install, one command:

ollama run llama3.1:8b

It pulls the matching GGUF from the official library (typically the Q4_K_M tier) and drops you into a chat session. The model stays loaded for a few minutes after your last message, then unloads automatically.

Importing your own GGUF

To run a specific quant from Hugging Face, write a Modelfile:

FROM ./Meta-Llama-3.1-8B-Instruct-Q4_K_M.gguf
PARAMETER num_ctx 8192

Then ollama create my-llama -f Modelfile to register it, and ollama run my-llama to chat.

The two settings that mirror -ngl and —ctx-size

• num_ctx (context): Ollama’s default is 4096 per the official Modelfile docs — far below the context window the model advertises. This one default is the shared answer to two of the most common complaints: “my measured VRAM is way below my 8k estimate” and “why does the model forget the start of my long document.” Set it interactively with /set parameter num_ctx 8192, in the Modelfile, or via the OLLAMA_CONTEXT_LENGTH environment variable on newer versions.
• num_gpu (GPU layers): the equivalent of llama.cpp’s -ngl, set in the Modelfile as PARAMETER num_gpu 99. You rarely touch it — Ollama tries to fit everything on the GPU by default and silently partial-offloads when it can’t. “Silently” is exactly why you need to check.

ollama ps: your deployment at a glance

ollama ps

The SIZE column is the model’s current total memory footprint; PROCESSOR is the column that matters. 100% GPU means fully resident; a split like 48%/52% CPU/GPU means some layers fell into RAM — it runs, but expect single-digit tok/s. Pair this with nvidia-smi and you have the first scene of any “why is it slow” investigation.

After it runs: verify VRAM and speed

A running deployment is not necessarily a correct one. Two acceptance checks:

Verify actual VRAM usage. On NVIDIA cards, nvidia-smi shows per-process usage; in Ollama, ollama ps; in llama.cpp, the startup log’s allocation lines. If reality is far from your formula estimate, debug in this order: ① the runtime’s default context is smaller than you assumed (Ollama: 4096); ② the KV cache got quantized to q8; ③ some layers are in system RAM.

Verify tok/s. llama.cpp prints eval time and tokens-per-second after each generation; Ollama prints eval rate at the end when you pass --verbose. Compare against the theoretical estimate — our speed model:

Theoretical tok/s ≈ VRAM bandwidth × 0.75 ÷ weight GB read per token

Reference anchors (theoretical estimates, ±30%; real results depend on framework, driver, and CPU):

Hardware	Model × quant	Est. speed
RTX 3060 (360 GB/s)	8B Q4_K_M	~55 tok/s
RTX 4090 (1,008 GB/s)	8B Q4_K_M	~154 tok/s
RTX 3090 (936 GB/s)	32B Q4_K_M	~35 tok/s
2× RTX 3090	70B Q4_K_M	~28 tok/s
RTX 4090	Qwen3-30B-A3B (MoE, reads 2.0GB/token)	~374 tok/s

Anything within 0.7–1.3× of the estimate is normal; a full order of magnitude below means you almost certainly partial-offloaded — go back to ollama ps or the startup log. Rule of thumb for feel: interactive chat below ~10 tok/s gets annoying fast, 20+ is smooth. If you’re under that, consider a lower quant, a shorter context, or a smaller model.

Common misconceptions

• “Just download and run, flags are optional.” Defaults are mined: Ollama’s num_ctx of 4096 silently truncates long-document work, and when VRAM runs short Ollama partial-offloads without telling you — the symptom is “works, but painfully slow.”
• “Half the layers on GPU means half the speed.” Not linear decay — a cliff. Once the slowest stage becomes system memory (60–100 GB/s), the whole pipeline moves at memory pace: from triple digits to single digits.
• “VRAM usage should roughly equal the file size.” The file ≈ the weights; add KV cache and ~1.5GB of overhead. An 8B model at 128k context needs ~17GB of KV alone — over three times its Q4 weights.
• “Ollama is much slower than llama.cpp.” Same engine underneath. Same model, same quant, same settings → same ballpark. Differences come from different default settings, not the runtime.
• “Low tok/s means the card is weak.” First rule out partial offload, maxed-out context, or a prefill-heavy workload. Pure decode speed is set by VRAM bandwidth, and on the same card Q4 is always faster than Q8.

The one-line version

Use the VRAM calculator to find the highest quant that fits, download Q4_K_M to start; on the llama.cpp path remember -ngl 99 and --ctx-size, on the Ollama path remember num_ctx and ollama ps; then validate against the startup log and tok/s estimates — if reality is an order of magnitude off, nine times out of ten you partial-offloaded.

FAQ

Sources

• llama.cpp (GitHub — -ngl / --ctx-size parameter docs)
• Ollama (GitHub main repository)
• Ollama Modelfile documentation (num_ctx, num_gpu, GGUF import)
• bartowski/Meta-Llama-3.1-8B-Instruct-GGUF (Hugging Face file listing)