Step 5: Too Slow or Won't Fit? A Five-Step Checklist

Updated 2026-09-11 · Verified 2026-09-11

Your model is installed, and the first run either crashes or crawls. Welcome to the most common beginner hurdle. First, identify which symptom you have:

• Won’t fit: it errors out or crashes while loading. The cause is VRAM — the fast memory on your graphics card that the entire model must live in while running.
• Too slow: it runs, but words dribble out one at a time.

Both share the same troubleshooting order, with one principle: turn the cheapest dial first, stop when it works. Here are the five dials, in order. (All speeds below are theoretical estimates from this site’s formulas, ±30% in practice.)

Dial 1: Drop one quantization level

Quantization is model compression: each step down (Q4 → Q3) shrinks the model by roughly 20% at a small quality cost.

When to use it: your total requirement exceeds VRAM by 1–4GB. Example: a 24B model at Q4 needs about 17.5GB — too big for a 16GB card. At Q3 it needs about 14.8GB, fits, and runs fully on-GPU at a theoretical ~42 tok/s — an order of magnitude faster than any “squeeze it in anyway” trick.

In Ollama, switching quants is just a different tag: ollama run qwen3:32b-q3_K_M.

Dial 2: Shorten the context

Context is how much conversation the model can “remember” at once. The memory it consumes (the KV cache) is a separate bill from the model itself: double the length, double the footprint.

When to use it: for everyday chat, writing, and translation, 4–8K is plenty — shrinking the window is free VRAM and free speed, with zero quality loss. In Ollama type /set parameter num_ctx 8192; in llama.cpp add -c 8192.

Watch for the reverse problem too: Ollama defaults to 4096, so if the model “forgets” long documents, you need to raise it.

Dial 3: Enable q8 KV cache

If you genuinely need long context but VRAM is tight, don’t touch the model yet — compress the KV cache itself. q8 KV halves that footprint, with a quality impact most people can’t detect.

When to use it: context above 16K with a modest VRAM shortfall — it’s a free lunch. Example: a 32B model on a 24GB card needs 25.9GB at 16K context and won’t fit; with q8 KV it drops back to 23.7GB and fits, at full speed. Enable it in Ollama with the OLLAMA_KV_CACHE_TYPE=q8_0 environment variable, or in llama.cpp with --cache-type-k q8_0 --cache-type-v q8_0.

For the mechanics, see the deep dive KV Cache Explained.

Dial 4: Partial CPU offload

Still short? llama.cpp can park the leftover layers in system RAM and run them on the CPU (the -ngl flag; Ollama does this automatically when a model doesn’t fit).

Know the cost up front: the slowdown is a cliff, not a slope. Once any layers sit in system RAM, generation moves at memory-bandwidth pace — a 24B model runs at a theoretical ~51 tok/s fully on an RTX 4090, but the offload floor is about 4 tok/s. “Instant” becomes “reading speed.”

When to use it: you’re 1–2GB short and the work isn’t interactive — offline evaluation, or a batch job that runs overnight. Don’t daily-drive a chatbot this way. One big exception: MoE models (next article) read only a slice of their weights per token, so offloaded MoE can still hit ~30 tok/s. Full math in Is CPU Offload Worth It?.

Dial 5: Multiple GPUs

The final weapon: VRAM adds up directly — two 12GB cards give you 24GB. Two used RTX 3060 12GB cards run a 24B Q4 model at a theoretical ~31 tok/s, beating any offload setup.

When to use it: you’re committed to running models that don’t fit in 24GB (70B-class), often enough to justify hardware. Need it once a month? That’s what rented cloud GPUs are for — covered next.

The order, in one sentence

Lower the quant → shorten the context → q8 KV → partial offload → multi-GPU. The first three cost you almost nothing, the fourth trades speed for capacity, the fifth trades money for capacity.

You don’t need to hand-calculate any of this — open the checker below and try every model-and-GPU combination you’re considering:

Further reading: Is CPU Offload Worth It? · KV Cache Explained