CPU Offload: Is Partial GPU Offloading Worth It?

updated 2026-09-10 · verified 2026-08-04

Your GPU is 2GB short of fitting the model. llama.cpp’s -ngl flag promises a solution: put most layers on the GPU, the rest on CPU. Here’s what that actually costs — the mechanism, the numbers, and the decision boundaries. All speeds below use our calculator’s theoretical model (bandwidth × 0.75 efficiency ÷ per-token weight size); real-world results vary with framework, driver, and CPU, so treat them as ±30% estimates.

How partial offload actually works

llama-cli -ngl 20 model.gguf puts 20 layers on the GPU; the rest run on the CPU in system RAM. The key point: every token still passes sequentially through all layers. Activations cross PCIe between GPU and CPU layers and back — but activations are only a few MB per token, so the bus transfer is not the problem.

The real cost is elsewhere: for every generated token, the CPU layers must re-read their share of the weights from system RAM. Decode is purely bandwidth-bound, so every token waits for the slowest stage of the pipeline: your system memory.

Two practical details:

• -ngl means “number of layers on the GPU.” Setting it to a value beyond the model’s layer count (e.g. 99) means “offload as much as fits” — the remainder automatically falls back to CPU. That’s usually the least fiddly way to run a partial offload.
• Prompt processing (prefill) suffers far less than decode. Prefill is compute-bound and reads weights in large batches, which amortizes the bandwidth pressure; token-by-token decode is where you hit the memory wall head-on. With partial offload you’ll notice the prompt gets read fine, then output crawls.

The memory wall, in numbers

Path	Bandwidth	Share of each token’s time
GPU VRAM (RTX 4090)	1,008 GB/s	fast
PCIe 4.0 x16	~32 GB/s (carries only MB-scale activations)	minor
System RAM (DDR5 dual-channel)	60–100 GB/s	the bottleneck

That system-RAM row deserves unpacking. Dual-channel DDR5-6000 has a theoretical peak of 96 GB/s (6,000 MT/s × 8 bytes × 2 channels); dual-channel DDR4-3200 tops out at 51 GB/s before efficiency losses. Real mainstream desktops land in the 60–100 GB/s range — which is why our calculator uses an 80 GB/s midpoint. Your memory configuration directly sets the offload speed ceiling: on the same machine, single-channel vs dual-channel can differ by 2×.

Meanwhile, a 4090 reads weights from VRAM roughly 12× faster than your CPU reads them from RAM.

Our speed model, and its caveat

Offloaded speed = min(GPU bandwidth, ~80 GB/s) × 0.75 ÷ per-token weight size

Note this is a conservative floor: it assumes that if any layer lives on the CPU, the whole token moves at system-memory pace. Real partial offload lands somewhere between full-GPU speed and this floor, closer to full speed when you offload only a few layers. For the decision “should I offload at all,” a conservative model is the right one.

Plugging in numbers (24B dense model, Q4_K_M, 14.7GB of weights read per token):

Setup	Est. speed	Feel
Fully on RTX 4090 (1,008 GB/s)	~51 tok/s	instant
Partially offloaded (floor, 80 GB/s)	~4 tok/s	reading speed
Fully on CPU (DDR5 dual-channel)	~4 tok/s	painful

Going from 51 to 4 isn’t “a bit slower” — it’s a cliff of a full order of magnitude.

Three worked examples from our data

All numbers below come from our models.json / gpus.json / quant-presets.json datasets through the same formulas — you can reproduce them in the VRAM calculator.

Case 1: a 24B model on a 16GB card (RTX 4070 Ti Super)

Mistral Small 3.2 24B at Q4_K_M: 14.7GB weights + 1.3GB KV cache (8k context, fp16) + 1.5GB runtime overhead = 17.5GB — doesn’t fit 16GB.

• Option A, offload ~2GB of layers: floor speed ~4 tok/s. It runs, but chat feels like reading speed.
• Option B, drop to Q3_K_M: weights shrink to 12.0GB, total need 14.8GB — a tight fit that works, at ~42 tok/s fully on the 4070 Ti Super (672 GB/s).

No contest: one quant level down buys 10× the speed, with a much smaller quality cost than the speed cost of offloading.

Case 2: Qwen3-32B on 24GB (RTX 4090), wanting 16k context

Q4_K_M weights are 20.1GB. At 8k context the total is 23.7GB — a tight fit, running fully on-GPU at ~37 tok/s. But stretch the context to 16k and the KV cache grows from 2.1GB to 4.3GB, pushing the total to 25.9GB. It no longer fits.

• Option A, offload: ~3 tok/s.
• Option B, switch the KV cache to q8: the 16k KV drops back to 2.1GB, the total returns to 23.7GB, and you keep 37 tok/s with negligible quality impact.

This is the most common offload misdiagnosis: the shortfall is in the KV cache, not the weights, and KV quantization is a free lunch that never touches the speed path. Details in our KV cache guide.

Case 3: MoE models are a different species

Qwen3-30B-A3B has 30.5B total parameters but activates only 3.3B per token — each token reads ~2.0GB of weights. Even at the 80 GB/s offload floor, that’s ~30 tok/s, still a usable chat speed. Same story for gpt-oss-20b (3.6B active): floor of ~27 tok/s.

Contrast: a dense Llama-3.3-70B reads 43GB per token, for an offload floor of ~1.4 tok/s — slower than you can read. MoE is the great exception to “offload is unusable,” because the memory wall scales with bytes actually read per token, not total parameter count.

When offload IS worth it

• Evaluation and one-off tasks: “can this 32B model do X at all?” — yes, run it overnight offloaded and read the results tomorrow.
• Batch pipelines: summarizing 500 documents at 800 output tokens each is 400k tokens; at 4 tok/s a single stream takes ~28 hours, but four parallel streams finish overnight. Throughput jobs don’t care about per-stream tok/s.
• Everyday partial offload of MoE models: as above, the floor is still ~30 tok/s.
• Capacity emergencies: the model you need is 2GB over, and the alternative is a cloud bill.

When it’s NOT

• Interactive chat — reading-speed output gets old in minutes.
• Agent loops — slowdowns compound. A 20-step agent task emitting 500 tokens per step is 10k tokens: a fully-resident 32B on a 4090 (~37 tok/s) finishes in about four and a half minutes; offloaded (~3 tok/s) it takes nearly an hour. Every added step widens the gap.
• When a smaller fully-resident model exists — and in 2026, one almost always does. In practice, a 13B/14B fully on-GPU will out-produce a 32B gasping through DDR5, because you’ll actually iterate with it.

Common misconceptions

1. “PCIe 5.0 will save offload.” It won’t. The bus isn’t the bottleneck — activations are only a few MB per token. The bottleneck is the CPU re-reading GB of weights from RAM every token, and that path never touches PCIe.
2. “Offloading 20% of layers costs 20% speed.” The decay isn’t linear, it’s a cliff: once the slowest stage becomes system memory, the whole pipeline runs at system-memory pace.
3. “RAM speed doesn’t matter.” Under offload, memory bandwidth is generation speed. Moving from dual-channel DDR4 to DDR5 nearly doubles the offload ceiling; populating both channels instead of one has an immediate effect.
4. “Same story on a Mac.” Apple Silicon is unified memory — CPU and GPU read the same pool, so there’s no offload penalty. A 48GB M4 Pro (~36GB usable, 273 GB/s) runs a 24B Q4 fully resident at ~14 tok/s. Not fast, but that’s the full speed, not a penalized one.
5. “Prefill will be just as slow.” Prefill is compute-bound and suffers much less; decode is what crawls. That’s why offloaded batch summarization “feels fine” while offloaded chat is misery.

The better escape hatches (in priority order)

1. q8 KV cache + shorter context: touches neither the weights nor the speed path, and frees real GBs (2.2GB in Case 2).
2. Drop a quant level (Q4→Q3): in Case 1 that’s the difference between 4 and 42 tok/s. The quality loss is usually smaller than the offload speed loss. See our quantization guide.
3. A second used GPU: two used RTX 3060 12GB cards (~$260 each in our used-price data) give you 24GB of layer-split capacity with an effective bandwidth of 2 × 360 × 0.85 ≈ 612 GB/s — a 24B Q4 runs at ~31 tok/s, beating any offload scheme. See our multi-GPU guide.
4. Reconsider unified memory: if the GPU budget is approaching Mac mini territory, price out a 48GB unified-memory box first.

If you must offload: the practical checklist

1. Confirm the actual shortfall in the VRAM calculator first — you’ll often find that q8 KV or one quant level down closes the gap, and no offload is needed at all.
2. If offload is truly necessary, set -ngl 99 and let llama.cpp fill VRAM to the brim; the startup log prints where each layer landed (GPU/CPU) — check it.
3. Populate both memory channels and enable XMP/EXPO in the BIOS — the offload ceiling is set by memory bandwidth.
4. Once running, look at actual tok/s. For anything interactive below reading speed (~8–10 tok/s), go back and take escape hatches 1–3 seriously.

Offload is a real feature that solves a real class of problems — but it’s a ladder, not a floor: use it to climb over the “2GB short” wall, not as a place to stand long-term.

FAQ

Sources

• llama.cpp documentation (-ngl / --n-gpu-layers)
• NVIDIA GeForce RTX 4090 specs (1,008 GB/s memory bandwidth)