GPUFits
The VRAM Cost of Long Context (and How to Right-Size It)
updated 2026-09-10 · verified 2026-09-10
“Supports 128K context” on a model card is an architecture statement, not a usage promise. What actually decides how much context you can open is how much VRAM remains after the weights are loaded — and that remainder is bought back by the KV cache, byte by byte, at a fixed price per token.
This article answers three questions: how VRAM grows as context lengthens, why the “long-context tax” differs by a factor of ten across models, and how long a context your task actually needs. The mechanism itself — why every layer stores K/V, how GQA saves memory, how MLA compresses it — is fully derived in our KV cache guide. Here we settle the cost account.
Linear growth: no kinks, just a unit price
In the KV cache formula, context length is the only variable; everything else is an architecture constant:
KV cache = cost per token (set by architecture) × context length (set by you)
So the growth curve is a straight line through the origin. Llama-3.1-8B, fp16 KV, 131,072 bytes per token:
| Context | KV cache | For scale |
|---|---|---|
| 4K (Ollama default) | 0.5 GB | barely noticeable |
| 8K | 1.1 GB | pocket change |
| 16K | 2.1 GB | worth tracking |
| 32K | 4.3 GB | nearly the Q4 weights (4.9GB) |
| 64K | 8.6 GB | exceeds the weights |
| 128K (native max) | 17.2 GB | 3.5× the weights |
Enable q8 KV cache and every number in the table halves — the slope stays. That’s the entire math: multiply the context by N, multiply the KV bill by N.
Read the table another way: Qwen3-32B costs exactly double per token (256 KB vs 128 KB), so a 32B model at 8K context is more expensive than an 8B model at 32K. “How big is the model” and “how expensive is long context” are two independent bills.
Architecture sets the tax rate: the per-token price list
All models in our dataset, sorted by KV cost per token (fp16, same formula as our calculator):
| Model | Architecture notes | Per token | at 32K | at 128K |
|---|---|---|---|---|
| gpt-oss-20b | 24 layers × 8 KV heads × 64 dim | 48 KB | 1.6 GB | 6.4 GB |
| Muse-Glimmer-30B | 52 layers × 2 KV heads | 52 KB | 1.7 GB | 7.0 GB |
| DeepSeek-R1 | MLA compressed latents | 69 KB | 2.3 GB | 9.2 GB |
| gpt-oss-120b | 36 layers × 8 KV heads × 64 dim | 72 KB | 2.4 GB | 9.7 GB |
| Qwen3-30B-A3B | 48 layers × 4 KV heads | 96 KB | 3.2 GB | 12.9 GB |
| Llama-3.1-8B | 32 layers × 8 KV heads | 128 KB | 4.3 GB | 17.2 GB |
| Qwen3-8B | 36 layers × 8 KV heads | 144 KB | 4.8 GB | 19.3 GB |
| Mistral-Small-3.2-24B | 40 layers × 8 KV heads | 160 KB | 5.4 GB | 21.5 GB |
| Qwen3-32B | 64 layers × 8 KV heads | 256 KB | 8.6 GB | 34.4 GB |
| Llama-3.3-70B | 80 layers × 8 KV heads | 320 KB | 10.7 GB | 42.9 GB |
| Gemma-3-27B | 62 layers × 16 KV heads | 496 KB | 16.6 GB | 66.6 GB |
Three facts worth remembering:
- Parameter count and KV cost are essentially unrelated. The 30B Qwen3-30B-A3B (4 KV heads) is 25% cheaper per token than the 8B Llama; the 671B DeepSeek-R1 uses MLA to land at 69 KB — under a quarter of Llama-3.3-70B. The price is set by three architecture constants: layers × KV heads × head dimension.
- GQA’s grouping ratio is the tax rate. Where most models ship 8 KV heads, Gemma-3 ships 16 and its cost doubles. When choosing a model,
num_key_value_headsdeserves a look alongside parameter count. - Sliding-window and hybrid models come in below the formula. Half of gpt-oss’s layers keep only a fixed window; Gemma-3 runs a 1024-token window on 5 of every 6 layers; Muse-Glimmer is a 3:1 sliding/global hybrid. Window and linear layers don’t grow with context. Our calculator deliberately applies the standard formula to these models anyway (conservative on purpose), so the table overestimates them.
For the full mechanism — the GQA paper, MLA’s 576-dim latent vector, how sliding windows stack with RoPE — see the KV cache guide.
Long context isn’t just expensive — it’s slower
VRAM is the first bill; speed is the second. During generation (decode), every token produced re-reads all the weights plus the entire KV cache from VRAM. The longer the context, the more KV each token must read, and the slower generation runs.
Worked example: RTX 4090 (1008 GB/s bandwidth, 0.75 efficiency factor) running Llama-3.1-8B Q4_K_M (4.9GB of weights). Theoretical estimates, ±30%:
| Context | Bytes read per token (weights + KV) | Theoretical speed |
|---|---|---|
| Short (KV negligible) | 4.9 GB | ~154 tok/s |
| 32K | 9.2 GB | ~82 tok/s |
| 128K | 22.1 GB | ~34 tok/s |
Going from near-zero to 128K context cuts generation speed to about a fifth. Note that our speed tool only counts weight reads, so at very long contexts real tok/s lands below the tool’s estimate — the further off it is, the bigger the KV share.
The intake side (prefill) has a cost too: roughly 2 × parameters × prompt tokens of compute. Feeding a 100K-token prompt to an 8B model takes about 1.6×10¹⁸ FLOPs; on an RTX 4090 at an effective ~40 TFLOPS, that’s roughly 40 seconds (theoretical estimate; real frameworks vary widely). The lived experience of long context: paste a big document, wait for prefill, then watch the output stream at reduced speed.
Size the context to the task
Bigger context isn’t better — sufficient is better. Every extra K you open charges both VRAM tax and speed tax:
| Task | Suggested context | Reasoning |
|---|---|---|
| Chat, writing, translation | 4–8K | dozens of turns won’t fill 8K |
| RAG Q&A | 8–16K | 5–10 chunks × 500–800 tokens + question + answer |
| Single-file / small-project code | 8–16K | one file plus its surroundings |
| Whole-codebase analysis, long-document summary | 32K+ | a 100-page PDF is roughly 60–70K tokens (rule of thumb: ~500 words per page × 1.3 tokens); either open 64K+ or chunk it |
| Agents / multi-turn tool use | provision 2× peak | history only grows; leave headroom |
| Self-hosted API serving | per-session length × concurrency | each session keeps its own KV; see below |
Two practical reminders:
- Check the default first. Ollama’s default context is only 4096 tokens (2048 in older versions) — plenty of people think they’re using a 128K model while running a 4K window. Change it with
/set parameter num_ctx 16384orPARAMETER num_ctx 16384in a Modelfile; llama.cpp takes-c; LM Studio exposes it in the load panel. - Smaller doesn’t hurt quality. The window is memory length, not capability; older content is simply dropped. Deliberately shrinking the window for short tasks saves VRAM and buys speed at zero quality cost.
Compute your context budget
Run the formula backwards and you get how much context your GPU can buy:
usable context = (VRAM − weights − 1.5GB fixed overhead) ÷ KV cost per token
Typical combinations (Q4_K_M weights, fp16 KV):
| GPU | Model | Remaining budget | fp16 usable context | with q8 KV |
|---|---|---|---|---|
| 12GB (RTX 3060) | Llama-3.1-8B (4.9GB) | 5.6 GB | ~41K | ~83K |
| 16GB (4070 Ti Super) | Qwen3-8B (5.0GB) | 9.5 GB | ~62K, but the native cap is 32K → full window, easily | even more headroom |
| 24GB (3090/4090) | Llama-3.1-8B (4.9GB) | 17.6 GB | ~131K → just covers the native 128K (23.6GB total, “tight”) | comfortable |
| 24GB (3090/4090) | Qwen3-32B (20.1GB) | 2.4 GB | ~9K | ~18K; the native 32K needs 25.9GB — beyond one card |
Verdict bands: total ≤ 80% of usable VRAM is comfortable, ≤ 100% is tight. Skip the arithmetic and open the VRAM calculator — pick a model, drag the context slider, and watch the KV bar grow in real time. To find out what your card runs at all, use the GPU compatibility checker.
For self-hosted serving, plug “single-session KV × concurrency” into the same formula: a 24GB card running Qwen3-32B has 2.4GB left, so four concurrent sessions get about 2K of context each. That’s why multi-user setups either add cards (see our multi-GPU guide) or pick a KV-cheap model.
Takeaways
- The KV cache grows strictly linearly with context, at a per-token price fixed by architecture constants — visible before you ever download the model.
- Parameter count doesn’t set the long-context price; KV heads, layer count, and MLA/sliding-window designs do. Same-size models differ by 3×, cross-size by 10×.
- Long context charges two taxes at once: VRAM (linear) and decode speed (slower as the KV share grows — theoretical estimates, ±30%).
- Size context to the task, not to the spec sheet. Chat: 4–8K. RAG: 8–16K. Long documents and whole codebases are where 32K+ earns its keep.
- When VRAM comes up short, in order: q8 KV cache → one lower weight quant → a different architecture or more cards. Weight and KV quantization are independent switches — details in our quantization guide and the KV cache guide.
Do this math before you buy a card, before you pick a model, and before you set num_ctx.
FAQ
How much extra VRAM does each doubling of context cost?
Every model has a fixed KV cost per token. Llama-3.1-8B is 131,072 bytes per token (fp16), so going from 8K to 16K costs an extra 1.1GB, and from 64K to 128K an extra 8.6GB. Perfectly linear — no kinks, no plateaus.
Does setting a smaller context hurt model quality?
No. The context window is memory length, not capability. Content older than the window is simply dropped. For tasks that don't need long memory, deliberately shrinking the window is pure profit: less VRAM, faster generation, zero quality cost.
Does long context slow down generation?
Yes. Every generated token re-reads the entire KV cache from VRAM. On an RTX 4090 running Llama-3.1-8B at Q4: ~154 tok/s ignoring KV, ~82 tok/s at 32K context, ~34 tok/s at 128K (theoretical estimates, ±30%). Quadruple the context, roughly halve the speed.
How much context does RAG actually need?
Size it to your retrieved chunks, not the model's maximum. 5–10 chunks at 500–800 tokens each, plus the question and the answer, lands in 8–16K for most setups. Opening 128K blindly just wastes VRAM and speed on a window you'll never fill.
How is the KV cache computed for multiple concurrent users?
Each concurrent session keeps its own full copy — nothing is shared. Total KV = single-session KV × concurrency. Four parallel 32K sessions of Qwen3-32B cost about 34GB of KV alone (fp16). For self-hosted serving, fix your concurrency target first, then the context length.
Which knob do I turn first when VRAM won't cover the context I need?
Enable q8 KV cache first (halves the footprint with negligible quality loss), then drop one weight quant level, then consider a KV-cheaper architecture or more VRAM. Note that weight quantization and KV quantization are independent switches — Q4 weights don't shrink the KV cache automatically.