KV Cache Explained: Why Long Contexts Eat Your VRAM

updated 2026-09-10 · verified 2026-08-04

Every “this model supports 128K context” claim comes with an asterisk measured in gigabytes. That asterisk is the KV cache.

Most people budget VRAM for the weights alone — right up until they drop a hundred-page PDF into the chat and watch memory explode. This article settles the account: what the KV cache is, how to compute it exactly, and the three levers you can pull when your GPU comes up short.

What it is

Transformers generate text one token at a time. Each new token must “attend” to every token before it — mathematically, its query vector (Q) is compared against the key (K) and value (V) vectors of the entire prefix. If the model recomputed that prefix for every single token, compute would grow quadratically with length, and a long chat would slow to a crawl within a few dozen turns.

The KV cache eliminates that recomputation. While reading your prompt (the prefill phase), the model stores the K and V vectors of every token, in every layer. From then on, each generated token (the decode phase) computes only its own K/V, appends them to the cache, and reads the whole cache back to attend. Generation gets fast — and a long conversation or a big document starts eating VRAM independently of the model’s weights.

Two direct consequences:

• Prefill is compute-bound; decode is bandwidth-bound. Reading a prompt is one big parallel matrix multiplication. Generation is serial: every step re-reads all the weights plus the entire KV cache from VRAM. The longer the context, the more KV each token must read, and the slower generation gets — our speed estimates only count weight reads, so at very long contexts real tok/s lands somewhat below the estimate.
• KV and weights are two separate bills. Weights are fixed once loaded. The KV cache grows linearly with your actual context length, up to the model’s window — or until VRAM runs out.

The formula

KV cache (bytes) = 2 × nLayers × nKvHeads × headDim × bytesPerValue × contextTokens

Term by term:

• 2: one copy each for K and V.
• nLayers (num_hidden_layers): every layer keeps its own K/V set. No exceptions.
• nKvHeads (num_key_value_heads): KV heads, not query heads — under GQA the two differ, and that’s where the savings come from. More below.
• headDim: the vector length per attention head, typically hidden_size ÷ query heads; 128 in most modern models.
• bytesPerValue: the KV dtype — fp16 = 2 bytes (the default), q8 = 1 byte.
• contextTokens: the length you actually use, not the advertised maximum. Rough conversion: one Chinese character ≈ 1 token, one English word ≈ 1.3 tokens.

Worked example, Llama-3.1-8B: 2 × 32 layers × 8 KV heads × 128 dim × 2 bytes = 131,072 bytes per token (128 KB). At 128K tokens: 131,072 × 131,072 ≈ 17.2 GB. For scale, its Q4_K_M weights are just 4.9GB (8.03B × 4.9 bits ÷ 8) — the context costs 3.5× the model itself.

Architecture constants come straight from each model’s config.json:

Model	Layers	KV heads	Head dim	KB per token	at 8K	at 128K
Llama-3.1-8B	32	8	128	128 KB	1.1 GB	17.2 GB
Qwen3-32B	64	8	128	256 KB	2.1 GB	34.4 GB
Gemma-3-27B	62	16	128	496 KB	4.2 GB	66.6 GB
Llama-3.3-70B	80	8	128	320 KB	2.7 GB	42.9 GB
gpt-oss-120b	36	8	64	72 KB	0.6 GB	9.7 GB
DeepSeek-R1 (MLA)	61	—	—	69 KB	0.6 GB	9.2 GB

All computed with fp16 KV using the same formula as our calculator (GB = 10⁹ bytes). Gemma-3 and gpt-oss actually use sliding-window attention, so their real footprint is below the formula values — see “The formula is an upper bound” below.

Read the Qwen3-32B row again: at 128K context, the KV cache (34GB) exceeds the Q4 weights (20GB). A 24GB card that “runs Qwen3-32B fine” cannot touch its 32K-rated context, let alone 128K.

The full VRAM ledger: weights + KV + fixed overhead

The KV cache never appears alone. Our calculator’s full accounting is:

total = weights (params × bytes per param) + KV cache + 1.5 GB fixed overhead (CUDA context/runtime)

Take the most common card class — 24GB (RTX 3090/4090) — running Qwen3-32B at Q4_K_M (20.1GB of weights):

Scenario	KV	Total	Verdict on 24GB
8K context, fp16 KV	2.1 GB	23.7 GB	⚠️ barely fits
16K context, fp16 KV	4.3 GB	25.9 GB	❌ doesn’t fit
16K context, q8 KV	2.1 GB	23.7 GB	⚠️ barely fits
32K (native max), q8 KV	4.3 GB	25.9 GB	❌ needs a 32GB card

Three takeaways: on 24GB, a 32B model with fp16 KV gives you about 8K of usable context; q8 KV doubles that to 16K; and using the full 32K native window takes a 32GB RTX 5090 (25.9GB lands in “tight”) or a lower weight quant.

Small VRAM is the same problem scaled down. A 12GB RTX 3060 running Llama-3.1-8B (Q4_K_M, 4.9GB weights): 32K context totals 10.7GB — runs; 64K needs 15.0GB — doesn’t; 64K with q8 KV drops back to 10.7GB — runs again. q8 KV is the first way small cards buy long context.

Our verdict bands: total ≤ 80% of usable VRAM is comfortable, ≤ 100% is tight. Skip the arithmetic and open the VRAM calculator — pick a model, drag the context slider, watch the KV bar grow in real time.

GQA already saved you (and you didn’t notice)

Early Transformers used multi-head attention (MHA): as many KV heads as query heads. Then came the extreme, MQA — all query heads sharing a single KV head — which shrinks KV dramatically but costs noticeable quality. GQA (grouped-query attention) is the compromise: query heads are grouped, each group sharing one KV head. The 2023 GQA paper showed it costs almost no quality, and it has been the mainstream architecture since Llama 2 70B.

How much did it save you? Qwen3-32B has 64 query heads but only 8 KV heads. Under MHA, its 128K row would read 275GB instead of 34GB. GQA cut long-context cost by a factor of eight before you ever touched a setting.

But generosity varies by model. Gemma-3-27B has 16 KV heads — 496KB per token, the priciest row in the table (it compensates with sliding-window attention, see below). The similarly-sized Mistral-Small-3.2-24B has 8 KV heads at 160KB/token: a 3× difference in long-context cost. When comparing models, num_key_value_heads deserves the same attention as parameter count: fewer KV heads = cheaper long context.

MLA: DeepSeek’s compression trick

DeepSeek-R1/V3 take a different route: instead of full K/V, each layer stores a compressed 512-dim latent vector (kv_lora_rank) plus a 64-dim RoPE position vector (qk_rope_head_dim) — just 576 values per token per layer. 61 layers × 576 × 2 bytes ≈ 69 KB per token.

That’s how a 671B model ends up with a smaller KV cache than Llama-8B: 9.2GB at 128K context, versus 42.9GB for Llama-3.3-70B. Our calculator handles MLA models with a separate formula — it’s why R1’s 128K context is genuinely usable (capacity permitting) while other models’ is aspirational.

The cost just moved line items: MLA shifts the memory pressure onto the weights — 671B parameters, 411GB even at Q4. No free lunch.

The formula is an upper bound: sliding windows and hybrids

The standard formula assumes every layer remembers the entire context. Two architecture families break that assumption, and their real KV footprint lands below the formula:

• Sliding-window attention (SWA): half of gpt-oss’s layers are sliding-window layers that keep only a fixed window of recent tokens; Gemma-3 runs a 1024-token window on 5 of every 6 layers, with only 1 global layer. Window layers’ KV doesn’t grow with context.
• Hybrid linear attention: Qwen3.8-27B uses full attention on only 1 of every 4 layers (the rest are Gated DeltaNet linear layers whose state is context-length-independent), so the standard formula overestimates its KV by roughly 4×; Muse-Glimmer-30B is a 3:1 sliding/global hybrid with the same effect.

Our calculator deliberately applies the standard formula to these models anyway — conservative on purpose: if the formula says it fits, reality definitely fits.

Your three levers when context-starved

1. q8 KV cache (halve it): minimal quality cost for most workloads; the first thing to try.
   • llama.cpp: --cache-type-k q8_0 --cache-type-v q8_0 (V quantization also needs -fa, flash attention)
   • Ollama: environment variable OLLAMA_KV_CACHE_TYPE=q8_0
   • LM Studio: KV Cache Quantization in the model load settings Effect: Llama-3.1-8B at 128K drops from 17.2GB to 8.6GB.
2. Shorter context (linear savings): Ollama’s default context window is only 4096 tokens (2048 in older versions), far below the advertised maximum — that’s why your measured VRAM sits below calculator estimates. Size it to the task: 4–8K is plenty for chat; RAG needs your retrieved chunk total; only whole-codebase or long-document work justifies 32K+. Set it with /set parameter num_ctx 32768 or PARAMETER num_ctx 32768 in a Modelfile; llama.cpp uses -c 32768. A deliberate 32K beats both a default 4K and a blind 128K.
3. A different architecture (the structural fix): fewer KV heads (Qwen3-30B-A3B has just 4), MLA (DeepSeek), or hybrids (Qwen3.8 / Muse). This is a decision you make when choosing a model, not a setting you flip afterwards.

Common misconceptions

• “My weights are Q4, so the KV cache shrank too” — no. Weight and KV quantization are independent switches; runtimes default KV to fp16 even with Q2 weights.
• “Supports 128K = my card can use 128K” — the advertised window is an architecture capability; the usable window is a VRAM budget. The difference is exactly this article’s formula.
• “The KV cache is one copy, extra chats are free” — every concurrent session stores its own. Four 32K sessions of Qwen3-32B cost ~34GB of KV alone (fp16); for self-hosted serving, KV often becomes the bottleneck before weights.
• “Lowering the context hurts model quality” — it doesn’t. The window is memory length, not capability; old content beyond the window is simply dropped.
• “Bigger KV = smarter model” — KV cost is an architecture tax, not a capability metric. Gemma-3’s 16 KV heads are a design choice, not a quality promise; MLA caches almost nothing and the model is excellent.

Do it yourself: three steps to any model’s KV footprint

1. Open the model’s config.json on Hugging Face and find three numbers: num_hidden_layers, num_key_value_heads, and head_dim (if absent, use hidden_size ÷ num_attention_heads).
2. Apply the formula: 2 × layers × KV heads × head dim × 2 (fp16) × your target context, divided by 10⁹ for GB. For MLA models (e.g. DeepSeek), look up kv_lora_rank + qk_rope_head_dim instead — and drop the leading 2.
3. Rather not do arithmetic? Open the VRAM calculator, pick the model, and drag the context slider from 4K to 128K. Once running, cross-check with nvidia-smi or ollama ps — discrepancies usually trace back to the framework’s default context differing from what you assumed.

The KV cache isn’t an implementation detail — it’s the other half of your VRAM budget. Do this math before you buy a card, before you pick a model, and before you set num_ctx.

FAQ

Sources

• Hugging Face model configs (nLayers, nKvHeads, headDim)
• deepseek-ai/DeepSeek-R1 config (MLA parameters)
• GQA paper (Ainslie et al., 2023)
• Ollama FAQ (num_ctx default and KV cache quantization)