KV Cache Explained: Why Long Contexts Eat Your VRAM

updated 2026-08-04 · verified 2026-08-04

Every “this model supports 128K context” claim comes with an asterisk measured in gigabytes. That asterisk is the KV cache.

What it is

As the model reads your prompt, it stores an intermediate representation (the K and V tensors) of every token, in every layer, so it never recomputes them. That store is the KV cache. It’s what makes generation fast — and it’s why a long conversation or a big document eats VRAM independently of the model’s weights.

The formula

KV cache = 2 × nLayers × nKvHeads × headDim × bytesPerValue × contextTokens

The 2 is for K and V. bytesPerValue is 2 for fp16 (the default), 1 for q8. The architecture constants come straight from each model’s config:

Model	Layers	KV heads	Head dim	KB per token	at 8K	at 128K
Llama-3.1-8B	32	8	128	128 KB	1.1 GB	17.2 GB
Qwen3-32B	64	8	128	256 KB	2.1 GB	34.4 GB
Llama-3.3-70B	80	8	128	320 KB	2.7 GB	42.9 GB
gpt-oss-120b	36	8	64	90 KB	0.7 GB	11.8 GB
DeepSeek-R1 (MLA)	61	—	—	70 KB	0.6 GB	9.4 GB

Read the Qwen3-32B row again: at 128K context, the KV cache (34GB) exceeds the Q4 weights (20GB). A 24GB card that “runs Qwen3-32B fine” cannot touch its 32K-rated context, let alone 128K.

GQA already saved you (and you didn’t notice)

Modern models use grouped-query attention: Qwen3-32B has 64 query heads but only 8 KV heads. Older architectures kept them equal — which would make that 128K row read 275GB instead of 34GB. When comparing models, fewer KV heads = cheaper long context.

MLA: DeepSeek’s compression trick

DeepSeek-R1/V3 cache a compressed latent vector instead of full K/V: 61 layers × (512 + 64) values per token. That’s how a 671B model ends up with a smaller KV cache than Llama-8B. Our calculator handles MLA models with a separate formula — it’s why R1’s 128K context is genuinely usable (capacity permitting) while other models’ is aspirational.

Your three levers when context-starved

q8 KV cache (halve it): supported by llama.cpp/Ollama/LM Studio, minimal quality cost. The first thing to try.
Shorter context (linear savings): Ollama’s default num_ctx=2048 is why your measured VRAM is lower than calculator estimates — set it deliberately.
A model with fewer KV heads or MLA: a structural fix when you shop for models, not a setting.

FAQ

How much VRAM does 128K context actually cost?

For Llama-3.1-8B: about 17GB — more than 3× the Q4 weights. For Qwen3-32B: about 34GB, exceeding the weights. This is why 'supports 128K context' and 'usable 128K context on your GPU' are different claims.

Does q8 KV cache hurt quality?

Measurably little for most workloads — it's the recommended trade when context-starved. llama.cpp, Ollama and LM Studio all support KV quantization; enabling it halves the KV footprint.

Why is my Ollama using less VRAM than your calculator says?

Ollama defaults to num_ctx=2048, not the model's maximum context. At 2048 tokens the KV cache is tiny. Our calculator defaults to 4096 and lets you slide up to the model's real maximum.