Why Memory Bandwidth Determines LLM Inference Speed

updated 2026-09-10 · verified 2026-09-10

The RTX 3090 and RTX 4090 both have 24GB of VRAM and load the same models. But running Llama-3.1-8B at Q4_K_M, the 3090 theoretically manages ~143 tok/s and the 4090 ~154 tok/s — a small gap. Drop to an RTX 3060 (12GB) and the same model falls to ~55 tok/s. The CUDA core counts of these three cards span nearly 5× (3584 to 16384), yet speed ranks almost perfectly by a different number: memory bandwidth, at 360 / 936 / 1008 GB/s.

That’s not a coincidence. This article explains why local inference tok/s is effectively locked by a single spec — memory bandwidth — and how to use that rule to pick a card, pick a quantization, and set expectations. Every speed figure here is a theoretical estimate, ±30%, computed with the same formulas as our on-site tools.

The decode mechanism: re-reading all weights, every token

An LLM generates text in two alternating phases:

Prefill: reading your prompt. All prompt tokens are processed in parallel in one large matrix multiplication — this eats GPU compute.
Decode: emitting tokens one at a time. Each new token requires a full forward pass — mathematically, that means reading every weight from VRAM once, multiplying by the current token’s activations, then reading the KV cache.

The crucial part is the decode step: in a single-user chat with batch size 1, the weights are pulled from memory to serve exactly one token. Nothing is amortized. A 4.9GB weight file means 4.9GB of memory traffic per token; 100 tokens of output means roughly 490GB of traffic. Since bandwidth is the ceiling on how many bytes you can move per second, the per-token time floor is simply weight size ÷ bandwidth — and tok/s is pinned. This is the core result of Pope et al.’s inference scaling paper: low-batch inference is memory-bandwidth-bound, not compute-bound.

The formula: bandwidth × 0.75 ÷ per-token weights

Our tools (and the GPU checker) use:

theoretical tok/s ≈ bandwidth (GB/s) × 0.75 ÷ weight bytes read per token (GB)
per-token weights = parameter count × bpw ÷ 8   (MoE models use *active* params — see below)

The 0.75 is an empirical efficiency factor: beyond the weight reads there is the KV cache, kernel scheduling, and irregular memory access, so real throughput typically lands at 60–90% of the theoretical peak; 0.75 is the midpoint. Treat every output as a theoretical estimate, ±30%.

Worked example, Llama-3.1-8B at Q4_K_M (bpw 4.9): per-token weights = 8.03 × 4.9 ÷ 8 ≈ 4.9GB. Theoretical speeds across hardware:

GPU / Mac	Bandwidth	8B Q4 theoretical tok/s
RTX 3060 12GB	360 GB/s	~55
RTX 4070 Ti Super	672 GB/s	~103
RTX A6000	768 GB/s	~117
RTX 3090	936 GB/s	~143
RTX 4090	1008 GB/s	~154
RTX 5090	1792 GB/s	~273
Mac mini M4 Pro	273 GB/s	~42
Mac Studio M3 Ultra	819 GB/s	~125

Bigger models scale the same way: Qwen3-32B (Q4_K_M, 20.1GB per token) runs at ~13 tok/s on a 3060, ~38 on a 4090, ~67 on a 5090; Llama-3.3-70B (Q4_K_M, 43.2GB per token) manages ~14 tok/s on an M3 Ultra, ~35 on an A100, ~58 on an H100 (all theoretical estimates, ±30%).

Look at the A6000 row: a 48GB professional card, far pricier than a 4090, with plenty of CUDA cores — and slower at decode, because its bandwidth (768 GB/s) trails the 4090’s (1008 GB/s). In the decode race, bandwidth is speed; the other specs don’t get a vote.

Why FLOPs and core counts barely matter

Split one decode token into its two time costs:

Reading the weights: 4.9GB ÷ 1008 GB/s ≈ 4.9ms
Doing the math: roughly 2 × 8B = 16 GFLOP, which on a 4090’s ~100 TFLOPS-class compute takes ≈ 0.16ms

The read takes 30× longer than the compute. The GPU spends almost all of its time waiting on memory with its cores idle — double the cores and 0.16ms becomes 0.08ms, which changes nothing; double the bandwidth and 4.9ms becomes 2.4ms, which doubles your speed.

In jargon: single-user decode has an arithmetic intensity of only 2–4 FLOPs per byte read, while a modern GPU’s ridge point (FLOPS ÷ bandwidth) sits in the hundreds. Intensity far below the ridge point means an ironclad bandwidth bottleneck. It’s also why the 4090→5090 upgrade — only ~33% more CUDA cores but 78% more bandwidth — lifts decode by about 78% (154 → 273, theoretical estimate ±30%). Whatever decode gain you see in reviews is GDDR7’s doing, not the extra cores.

Buying rule: on a spec sheet, VRAM capacity decides what fits (verify with the VRAM calculator), bandwidth decides how fast it runs; core counts and TFLOPS are nearly irrelevant for single-user decode.

Exception one: prefill is compute-bound

The rule flips during prefill. Reading a 4,000-token prompt arranges 4,000 tokens into one matrix and multiplies once — the same weights are now amortized across thousands of tokens, arithmetic intensity explodes, and the bottleneck switches back to compute.

This explains two common experiences:

On the same GPU, time to first token (prefill) and generation speed (decode) are two separate numbers. Waiting ages after pasting a long document, then watching tokens stream out quickly, is just the two bottlenecks taking turns.
Agent workloads re-prefill the entire conversation history every turn, so prefill dominates — which is why Macs (weak compute) feel awkward for agents, and why compute monsters like the H100 earn their keep on the serving side. The full trade-off is in Mac vs GPU for LLM inference.

Exception two: MoE pays by active parameters

Mixture-of-experts models activate only a slice of their parameters per token. During decode, each token only needs to read the weights of the experts it routes to — so the formula’s denominator uses active parameters:

Qwen3-30B-A3B: 30.5B total, 3.3B active → 3.3 × 4.9 ÷ 8 ≈ 2.0GB per token. A 3060 theoretically hits ~134 tok/s — faster than the dense 8B model a quarter its size.
gpt-oss-120b: 116.8B total, 5.1B active → ~3.1GB per token, ~197 tok/s theoretical on an M3 Ultra.
DeepSeek-R1: 671B total, 37B active → ~22.7GB per token at Q4, ~111 tok/s theoretical on an H100.

(All theoretical estimates, ±30%.) This is the engineering value of MoE: you pay VRAM by total parameters but pay bandwidth by active parameters. High-capacity, low-bandwidth machines (Mac Studio) paired with MoE models are the natural sweet spot this formula predicts.

Four real-world factors that bend the formula

The formula is a first-order approximation. These factors systematically push real results off it:

Quantization directly changes the denominator. From FP16 (bpw 16) to Q4_K_M (4.9) is a 3.3× reduction in bytes read per token, and a 3.3× speedup — an 8B on a 4090 goes from ~47 tok/s at FP16 to ~154 at Q4, theoretically. Quantization is a speed tool, not only a memory saver; the quality trade-offs are covered in Quantization explained.
Long context adds KV reads. The formula counts weights only; as context grows, each token also reads the entire KV cache, and real speed drops further. At 128K context the KV cache alone can reach tens of GB (see KV cache explained). Slowing down late in a long conversation is physics, not a software bug.
Multi-GPU bandwidth carries a 15% tax. Two identical cards ideally double bandwidth, but inter-GPU communication costs something; we model it as Σbandwidth × 0.85: dual 3090s running 70B Q4 land at ~28 tok/s theoretical (±30%). Mixed cards are worse — effective bandwidth is set by the slowest card.
CPU offload is a cliff. Whatever doesn’t fit in VRAM spills into system memory, dragging effective bandwidth down to 60–100 GB/s (we model 80). An 8B Q4 falls from ~154 tok/s on a 4090 to ~12. Whether that trade is worth it: Is CPU offload worth it.

Common misconceptions

“More cores means a faster LLM GPU” — only for prefill and batched throughput. In single-user decode, a 768 GB/s A6000 simply loses to a 1008 GB/s 4090.
“More VRAM = more speed” — capacity and bandwidth are independent specs that merely tend to rise together. A hypothetical 24GB 3060 would be no faster than the 12GB one.
“Overclocking a 3090 catches it up to a 4090” — memory overclocking can squeeze out a few percent of bandwidth, but the 936→1008 gap is only 8%, and framework overhead unrelated to the card swallows it. Don’t overclock for decode speed.
“The formula says 150 tok/s and I get 120 — is my card defective?” — ±30% is the stated tolerance, and framework, drivers, and context length all live inside it. First check ollama ps or nvidia-smi to confirm nothing was offloaded to CPU — that’s the only failure mode with order-of-magnitude impact.
“APIs serve hundreds of tok/s, my box is slow” — API providers amortize bandwidth across dozens or hundreds of concurrent requests; that changes the arithmetic intensity entirely. Single-user tok/s and datacenter tok/s are different quantities; don’t compare them.

Do it yourself: estimate any combo in three steps

Compute per-token weights: parameters (active params for MoE) × bpw ÷ 8. Look up bpw in the table in Quantization explained, or just use Q4_K_M = 4.9.
Find the bandwidth: the GPU spec sheet or Apple’s site, in GB/s. Apply the formula: bandwidth × 0.75 ÷ per-token weights.
Skip the arithmetic: open the GPU checker, pick a card and a model, and get the fit verdict and theoretical speed in one shot; then confirm with the VRAM calculator that your target context still fits.

Two things are worth memorizing: decode speed = bandwidth ÷ bytes read per token, give or take 30%; prefill and long context are a separate ledger. Run those two calculations before buying a card and you’ll learn more than from any benchmark roundup.

FAQ

Why do two GPUs with wildly different core counts run the same 8B model at speeds matching their bandwidth ratio?

Because single-user decode is memory-bandwidth-bound, not compute-bound. Every token re-reads all weights (about 4.9GB for an 8B at Q4) while the actual math is only ~16 GFLOP — on a 4090 the read takes 4.9ms, the compute 0.16ms, so speed is set by the read side. Moving from an RTX 4090 to a 5090 adds only ~33% more CUDA cores but 78% more bandwidth, and decode gets proportionally ~78% faster (theoretical estimate, ±30%).

Why is Q4 more than 3× faster than FP16?

Because each token reads fewer bytes. The same 8B model reads 16GB per token at FP16 and 4.9GB at Q4_K_M — on a 4090 the theoretical speed goes from ~47 to ~154 tok/s (±30%). Quantization is a speed tool, not just a VRAM saver, at the cost of some quality.

My measured tok/s is 20% off your estimate. Is that normal?

Yes. The 0.75 in the formula is an empirical efficiency factor; real throughput depends on the framework (llama.cpp/vLLM/MLX), drivers, sampling settings, and thermal throttling. Our stated tolerance is ±30% — use the number to compare hardware and predict orders of magnitude, not as an exact promise.

Is there any scenario where more cores at the same bandwidth do matter?

Yes — prefill. Prompt processing is one big parallel matrix multiplication and is compute-bound. Ingesting long documents, or agent loops that re-prefill a long history every turn, will directly reward more cores and FLOPs. Decode eats bandwidth, prefill eats compute; don't apply one rule to the other.

Which spec should I look at first when buying a GPU for local LLMs?

VRAM capacity first (it decides what fits — check with the VRAM calculator), then memory bandwidth (it decides how fast). Core counts and TFLOPS have almost no bearing on single-user decode speed; they only pay off in prefill and batched throughput.