Step 2: Hardware 101: VRAM Decides If It Runs, Bandwidth Decides How Fast

Updated 2026-09-11 · Verified 2026-09-11

One analogy: desk size and hand speed

Picture your GPU as the desk you do homework on.

VRAM is the size of the desktop. All of the model’s “parts” — the technical term is weights, the knowledge the model learned — must be laid out on the desk at once before work can start. If the desk is too small, the books go back on the shelf — that’s your system RAM — and every use means getting up to fetch them. How slow that is comes later.

Memory bandwidth is your hand speed. A model generates text in a dumb way: for every single word it writes, it flips through all the material on the desk from cover to cover. The thicker the material (the bigger the model), the slower it writes; the faster your hands (the higher the bandwidth), the faster it writes. How smart your brain is — the GPU’s compute, its core count — barely helps with pure page-flipping.

Why VRAM comes first

Because it’s a gate: fail it and nothing else matters. How much VRAM a model needs is simple arithmetic:

VRAM needed ≈ weights (parameters × bytes per parameter) + conversation cache + ~1.5GB overhead

Take Llama 3.1 8B: 8 billion parameters. With Q4 quantization (“quantization” means storing the model’s numbers compressed — smaller file, minimal quality loss), the weights are about 4.9GB; add the conversation cache and overhead and you get roughly 7.5GB — any 12GB card fits it comfortably. But a 70B model needs about 43GB of weights alone, which no single consumer GPU can hold.

Not fitting doesn’t mean “a bit slower” — it means “won’t run” or “too slow to use.” So the first question when picking hardware is always: how much VRAM does the model I want need, and how much does my card have?

Bandwidth decides how fast

Once the model fits, speed comes down to bandwidth alone. The estimation formula (the same one our tools use):

theoretical speed (tok/s) ≈ bandwidth (GB/s) × 0.75 ÷ weight bytes read per token (GB)

tok/s means “tokens generated per second” — roughly, words per second. Plugging in an 8B Q4 model (about 4.9GB of weights read per token):

• RTX 3060 (360 GB/s): ~55 tok/s
• RTX 4090 (1008 GB/s): ~154 tok/s
• Mac mini M4 Pro (273 GB/s): ~42 tok/s

These are theoretical estimates — real results shift with software and thermals, ±30% — but they’re more than fair for comparing hardware. Note the counterintuitive part: speed isn’t set by how expensive the card is or how many cores it has. The 48GB professional RTX A6000 costs far more than a 4090, but its bandwidth is lower (768 GB/s), so it runs the same model slower.

Why system RAM can’t substitute for VRAM

Everyone’s first thought: “I have 64GB of RAM — way more than VRAM.” True on capacity, but the problem is hand speed: ordinary dual-channel DDR5 delivers about 60–100 GB/s, versus 1008 GB/s on an RTX 4090 — a tenfold gap. Move an 8B model from VRAM into RAM and speed drops from ~154 tok/s to ~12 tok/s (theoretical estimate) — from a typewriter to one word at a time, and chat becomes unusable.

The one exception is Apple-silicon Macs: CPU and GPU share a single memory pool, so there’s no “fetching from the shelf” at all, and a high-memory Mac can run big models directly. But that pool’s bandwidth (120–819 GB/s) is still below high-end GPUs — Macs win on capacity, not speed.

How to check your own card

On Windows, press Ctrl+Shift+Esc to open Task Manager, go to Performance → GPU, and “Dedicated GPU Memory” in the corner is your VRAM capacity. Bandwidth isn’t shown anywhere in the OS — look it up on the card’s spec sheet, or just open our GPU library, where both numbers are listed for every card.

Hands-on: find your card

Remember two things from today: capacity decides whether it fits, bandwidth decides how fast it runs. You don’t need to memorize any numbers — open the GPU library below, find your card, and read those two columns.

For the full derivation of why bandwidth specifically, in depth: Why Memory Bandwidth Decides LLM Speed.