Step 6: The Roadmap: Where to Go After Your First Model

Updated 2026-09-11 · Verified 2026-09-11

You can now pick a model, install the tools, run it, and fix it when it breaks. This final step isn’t a finish line — it’s a map. Four directions worth exploring, each with a dedicated guide on this site. All speeds are theoretical estimates, ±30% in practice.

Direction 1: The VRAM cost of long context

“Supports 128K context” on a model card describes the architecture, not your GPU. The KV cache — the scratchpad where the model stores what it’s read — grows strictly linearly with context length, and the per-token price isn’t cheap: Llama-3.1-8B’s Q4 weights are 4.9GB, but a 128K-context KV cache costs about 17.2GB. The context costs 3.5× more than the model itself.

There’s a second tax: the longer the context, the more scratchpad every generated token must re-read, so generation slows down. The same RTX 4090 running an 8B model drops from a theoretical ~154 tok/s on short prompts to ~34 tok/s at 128K.

One sentence to remember: size the context to the task, not to the spec sheet. Everyday chat: 4–8K. RAG Q&A: 8–16K. Hundred-page PDFs: 32K and up. Full derivation in The VRAM Cost of Long Context.

Direction 2: How MoE models change the math

MoE (Mixture of Experts) models like Qwen3-30B-A3B and gpt-oss-120b always list two parameter numbers, and beginners habitually grab the wrong one:

• VRAM follows total parameters: which “experts” fire for the next token is decided at generation time, so every parameter must stay resident. Qwen3-30B-A3B totals 30.5B parameters — about 21GB at Q4, nearly the same as a dense 32B.
• Speed follows active parameters: each token only engages 3.3B of them, reading about 2.0GB of weights. On the same RTX 4090, that’s a theoretical ~374 tok/s versus ~38 tok/s for the dense 32B that takes up almost identical VRAM — a 10× gap.

So MoE doesn’t shrink VRAM; it makes hardware that can fit the model absurdly fast. It’s also the great exception for big-RAM CPU-offload setups. Details in MoE Model Hardware Requirements.

Direction 3: Picking a backend — Ollama, llama.cpp, vLLM

These three names come up together, but they’re three different things:

• Ollama: one installer, one command, the default answer for personal daily use. Its engine underneath is llama.cpp.
• llama.cpp: the engine itself, with every knob exposed (-ngl, --cache-type-k, multi-GPU split ratios…). Reach for it when you want full control, embedded deployment, or to understand what each switch does.
• vLLM: a serving framework built for many simultaneous users, using continuous batching to maximize total throughput across dozens or hundreds of concurrent requests. Solo chat doesn’t use its strengths; a team or product API is its home turf.

In one line: for yourself, start with Ollama and graduate to llama.cpp; for serving others, go vLLM.

Direction 4: When to leave local and rent cloud GPUs

Local isn’t a religion — it’s an equation. Three situations should push you toward rented cloud GPUs (billed hourly, returned when done):

1. Occasional use of huge models: a 70B+ model a few times a month is cheaper to rent by the hour on an A100 than to house on four cards of your own.
2. Short burst workloads: tens of thousands of documents to process this week? Rent a multi-GPU box, blitz through it, hand it back.
3. Unpredictable public traffic: if usage spikes and collapses, the cloud scales; your basement doesn’t.

Conversely, a model you run daily pays back local hardware faster than you’d expect — which is why this site has a cost comparison tool. Before committing to “stay local and add cards,” read Multi-GPU Setups for LLMs to learn what extra cards buy and what they don’t.

Graduation gift: your own build list

The learning path ends here, but your hardware planning is just beginning. Hand your target model and budget to the build planner below and let it draft your first — or next — rig:

Further reading: The VRAM Cost of Long Context · MoE Model Hardware Requirements · Multi-GPU Setups for LLMs