Guides

In-depth guides: VRAM per billion parameters, quantization explained, best GPUs for local LLMs, Mac vs GPU, multi-GPU setups, and more.

• Best GPU for Local LLMs in 2026: Every Budget Tier

  The 2026 buyer's guide for local LLM hardware: best used value (RTX 3090), best new cards (RTX 5090), best big-memory option (Mac Studio), and what to skip.

  verified 2026-08-04

• CPU Offload: Is Partial GPU Offloading Worth It?

  llama.cpp -ngl explained: what partial offload does to speed (the 60-100 GB/s memory wall), when it's a fair trade, and when a smaller model wins.

  verified 2026-08-04

• FP8, NVFP4, or GGUF Quants: Datacenter Formats vs llama.cpp Formats

  FP8 and NVFP4 run natively on Tensor Cores; GGUF Q4/Q8 is llama.cpp's portable quantization. Hardware gates, size and quality trade-offs, which to pick.

  verified 2026-09-10

• How Much VRAM per Billion Parameters? (2026 Cheat Sheet)

  VRAM cheat sheet: GB per billion parameters at FP16, Q8, Q6, Q5, Q4, Q3 and Q2, plus KV cache and overhead — with worked examples and GPU tiers.

  verified 2026-08-04

• KV Cache Explained: Why Long Contexts Eat Your VRAM

  What the KV cache is, the exact formula, why 128K context can cost more than the model itself, and how MLA and q8 KV cache change the math.

  verified 2026-08-04

• The VRAM Cost of Long Context (and How to Right-Size It)

  Double the context, double the KV cache. Per-token KV costs by model, GQA/MLA/sliding-window differences, the speed penalty, and sizing context per task.

  verified 2026-09-10

• Mac vs GPU for LLM Inference: Unified Memory Explained

  Apple Silicon unified memory vs NVIDIA VRAM: real bandwidth numbers (273/546/819 GB/s), the 75% usable-memory rule, worked examples, and who should buy which.

  verified 2026-08-04

• Why Memory Bandwidth Determines LLM Inference Speed

  Decode re-reads all weights per token: tok/s ≈ bandwidth × 0.75 ÷ weight GB. Why FLOPs barely matter, why prefill is the exception, why MoE runs fast.

  verified 2026-09-10

• MoE Model Hardware Requirements: Total vs Active Parameters

  MoE models like Qwen3-30B-A3B, gpt-oss-120b and DeepSeek-R1 have two parameter counts: VRAM follows total parameters, speed follows active ones.

  verified 2026-09-10

• Multi-GPU Setup for Local LLMs: NVLink, PCIe & Layer Splitting

  How to run 70B+ models on multiple GPUs: tensor parallelism vs layer splitting, why you don't need NVLink for inference, and the real speed math.

  verified 2026-08-04

• Quantization Explained: Q4 vs Q8 and What You Actually Lose

  What GGUF quantization costs: Q8_0 vs Q6_K vs Q5_K_M vs Q4_K_M vs Q3/Q2 — measured file sizes, VRAM and speed math, per-scenario picks, when to go below Q4.

  verified 2026-08-04

• Running DeepSeek-R1 Locally: The Complete Hardware Guide

  DeepSeek-R1 671B needs ~413GB at Q4 — every realistic path: the Mac Studio truth, the multi-GPU bill, R1 distills that actually fit, and when to use the API.

  verified 2026-08-04

• Run GGUF Models Locally: llama.cpp and Ollama Walkthrough

  A practical start-to-finish guide: downloading GGUF files, picking a quant, llama.cpp's -ngl and --ctx-size, Ollama's num_ctx, and verifying VRAM and tok/s.

  verified 2026-09-10