GPUFits
GPUs / H100 80GB
H100 80GB for Local LLMs: What It Runs and How Fast
The H100 80GB is the end of the inference performance line — and something individual buyers should not touch: 80GB HBM3 at ~3.35 TB/s, theoretically ~58 tok/s on 70B Q4 and ~805 tok/s on gpt-oss-120b (5.1B active), with new cards at $25,000-40,000 (we use $30,000, est.).
It is designed for training and large-scale concurrent serving — a single local user cannot feed it: decode-phase bandwidth utilization and the 700W draw are waste, and its killer FP8/tensor-parallel features go unused. The correct way to use one is renting: hourly H100 for batch inference or fine-tuning is two orders of magnitude cheaper than ownership. The only legitimate buyers are institutions with deterministic demand. For individuals it is a reference frame: our speed/VRAM data shows how far your needs are from an H100 — and for the vast majority of local needs, a 3090 or a Mac Studio is enough.
Specs
| Nominal VRAM | 80 GB |
| Usable VRAM (for models) | 80.0 GB |
| Memory bandwidth | 3350 GB/s |
| Type | Discrete GPU |
| Release year | 2022 |
| MSRP | $30,000 |
| TDP | 700 W |
Value Metrics
Based on MSRP (no used price yet); used prices are market estimates, see commentary for volatility.
Model Verdict Matrix (Q4_K_M @8K)
| Model | VRAM needed | Verdict | Recommended quant | Theoretical speed | |
|---|---|---|---|---|---|
| Llama 3.2 3B | 4.4 GB | Comfortable | FP16 | ≈391 tok/s | Try it → |
| Llama 3.1 8B | 7.5 GB | Comfortable | FP16 | ≈156 tok/s | Try it → |
| Qwen3 8B | 7.7 GB | Comfortable | FP16 | ≈153 tok/s | Try it → |
| Phi-4 14B | 12.2 GB | Comfortable | FP16 | ≈85 tok/s | Try it → |
| Mistral Small 3.2 24B | 17.5 GB | Comfortable | FP16 | ≈52 tok/s | Try it → |
| Gemma 3 27B | 22.4 GB | Comfortable | FP16 | ≈46 tok/s | Try it → |
| Qwen3.8 27B | 20.7 GB | Comfortable | FP16 | ≈45 tok/s | Try it → |
| Muse Glimmer 30B | 20.1 GB | Comfortable | FP16 | ≈42 tok/s | Try it → |
| Qwen3 30B-A3B | 21.0 GB | Comfortable | FP16 | ≈381 tok/s | Try it → |
| Qwen3 32B | 23.7 GB | Comfortable | FP16 | ≈38 tok/s | Try it → |
| gpt-oss-20b | 14.7 GB | Comfortable | FP16 | ≈349 tok/s | Try it → |
| Llama 3.3 70B | 47.4 GB | Comfortable | Q8_0 | ≈33 tok/s | Try it → |
| gpt-oss-120b | 73.6 GB | Tight fit | Q4_K_M | ≈804 tok/s | Try it → |
| DeepSeek-R1 671B | 413.1 GB | Not feasible | — | — | Try it → |
Theoretical speed = bandwidth × 0.75 ÷ per-token weight bytes (active params for MoE); real-world results vary with framework/driver/CPU, ±30%.
FAQ
- What can the H100 80GB run?
- Everything in our 14-model list comfortably except full DeepSeek-R1 (4+ cards). gpt-oss-120b Q4 theoretically ~805 tok/s — the single-card experience ceiling.
- Should an individual buy an H100 for local LLMs?
- No. $30,000 of ownership cost converts to many years of Vast.ai/RunPod hourly rental; and single-user decode cannot saturate 3.35 TB/s, leaving most performance idle.
- H100 vs A100 for inference?
- Bandwidth 3350 vs 2039 GB/s — theoretically 64% faster decode at ~2× the price. A100 is better single-user inference value; H100's value is training, FP8, and high-concurrency serving.
Related guides
- FP8, NVFP4, or GGUF Quants: Datacenter Formats vs llama.cpp Formats
- Why Memory Bandwidth Determines LLM Inference Speed
- Multi-GPU Setup for Local LLMs: NVLink, PCIe & Layer Splitting
Try H100 80GB in the GPU compatibility checker →
Data verified 2026-09-01