H100 vs H200: The Difference Is Memory, and When It Actually Matters

The Short Answer

The H100 and H200 are the same GPU with a different memory system. If your model, context, and batch fit comfortably in the H100's 80GB and you are not bandwidth-starved, buy or rent the H100, because it is cheaper and the compute is identical. If your workload is memory-bound (a 70B-and-larger model, long context, or high batch where the KV cache dominates), pick the H200: the 1.4x bandwidth and 1.8x capacity translate into 2 to 4x the throughput per GPU, which usually makes it cheaper per token despite costing more per hour.

Same Die, Different Memory

The H200 is not a new chip. It is the Hopper GH100 die, the same silicon as the H100, wired to a bigger and faster memory stack. NVIDIA's own spec tables list identical Tensor Core throughput for both cards: 3,958 TFLOPS of FP8 with sparsity, 1,979 TFLOPS of FP16 and BF16 with sparsity, and 34 TFLOPS of FP64. Both SXM boards draw 700W and carry 900GB/s of NVLink. Nothing about the compute changed between the two generations.

What changed is the memory. The H100 SXM uses five 16GB HBM3 stacks for 80GB at 3.35TB/s. The H200 uses six 24GB HBM3e stacks for 141GB at 4.8TB/s. NVIDIA describes this as nearly 1.8x more capacity and 1.4x more bandwidth. The H200 was the first shipping GPU with HBM3e memory.

Why the 1.4x Bandwidth Becomes 1.9x tok/s

Autoregressive text generation is memory-bandwidth-bound, not compute-bound. On each decode step, the GPU loads the model layer weights and the cached keys and values into the compute cores, performs a relatively small amount of arithmetic, and emits one token. The bottleneck is the data movement, not the math. A card that moves memory faster produces more tokens per second on the same model, even when its Tensor Cores are identical.

The H200's 4.8TB/s is 1.4x the H100's 3.35TB/s, so a purely bandwidth-limited decode should run about 1.4x faster. NVIDIA measures 1.9x on Llama 2 70B, higher than the raw bandwidth ratio, and the extra gain comes from capacity. On the H100, a 70B model has to be split with tensor parallelism across two cards for optimal serving, which adds inter-GPU communication. On the H200, 141GB holds the whole model on one card, so that communication overhead disappears. Bandwidth plus fewer GPU hops is why the measured speedup exceeds the bandwidth number alone.

Two things compete for the memory during serving: the model weights, fixed in size, and the KV cache, which grows with every token of context and every concurrent request. The weights set the floor; the KV cache consumes whatever capacity is left. For a Llama 2 7B model in float16 at a 10,000-token context, the KV cache alone needs roughly 5GB, about one-third of the model's half-precision weight storage, per Hugging Face. That number scales with batch and context, which is why the H100's 80GB fills faster than the H200's 141GB on the same long-context load, and why the H200 can hold more concurrent requests before it runs out of room.

Measured Inference Benchmarks

The headline number is NVIDIA's MLPerf result: 31,712 tokens per second for the H200 against 22,290 for the H100 on Llama 2 70B in the offline scenario, a 1.9x gap. But throughput at maximum batch is only one operating point. What matters for a real service is throughput at a fixed per-user speed, because users care about how fast their own tokens arrive, not the aggregate.

SemiAnalysis's InferenceX measured exactly that on Llama 3.3 70B, sweeping interactivity from low to high. The gap widens sharply as you demand faster per-user output, because the H100 runs out of memory headroom and has to shrink its batch while the H200 keeps serving more concurrent users.

The pattern is the point. At low interactivity, where both cards can run large batches, the H200 leads by about 2x, close to its bandwidth-plus-capacity edge. At high interactivity, where the H100's 80GB forces a small batch to keep latency down, the H200 leads by nearly 5x, because its 141GB sustains a far larger batch at the same per-user speed. The memory gap compounds precisely in the regime most production chat and agent workloads run in.

Where the H200 Does Not Help

The H200 is a memory upgrade, so it does nothing for work that is not memory-bound. Two cases are worth naming explicitly, because paying the H200 premium for either is wasted money.

The first is prefill. Before a model emits its first token, it processes the entire prompt in a single compute-heavy pass. Prefill is compute-bound, and the compute is identical on both cards. On a prompt-heavy, generation-light workload (classification, short answers over long documents, embedding), the H200's bandwidth advantage barely shows, because the bottleneck is the Tensor Cores both cards share.

The second is small models. A 7B or 13B model in FP8 fits in a fraction of 80GB with room for a large KV cache and batch. At that size the H100 is rarely bandwidth-starved at practical batch sizes, so the H200 buys little. The two cards converge, and the H100's lower price wins. The H200's memory only pays off once the model plus KV cache pushes against the 80GB ceiling.

What Fits: 80GB vs 141GB

The rule for fitting a model on a GPU is two terms: weight storage plus KV cache. Weight storage is parameters multiplied by bytes per weight. FP16 and BF16 use 2 bytes per parameter; FP8 uses 1 byte. So a 70B model needs about 140GB in FP16 and about 70GB in FP8. The KV cache is per-token memory multiplied by context length and batch, and it grows as generation proceeds.

// weights = parameters * bytes_per_weight   (FP8 -> 1, FP16 -> 2)

const h100 = 80    // GB, HBM3
const h200 = 141   // GB, HBM3e

// --- 70B in FP8 (~70 GB of weights) ---
const w70 = 70e9 * 1 / 1e9            // ~70 GB
const kv_on_h100 = h100 - w70         // ~10 GB  -> tight KV budget
const kv_on_h200 = h200 - w70         // ~71 GB  -> comfortable

// --- 120B in FP8 (~120 GB of weights) ---
const w120 = 120e9 * 1 / 1e9          // ~120 GB
// h100 - w120 -> NEGATIVE: does not fit on one 80 GB card
const kv_120_on_h200 = h200 - w120    // ~21 GB  -> fits on one H200

// --- 70B in FP16 (~140 GB of weights) ---
const w70_fp16 = 70e9 * 2 / 1e9       // ~140 GB -> needs 2 cards either way

// Reference: Llama 2 7B, float16, 10,000-token context
// needs ~5 GB of KV cache (Hugging Face) -- scales with context * batch.

Both cards run a 70B model in FP8, but the headroom differs by 7x: about 10GB of KV budget on the H100 against 71GB on the H200. That headroom is the batch and context you can serve. The sharper line is at 120B: a 120B model in FP8 needs about 120GB of weights, which overflows the H100's 80GB entirely but fits on a single H200 with room for a small KV cache. This is the concrete meaning of the H200's extra 61GB. It moves the single-GPU ceiling from the 70B class toward the 120B class.

The gap scales at node level too. An 8-GPU H100 node holds 640GB; an 8-GPU H200 node holds 1,128GB. GLM-5.2 at 744B needs roughly 755GB in FP8, which is over what a single H100 node can hold but fits on an H200 node with overhead to spare. For the full single-card breakdown of each card, see NVIDIA H100 and NVIDIA H200.

Price in July 2026

The rental prices sit close. On-demand H100 runs about $2 to $3 per GPU-hour on specialized clouds and H200 runs about $2.20 to $4.50, so the H200 premium is roughly $1 to $2 per hour, not a multiple. Hyperscalers charge more for both because they bundle networking and support. On the buy side the gap is wider, because the H100 has depreciated into the secondary market while the H200 is still in its first deployment cycle and rarely appears used.

Per-provider July 2026 H200 on-demand rates span Hyperbolic at $2.40, Vast.ai at $3.71, Crusoe at $4.29, RunPod at $4.39, Nebius at $4.50, CoreWeave at $6.16, AWS p5e at $7.91, and Azure ND H200 v5 at $10.60. The 4x spread is not the GPU; it is the wrapper of dedicated versus multi-tenant, InfiniBand versus none, and enterprise SLA versus pay-as-you-go. Match the wrapper to the workload rather than paying for interconnect a single-GPU job never touches.

Price per Token, Not per Hour

The per-hour premium is misleading, because the H200 serves far more tokens per hour on memory-bound work. The right unit is cost per million tokens, which folds throughput and rental rate into one number. On that unit the H200 is usually cheaper on the workloads it is built for.

At mid interactivity the H200 costs $0.18 per million tokens against the H100's $0.45, roughly 60 percent cheaper, even though it rents for more per hour. At high interactivity the gap is 4.5x, because the H100's throughput collapses when its memory forces a small batch. The lesson generalizes: for a card whose only advantage is memory, the honest cost comparison has to be per token on your workload, never per hour on the spec sheet.

B200 Over the Decision

The B200 changes the frame. It is a Blackwell-architecture GPU that puts two dies in one package linked by a 10TB/s die-to-die interconnect, with 192GB of HBM3e at up to 8TB/s and fifth-generation Tensor Cores. Its native FP4 format halves memory per weight again beyond FP8 and roughly doubles throughput where accuracy holds. NVIDIA claims up to 4x faster LLM inference than the H100, at $30,000 to $50,000 per GPU and 1,000W.

The B200 wins frontier-scale, high-throughput serving and new FP4 training runs. But it does not erase the H100-versus-H200 question. For inference on models below 70B, or workloads that fit comfortably in Hopper memory, H100 and H200 clusters stay cost-competitive with the B200 per token, because the FP4 advantage compounds only at very high throughput and model sizes. Every generation prices at a premium while supply is tight, then falls as the next ships. The H100 went from $8 per hour in early 2024 to under $3 in 2026; the B200 will follow once Vera Rubin volume pulls demand off Blackwell. If your workload is memory-bound Hopper-class inference today, the live choice is still H100 or H200.

Which One to Pick

The decision reduces to two questions about your workload, both about memory. First: does the model plus its KV cache fit in 80GB at your target context and batch? If yes with room to spare, the H100 is the value pick. If it is tight or overflows, the H200's 141GB earns its price. Second: is your traffic interactive at 70 to 90 tokens per second per user? If so, the H200's throughput edge widens to 2.8x to 4.9x, and it is almost certainly cheaper per token.

Skip the Procurement Question

The reason to compare these cards at all is usually to serve an open model. Morph runs a GPU fleet on this class of hardware with custom kernels and serves open models through one OpenAI-compatible API, which removes the buy-versus-rent, H100-versus-H200 decision entirely. The same physics on this page (VRAM sets model size, bandwidth sets tokens per second, FP8 makes large models fit) is what that fleet is tuned around, so you consume the throughput without sizing GPUs yourself.

The served lineup spans sizes that map to node types: GLM-5.2 (753B), Qwen 3.5 397B and Qwen 3.6 27B, MiniMax M3, and DeepSeek V4 Flash, alongside the fast-apply model morph-v3-fast at ~10,500 tok/s with speculative decoding, a technique that, like the H200's bandwidth, attacks the memory-bandwidth bottleneck rather than the compute one. See the full model lineup.

If you are choosing a GPU to serve a model, the decision is the two-question test above. If you would rather not run the test, the served path answers both for you. For how serving systems extract throughput from a fixed memory budget, see LLM Inference Optimization and Continuous Batching. For how the H100 stacks against Ampere, see A100 vs H100.

Frequently Asked Questions

What is the difference between the H100 and H200?

The H100 and H200 are the same Hopper GH100 die with identical Tensor Core compute (both 3,958 FP8 TFLOPS and 1,979 FP16/BF16 TFLOPS with sparsity). The only difference is memory: the H100 SXM has 80GB of HBM3 at 3.35TB/s, while the H200 has 141GB of HBM3e at 4.8TB/s, which NVIDIA describes as nearly 1.8x the capacity and 1.4x the bandwidth. Because LLM inference is memory-bound, that bandwidth gap makes the H200 up to 1.9x faster on Llama 2 70B.

Is the H200 worth the extra cost over the H100?

It depends on whether your workload is memory-bound. For 70B-and-larger models, long context, or high batch sizes where the KV cache dominates, the H200 delivers 2 to 4x the throughput per GPU at fixed interactivity, which usually makes it cheaper per token despite the ~$1 to $2 per hour rental premium. For small models that fit in 80GB and are not bandwidth-starved, the two perform similarly and the H100 is the value choice, since the compute is identical.

How much faster is the H200 than the H100 for inference?

NVIDIA measures the H200 at 1.9x the H100 on Llama 2 70B in the MLPerf offline scenario (31,712 vs 22,290 tokens/second). SemiAnalysis measures larger gaps at fixed interactivity on Llama 3.3 70B: at 72 tokens/second per user, the H200 delivers 2,184 tokens/second per GPU versus the H100's 773 (2.8x), and at 91 tokens/second per user, 1,480 versus 305 (4.9x). The speedup comes entirely from the H200's 1.4x bandwidth and larger capacity, not faster math.

What is the price difference between the H100 and H200?

In July 2026, H100 on-demand cloud rentals run about $2 to $3 per GPU-hour on specialized clouds, while H200 rentals run about $2.20 to $4.50, a premium of roughly $1 to $2 per hour. To buy, a used H100 SXM5 trades around $15,000 to $20,000 and an H200 SXM lists around $30,000 to $45,000. Per token, the H200 is often cheaper on memory-bound workloads because it serves far more throughput per GPU.

Can the H200 run a 70B model that the H100 cannot?

Both can run a 70B model in FP8, but with very different headroom. On the H100's 80GB, a 70B model in FP8 (about 70GB of weights) leaves only ~10GB for KV cache, which is tight for context and batch. On the H200's 141GB, the same model leaves ~71GB free. The bigger difference is at 120B: that model in FP8 needs ~120GB of weights, which does not fit on one H100 but fits on a single H200.

Do the H100 and H200 have the same compute?

Yes. The H100 and H200 use the same Hopper GH100 die and have identical Tensor Core throughput: both 3,958 FP8 TFLOPS and 1,979 FP16/BF16 TFLOPS with sparsity, and both draw 700W in SXM form. Only the memory subsystem differs. This is why the H200 gives no advantage on compute-bound work like prefill or small models, and only pulls ahead on memory-bound decoding.

Should I wait for the B200 instead of buying an H100 or H200?

The B200 is a Blackwell-architecture GPU with 192GB of HBM3e at up to 8TB/s and native FP4, and NVIDIA claims up to 4x faster inference than the H100. It costs $30,000 to $50,000 per GPU and draws 1,000W. For frontier-scale, high-throughput serving the B200 wins, but for inference on models below 70B or workloads that fit comfortably in Hopper memory, H100 and H200 clusters remain cost-competitive per token. The H100-vs-H200 choice is still live for memory-bound Hopper inference.

Related Resources

• NVIDIA H100: Specs, Price, and Comparisons
• NVIDIA H200: VRAM, Bandwidth, and How It Compares
• A100 vs H100: The Generational Jump
• Morph Model Lineup
• LLM Inference Optimization
• Continuous Batching
• FP8 Quantization
• Fast Apply Model: morph-v3-fast at 10,500 tok/s

Sources

• NVIDIA H100 and H200 Tensor Core GPU datasheets and spec pages (FP8/FP16/FP64 TFLOPS, 80GB HBM3 at 3.35TB/s, 141GB HBM3e at 4.8TB/s, 700W, NVLink 900GB/s, 5x16GB vs 6x24GB stacks).
• NVIDIA Technical Blog and MLPerf Inference v4.0 (H200 1.9x on Llama 2 70B offline; 31,712 vs 22,290 tokens/second; single-card serving removes tensor-parallel overhead).
• SemiAnalysis InferenceX, "Llama 3.3 70B H100 vs H200" (throughput per GPU and cost per million tokens at 53/72/91 tok/s/user interactivity).
• Hyperbolic, "H200 vs H100 for LLM Inference" (141GB vs 80GB, 6x24GB vs 5x16GB stacks, single-GPU 70B serving).
• Thunder Compute, "NVIDIA H200 / H100 Pricing (July 2026)" (per-provider on-demand $/GPU-hr; buy and HGX system prices).
• Baseten and Data Science Collective, "H100 vs H200 vs B200" (same GH100 die, 8-GPU node capacities 640/1,128/1,440GB, GLM-5.2 755GB FP8 fit, decision guidance).
• Spheron and Civo (B200 192GB/8TB/s/FP4/1,000W, price trajectory, per-token cost-competitiveness of Hopper below 70B).
• Hugging Face, LLM inference memory documentation (KV cache ~5GB for Llama 2 7B at 10,000-token context).