NVIDIA H100: Specs, Price, and How It Compares to the H200 and B200 for LLM Inference

H100 Key Specs

The NVIDIA H100 is built on the Hopper architecture with roughly 80 billion transistors. Per NVIDIA's H100 datasheet, the SXM5 form factor delivers 3,958 TFLOPS of FP8 Tensor Core performance with sparsity (1,979 dense), 1,979 TFLOPS of FP16 and BF16 with sparsity (989 dense), and 34 TFLOPS of FP64. Memory is 80GB of HBM3 running at 3.35TB/s, and the board draws up to 700W. NVLink bandwidth is 900GB/s per GPU.

The H100 was the workhorse of the 2022 to 2024 generative AI buildout. It trained most of the frontier models of that era and still serves the majority of production LLM inference in 2026. Its successors change the memory system (H200) or the architecture (Blackwell B200), but the H100's Hopper compute is the baseline that later cards are measured against.

SXM vs PCIe vs NVL

The H100 ships in three forms that trade bandwidth and power against server compatibility. The SXM5 is the high-bandwidth, high-power module for HGX and DGX systems. The PCIe card fits standard servers at half the power but loses bandwidth. The NVL bridges two GPUs with extra memory per die for large-model inference.

The bandwidth column decides inference throughput. The PCIe card's 2.0TB/s is about 60 percent of the SXM5's 3.35TB/s, so on the same model it produces proportionally fewer tokens per second even though both hold 80GB. The NVL pair carries 94GB per die, so two bridged GPUs offer 188GB total, which is why it targets 70B-and-larger inference where the SXM5's 80GB gets tight.

Why Memory Dominates LLM Inference

Autoregressive text generation is memory-bandwidth-bound, not compute-bound. On each decode step, the GPU loads the model layer weights and the cached keys and values into the compute cores, performs a relatively small amount of arithmetic, and emits one token. The bottleneck is the data movement, not the math.

This reorders GPU selection. A card with more bandwidth produces more tokens per second on the same model, because it feeds the cores faster. A card with more VRAM holds a larger model, a longer context window, or a bigger batch, because all of those live in memory. The H100 SXM5's 3.35TB/s and 80GB are the two numbers that set its inference envelope.

Two things compete for the 80GB during serving: the model weights, which are fixed in size, and the KV cache, which grows with every token of context and every concurrent request. The weights set the floor; the KV cache consumes whatever capacity is left. At long context and high batch size, the KV cache can dominate.

For a Llama 2 7B model in float16 with a 10,000-token context, the KV cache alone needs roughly 5GB, about one-third of the model's half-precision weight storage, per Hugging Face. That number scales with batch size and context length, which is why the H100's 80GB fills faster than an H200's 141GB on the same long-context workload.

H100 Price in 2026

H100 prices fell hard in 2026. A used H100 SXM5 that peaked near $30,000 to $40,000 in late 2023 now trades around $15,000 to $20,000 on the secondary market, with the decline driven by data centers retiring Hopper inventory as they move to Blackwell B200. New H100 PCIe cards still list around $25,000 to $30,000, and a full 8-GPU DGX H100 system runs $300,000 to $400,000.

H100 Cloud Rental by Provider

Renting an H100 is where the price collapse is most visible. On-demand rates that ran roughly $8 per hour in late 2024 now sit near $2 to $3 per GPU-hour on specialized clouds, a 64 to 75 percent drop. Hyperscalers charge more for the same silicon because they bundle networking, support, and reservations. Spot and preemptible instances undercut on-demand by 30 to 50 percent.

The 4x spread between the cheapest marketplace spot ($1.49) and the most expensive hyperscaler on-demand ($6.98) is not about the GPU. It is the same H100 silicon everywhere. The difference is what wraps it: dedicated versus multi-tenant, InfiniBand versus none, enterprise support and SLAs versus none, and reservation commitment versus pay-as-you-go. Match the wrapper to the workload rather than paying for interconnect a single-GPU inference job never uses.

H100 vs H200 vs B200

The H100, H200, and B200 form a clean ladder. The H200 is an H100 with a bigger, faster memory system and the same Hopper compute. The B200 is a new Blackwell architecture with roughly 2.3x the FP8 compute, native FP4, and 2.4x the memory of the H100.

The H200's gain over the H100 is purely memory. Both share identical Tensor Core compute, so on a model that fits in 80GB and is not bandwidth-starved they perform similarly; the H200 pulls ahead exactly when memory is the wall, at 70B-and-larger models and long context. See NVIDIA H200 for the full memory breakdown.

The B200 is a bigger jump: 2.4x the H100's VRAM, up to 8TB/s of bandwidth, and native FP4 for another halving of memory per weight beyond FP8. NVIDIA claims up to 4x faster LLM inference. It also draws 1,000W and costs $30,000 to $50,000 per GPU. In 2026 the B200 is the frontier card and the H100 is the value card. Which one wins depends on whether the model you serve needs more than 80GB and 3.35TB/s.

What Fits on One H100

The rule of thumb for fitting a model on a GPU is two terms: weight storage plus KV cache. Weight storage is parameters multiplied by bytes per weight. KV cache is the per-token memory multiplied by context length and batch size, and it grows as generation proceeds.

Bytes per weight depends on precision. FP16 and BF16 use 2 bytes per parameter; FP8 uses 1 byte. So a 70B model needs roughly 140GB in FP16 but only about 70GB in FP8. On the H100's 80GB, that is the difference between needing two cards and fitting on one with a small KV budget.

// Weight storage = parameters * bytes_per_weight
//   FP16/BF16 -> 2 bytes/param
//   FP8       -> 1 byte/param

const h100_capacity = 80                     // GB, SXM5 or PCIe

// 70B model
const params_70B = 70e9
const weights_fp16 = params_70B * 2 / 1e9    // ~140 GB -> needs 2x H100
const weights_fp8  = params_70B * 1 / 1e9    // ~70 GB  -> fits one H100
const kv_headroom  = h100_capacity - weights_fp8  // ~10 GB for KV cache

// 13B model in FP16
const weights_13B_fp16 = 13e9 * 2 / 1e9      // ~26 GB -> fits with large KV budget

// Reference: Llama 2 7B in float16 at 10,000-token context
// needs ~5 GB of KV cache (per Hugging Face) -- scales with context and batch.

Working the numbers: a 70B model in FP8 needs about 70GB for weights, leaving roughly 10GB of the H100's 80GB for KV cache and overhead. That fits, but with a tight context and batch budget. Push context or batch higher and you spill to a second H100, or move to an H200's 141GB where the same 70B model leaves about 71GB free. A 13B model in FP16 at 26GB, by contrast, has room to spare on a single H100.

Across 8 H100s (a full HGX or DGX node with 640GB total), models up to the 400B-to-700B class fit in FP8 with tensor parallelism, which is the configuration most open frontier models are served on. The single-card ceiling is roughly the 70B class in FP8; the node ceiling is where the large open models live.

FP8 and Why It Matters

FP8 is an 8-bit floating-point format with two encodings: E4M3 (4-bit exponent, 3-bit mantissa) and E5M2 (5-bit exponent, 2-bit mantissa). NVIDIA recommends E4M3 for weights and activations and E5M2 for gradients. The H100 supports both in hardware through fourth-generation Tensor Cores and the Transformer Engine.

The payoff is twofold. FP8 halves data storage versus FP16/BF16 and doubles throughput. Halving storage is what lets a 70B model occupy 70GB instead of 140GB and fit on one H100. Doubling throughput is why the H100 SXM5 lists 3,958 FP8 TFLOPS against 1,979 FP16 TFLOPS, an exact 2x.

FP8 is not free. The E4M3 format has a small dynamic range, so naive per-tensor quantization can introduce outlier errors. Production systems mitigate this: in vLLM, FP8 W8A8 quantization yields a 2x reduction in model memory and up to 1.6x higher throughput with minimal accuracy impact, storing a higher-precision scaling factor alongside each quantized tensor. DeepSeek-V3 validated FP8 training on a 671B model with relative loss error below 0.25 percent versus BF16.

The B200 extends this with native FP4, halving memory per weight again beyond FP8. The H100 has no FP4 hardware path, so 1 byte per weight is its floor. That is one of the structural reasons a memory-bound model that overflows 80GB in FP8 moves to a Blackwell card rather than staying on Hopper. For the mechanics of applying it, see FP8 Quantization.

The Used-Price Trajectory

The H100's secondary-market price is on a clear downward path, and the driver is supply, not weakening demand. As hyperscalers and neo-clouds move their fleets to B200, they retire H100 inventory that lands on the secondary market. Used SXM5 cards that peaked at $30,000 to $40,000 in late 2023 now trade around $15,000 to $20,000, and multiple analyses project another 30 to 50 percent decline over the next 18 to 24 months as B200 adoption accelerates.

For a buyer, that trajectory is a warning against owning. A depreciating asset that also needs 700W of power, dense cooling, and colocation space is a poor store of value when the same compute rents for $2 to $3 per hour with no capital outlay. The people who benefit from the price crash are renters and API users, not owners holding cards whose resale value is falling faster than they can amortize.

Skip the GPU: Serve Open Models Instead

The reason to size an H100 at all is usually to serve an open model. Morph runs a GPU fleet with custom kernels and serves open models directly through one OpenAI-compatible API, which removes the step of buying, powering, and sizing H100s yourself. The same physics on this page (VRAM sets model size, bandwidth sets tokens per second, FP8 makes large models fit) is what that fleet is tuned around.

The served lineup spans sizes that map to node types: GLM-5.2 (753B), Qwen 3.5 397B and Qwen 3.6 27B, MiniMax M3, and DeepSeek V4 Flash, alongside the fast-apply model morph-v3-fast at ~10,500 tok/s with speculative decoding, a technique that, like FP8, attacks the memory-bandwidth bottleneck rather than the compute one. See the full model lineup.

If you are choosing a GPU to serve a model, the decision reduces to two questions. Does the model plus its KV cache fit in the VRAM at your target context and batch size? And is the bandwidth high enough to hit your tokens-per-second target? The H100's 80GB at 3.35TB/s answers both for the 70B class in FP8 on a single card, and for larger open models across an 8-GPU node. For how serving systems extract throughput from a fixed memory budget, see LLM Inference Optimization.

Frequently Asked Questions

What are the NVIDIA H100 specs?

The H100 SXM5 has 80GB of HBM3 memory at 3.35TB/s of bandwidth, delivers 3,958 TFLOPS of FP8 Tensor Core compute with sparsity (1,979 dense) and 1,979 FP16/BF16 TFLOPS with sparsity, 34 TFLOPS of FP64, and draws up to 700W. It is built on the Hopper architecture with roughly 80 billion transistors and 900GB/s of NVLink bandwidth. The PCIe variant has 80GB of HBM2e at 2.0TB/s and a 350W TDP.

How much does an H100 cost in 2026?

As of July 2026, a used H100 SXM5 trades around $15,000 to $20,000 on the secondary market, down from a late-2023 peak near $30,000 to $40,000, as Blackwell B200 fleets displace Hopper inventory. New H100 PCIe cards run roughly $25,000 to $30,000, and an 8-GPU DGX H100 system is $300,000 to $400,000. Renting is far cheaper at about $2 to $3 per GPU-hour on specialized clouds.

How much does it cost to rent an H100 per hour?

In July 2026, H100 on-demand rentals run about $1.80 to $3.00 per GPU-hour on specialized clouds (Cudo $1.80, Vast.ai around $1.87, RunPod community $1.99, Lambda $2.99) and $3 to $7 on hyperscalers (AWS P5 around $3.90, GCP around $3.00, Azure around $6.98). Spot and preemptible instances undercut on-demand by roughly 30 to 50 percent, with marketplace spot rates near $1.49.

What is the difference between the H100 and H200?

The H100 and H200 use the same Hopper compute silicon and have identical Tensor Core throughput (both 3,958 FP8 TFLOPS and 1,979 FP16/BF16 TFLOPS with sparsity). The difference is memory: the H100 SXM has 80GB of HBM3 at 3.35TB/s, while the H200 has 141GB of HBM3e at 4.8TB/s. NVIDIA reports the H200 is 1.9x faster on Llama 2 70B inference because that workload is memory-bound.

H100 vs B200: how much faster is Blackwell?

The B200 is a Blackwell-architecture GPU with 192GB of HBM3e at up to 8TB/s and native FP4 support, which the H100 lacks. NVIDIA claims up to 4x faster LLM inference than the H100. The B200 draws 1,000W versus the H100's 700W and costs $30,000 to $50,000 per GPU. The H100 remains the value choice where its 80GB and 3.35TB/s are enough for the model you serve.

What is the difference between the H100 SXM, PCIe, and NVL?

The H100 SXM5 is the 700W board-mounted module for HGX and DGX systems, with 80GB HBM3 at 3.35TB/s and 900GB/s NVLink. The H100 PCIe is a 350W dual-slot card with 80GB HBM2e at 2.0TB/s that fits standard servers but has lower bandwidth and no board-level NVLink. The H100 NVL is a bridged two-GPU pair, each with 94GB HBM3 at about 3.9TB/s, tuned for large-model inference.

Can an H100 run a 70B model?

Yes, in FP8. A 70B model in FP8 needs roughly 70GB for weights, leaving about 10GB of the H100's 80GB for KV cache and overhead, which is enough for moderate context and batch on a single card. In FP16 the same model needs about 140GB of weights and requires two H100s. Larger models, long context, or high batch push you to multiple cards or to an H200's 141GB.

Related Resources

• NVIDIA H200: VRAM, Bandwidth, and How It Compares
• Morph Model Lineup
• GLM-5.2: 744B Open Coding Model
• FP8 Quantization
• LLM Inference Optimization
• AI Inference: How LLMs Serve Tokens
• Fast Apply Model: morph-v3-fast at 10,500 tok/s
• LLM Cost Optimization

Sources

• NVIDIA H100 Tensor Core GPU datasheet (SXM/PCIe FP8/FP16/FP64 TFLOPS, 80GB HBM3, 3.35TB/s, 700W, NVLink 900GB/s).
• Thunder Compute, "NVIDIA H100 Specs: Full Guide (2026)" (SXM/PCIe/NVL spec table, transistor count, July 2026 pricing).
• IntuitionLabs, "H100 Rental Prices Compared" (per-provider on-demand and spot $/GPU-hr, 2026).
• IntuitionLabs and Silicon Analysts, "NVIDIA AI GPU Pricing" (used H100 $15k-20k, secondary-market decline, B200 displacement).
• Spheron, "GPU Cloud Pricing Comparison 2026" and "NVIDIA B200 Complete Guide" (rental spread, B200 192GB/8TB/s/FP4/1,000W).
• Civo, "Comparing NVIDIA's B200 and H100" (B200 memory/bandwidth/compute multipliers vs H100).
• Hugging Face, LLM inference memory documentation (KV cache ~5GB for Llama 2 7B at 10,000-token context).
• vLLM FP8 quantization docs and DeepSeek-V3 technical report (FP8 memory/throughput and accuracy figures).