A100 vs H100: Specs, Price, and Which GPU to Buy for LLM Inference in 2026

A100 vs H100 at a Glance

The A100 launched in 2020 on the Ampere architecture; the H100 launched in 2022 on Hopper. Two years of process and architecture progress separate them. The H100 moves to TSMC 4N with 80 billion transistors against the A100's 54.2 billion on 7nm, and every Tensor Core precision row moves up. The single most consequential addition is FP8: the H100's fourth-generation Tensor Cores and Transformer Engine execute attention and feed-forward math in FP8 (E4M3 and E5M2), which the A100 cannot do at all.

For the buyer, the decision is not which card is faster. The H100 is faster on every axis. The decision is whether the H100's speed justifies roughly twice the hourly price for your specific model and workload, and that answer flips depending on model size, whether you can use FP8, and how much throughput you actually need.

Full Spec Comparison

Comparing the 80GB SXM variant of each card, the one most teams deploy for LLM serving. The A100 also ships in a 40GB variant and both come in lower-power PCIe form factors; those rows follow the table.

The A100 40GB uses HBM2 at 1.56TB/s; the 80GB SXM steps up to HBM2e at 2.04TB/s (the 80GB PCIe card runs 1.94TB/s). PCIe versions of both cards trade bandwidth and interconnect for standard-server compatibility: the A100 PCIe draws 250W to 300W, and the H100 PCIe drops to 350W with 80GB of HBM2e at 2.0TB/s. Every form factor of both cards supports up to 7 Multi-Instance GPU partitions, so a single card can be sliced into isolated inference instances.

Why Bandwidth Decides Inference

Autoregressive text generation is memory-bandwidth-bound, not compute-bound. On each decode step, the GPU loads the model layer weights and the cached keys and values into the compute cores, performs a relatively small amount of arithmetic, and emits one token. The bottleneck is the data movement, not the math.

This is why the H100's 1.64x bandwidth advantage, not its 3x compute advantage, is the first-order predictor of its inference speedup. The Tensor Cores on an A100 already sit idle waiting for memory during decode; adding more compute headroom does not help a workload that is starved for bandwidth. Widening the memory pipe from 2.04TB/s to 3.35TB/s is what feeds the cores faster and produces more tokens per second.

Two things compete for the 80GB during serving: the model weights, which are fixed in size, and the KV cache, which grows with every token of context and every concurrent request. On the A100 the weights must be stored in at least 1 byte per parameter (INT8) or 2 bytes (BF16), because there is no FP8 path. On the H100, FP8 stores weights at 1 byte with a floating-point format that preserves more accuracy than INT8, which is why the same model leaves more room for KV cache on an H100 than on an A100.

For a Llama 2 7B model in float16 with a 10,000-token context, the KV cache alone needs roughly 5GB, about one-third of the model's half-precision weight storage, per Hugging Face. That number scales with batch size and context length, and it is why the precision advantage compounds: FP8 weights on the H100 free up capacity that becomes KV-cache headroom for longer context and higher batch.

Measured Inference Benchmarks

The marketing multiple for the H100 is up to 4x. Measured LLM inference lands lower and depends heavily on model size, quantization, and batch size. Across independent tests the H100 runs roughly 1.7x to 2.8x faster than the A100, with the top of that range requiring FP8 or W4A8 quantization the A100 cannot match.

The pattern is consistent. At low batch and FP16, where neither card can lean on its quantization hardware, the H100 leads by about 1.7x, roughly its bandwidth ratio plus a little. As batch size rises and the workload can use FP8 or W4A8, the H100 pulls to 2.6x to 3.4x because it is now exploiting a Transformer Engine the A100 does not have. The A100 tops out at INT8 and INT4, which cover fewer models cleanly and cost more accuracy than FP8.

Training Speedups

For training, the H100's FP8 support matters even more than it does for inference, because training throughput is more compute-bound than decode. On Llama 3 70B across an 8-GPU NVLink node, one benchmark reports the A100 in BF16 at about 18,000 tokens/sec, the H100 in BF16 at about 38,000 (2.1x), and the H100 in FP8 at about 62,000 (3.4x).

The BF16-to-BF16 comparison isolates the silicon: the H100 is about 2.1x an A100 on identical precision, driven by its higher Tensor Core throughput and bandwidth. The jump to 3.4x comes entirely from FP8, which the A100 cannot run. For a fine-tune long enough that wall-clock time dominates cost, the H100's FP8 path can make the whole job cheaper even at twice the hourly rate. For a short fine-tune or a small model, the A100's lower price usually wins on total cost.

A100 vs H100 Price in 2026

Both cards fell hard in 2026 as Blackwell B200 volume shipped and multi-year reservations began expiring, returning inventory to the secondary market. The A100 fell faster, because its supply came online while demand shifted toward Hopper, which widened the price gap between the two.

The used A100 80GB at $4,000 to $9,000 is roughly a third to a half the price of a used H100. That discount is the core of the A100 value case: for any workload the A100 can serve without needing FP8 or peak throughput, it delivers the same result for far less capital or hourly spend.

Cloud Rental by Provider

Renting is where the price gap is cleanest to read. Across specialized clouds the A100 80GB runs about $1.10 to $2.00 per GPU-hour and the H100 about $2.00 to $3.00, so the A100 is consistently 35 to 50 percent cheaper. Hyperscalers charge more for both because they bundle networking, support, and reservations, and often require 8-GPU instances.

Spot and preemptible instances undercut these on-demand rates by 30 to 50 percent on both cards, with A100 spot dipping near $0.45 to $1.30 per hour on some clouds. The spread between the cheapest A100 ($1.09) and the most expensive H100 hyperscaler rate ($6.98) is more than 6x, and almost none of that is the silicon. It is the wrapper: dedicated versus multi-tenant, InfiniBand versus none, enterprise SLAs versus none. Match the wrapper to the workload rather than paying for interconnect a single-GPU inference job never uses.

Cost Per Token: The Real Comparison

The speed multiple and the price multiple roughly cancel. If the H100 is 2x the price and 2x the throughput, cost per token is a wash, and the choice comes down to latency and capacity rather than economics. When the H100 is 2.8x faster at 2x the price, it wins on cost per token. When it is only 1.7x faster at 2x the price, the A100 wins.

// Cost per token = hourly_rate / (tokens_per_sec * 3600)
// Using measured throughput and July 2026 rental rates.

// --- High-batch W4A8/INT4 workload (Llama 3 70B, batch 32) ---
const a100_rate = 1.49        // $/GPU-hr (RunPod)
const h100_rate = 1.99        // $/GPU-hr (RunPod)
const a100_tps  = 1400        // tok/s, INT4
const h100_tps  = 4800        // tok/s, W4A8

const a100_cost_per_M = a100_rate / (a100_tps * 3600) * 1e6  // ~$0.296 / M tokens
const h100_cost_per_M = h100_rate / (h100_tps * 3600) * 1e6  // ~$0.115 / M tokens
// -> H100 is ~2.6x cheaper per token here. FP8/W4A8 favors the H100.

// --- Low-batch FP16 workload (70B, ~1.7x speedup) ---
const a100_tps_fp16 = 56
const h100_tps_fp16 = 95
const a100_cpm_fp16 = a100_rate / (a100_tps_fp16 * 3600) * 1e6  // ~$7.39 / M tokens
const h100_cpm_fp16 = h100_rate / (h100_tps_fp16 * 3600) * 1e6  // ~$5.82 / M tokens
// -> H100 ~1.3x cheaper. Gap narrows sharply without quantization.

The takeaway: the H100 wins cost per token when the workload can use its FP8 or W4A8 path, which is most high-throughput production inference. On plain FP16 at low batch, the two cards land close enough that the A100's lower entry price, lower power, and wider availability tip the decision. The raw 1.7x-to-2.8x speed multiple overstates the H100's cost advantage because it ignores that you pay roughly twice as much per hour for it.

When the A100 Is Still the Right Buy

The A100 is not the frontier card, which is exactly why it is the value card. The crossover between the two is roughly the 30B parameter mark and whether the workload needs FP8.

The H100 earns its premium in the opposite cases: models 30B and larger where FP8 lets weights fit in half the memory, latency-sensitive production serving where the 1.7x-to-2.8x speed directly improves user experience, and long training runs where FP8 throughput compounds over many hours. For everything in between, the honest answer is to price both on cost per token for your specific model and pick the cheaper one.

What Fits on Each Card

The rule of thumb for fitting a model on a GPU is two terms: weight storage plus KV cache. Weight storage is parameters multiplied by bytes per weight, and bytes per weight depends on the lowest precision the card supports. This is where the A100's lack of FP8 changes the math against an H100 of identical 80GB capacity.

// Weight storage = parameters * bytes_per_weight
//   BF16/FP16 -> 2 bytes/param   (A100 and H100)
//   INT8      -> 1 byte/param    (A100 floor)
//   FP8       -> 1 byte/param    (H100 only, better accuracy than INT8)

const capacity = 80                       // GB, both 80GB SXM cards

// 70B model
const w_bf16 = 70e9 * 2 / 1e9             // ~140 GB -> needs 2 cards on both
const w_int8 = 70e9 * 1 / 1e9             // ~70 GB  -> fits A100, ~10 GB for KV
const w_fp8  = 70e9 * 1 / 1e9             // ~70 GB  -> fits H100 in FP8, ~10 GB KV
// Same footprint, but FP8 (H100) holds accuracy better than INT8 (A100).

// 30B model
const w30_bf16 = 30e9 * 2 / 1e9          // ~60 GB  -> fits A100 in BF16, ~20 GB KV
// A100 serves this cleanly with no quantization at all.

// 13B model in BF16
const w13_bf16 = 13e9 * 2 / 1e9          // ~26 GB  -> large KV budget on either card

The practical ceilings: a single A100 80GB comfortably serves up to the 30B class in BF16 with room for KV cache, or a 70B model in INT8 with a tight KV budget. A single H100 80GB serves the same 70B model in FP8, which preserves more accuracy than INT8 at the same 1-byte footprint. Above 70B, both cards need multiple GPUs. Across an 8-GPU node (640GB), both scale to the 400B-to-700B class, though the H100's FP8 and higher bandwidth serve those models faster.

If your model needs more than 80GB per card, the comparison shifts to the H200 (141GB) or Blackwell. For the H100's own place in that ladder, see NVIDIA H100. For the precision mechanics that separate these cards, see FP8 Quantization.

Skip the GPU: Serve Open Models Instead

The reason to compare an A100 and an H100 at all is usually to serve an open model. Morph runs a GPU fleet with custom kernels and serves open models directly through one OpenAI-compatible API, which removes the step of buying, powering, and sizing GPUs yourself. The same physics on this page (bandwidth sets tokens per second, capacity sets model size, FP8 makes large models fit) is what that fleet is tuned around, and it is why the served models run on Hopper-class and newer hardware rather than Ampere.

The served lineup spans sizes that map to node types: GLM-5.2 (753B), Qwen 3.5 397B and Qwen 3.6 27B, MiniMax M3, and DeepSeek V4 Flash, alongside the fast-apply model morph-v3-fast at ~10,500 tok/s with speculative decoding, a technique that, like FP8, attacks the memory-bandwidth bottleneck rather than the compute one. See the full model lineup.

If you are choosing between an A100 and an H100 to serve a model, the decision reduces to three questions. Does the model plus its KV cache fit in the VRAM at your target context and batch? Is the bandwidth high enough to hit your tokens-per-second target? And on cost per token, does the H100's speed beat the A100's cheaper hours for your precision? For how serving systems extract throughput from a fixed memory budget, and how to think about the total bill, see LLM Inference Optimization and LLM Cost Optimization.

Frequently Asked Questions

What is the difference between the A100 and H100?

The A100 is a 2020 Ampere GPU; the H100 is a 2022 Hopper GPU. The 80GB variants hold the same capacity, but the H100 SXM has 3.35TB/s of HBM3 bandwidth versus the A100's 2.04TB/s of HBM2e (1.64x), 1,979 FP16/BF16 TFLOPS with sparsity versus 624 (about 3x), and native FP8 through its Transformer Engine, which the A100 lacks. The H100 draws 700W to the A100's 400W. On measured LLM inference the H100 runs roughly 1.7x to 2.8x faster.

Is the H100 worth twice the price of an A100?

It depends on the workload. The H100 runs 1.7x to 2.8x faster on LLM inference but rents for about twice the A100's hourly rate, so cost per token is often close. The H100 wins when you need FP8, serve models 30B and larger, or run high-throughput production inference where its speed lowers total job cost. The A100 wins for models under 30B, LoRA fine-tuning, batch jobs, and budget-sensitive development where speed per hour is not the bottleneck.

How much faster is the H100 than the A100 for LLM inference?

Measured benchmarks put the H100 at roughly 1.7x to 2.8x the A100's inference throughput, depending on model, quantization, and batch size. Hyperstack measured 3,311 tokens/sec on an H100 SXM versus 1,148 on an A100 NVLink (about 2.8x). On Llama 3 70B with TensorRT-LLM at batch 32, an H100 in W4A8 hit about 4,800 tokens/sec against an A100 INT4's 1,400 (about 3.4x). On smaller FP16 workloads the gap narrows to roughly 1.7x.

How much does an A100 cost in 2026?

As of July 2026, a used A100 80GB trades around $4,000 to $9,000 and a used 40GB around $2,000 to $4,000. New 80GB PCIe cards run about $8,000 to $12,000 and SXM about $10,000 to $15,000. Cloud rental is roughly $1.10 to $2.00 per GPU-hour on specialized clouds, about half the H100's rate. A100 prices fell faster than H100 prices as Blackwell B200 fleets ship and reservations expire.

Does the A100 support FP8?

No. FP8 was introduced with the H100's fourth-generation Tensor Cores and Transformer Engine. The A100's precision floor is BF16 or INT8, so its 80GB holds at most a roughly 70B-parameter model in INT8 (1 byte per weight) or a 40B-class model in BF16 (2 bytes), before any KV cache. The lack of FP8 is the main reason large models that fit on an 80GB H100 in FP8 do not fit as cleanly on an A100 of the same capacity.

Is the A100 still worth buying in 2026?

Yes, for the right workload. The A100 has the best price-to-performance ratio for inference of models under 30B, LoRA and full fine-tuning of small models, and medium-scale training where wall-clock time is not critical. Its used price of $4,000 to $9,000 for an 80GB card and cloud rate near $1.10 to $2.00 per hour make it the cost-efficiency leader. The H100 is worth the premium for FP8 workloads, 30B-plus models, and production inference that needs peak throughput.

A100 vs H100: which is better for fine-tuning?

For small models (under 30B) and LoRA fine-tuning, the A100 usually delivers a lower total job cost because it rents for about half the price and the speed gap does not fully close that discount. For 30B-plus models and long fine-tunes running 20-plus hours, the H100's 2x to 3x throughput and FP8 support can make the whole job cheaper despite the higher hourly rate. The crossover is roughly the 30B parameter mark.

Related Resources

• NVIDIA H100: Specs, Price, and How It Compares
• NVIDIA H200: VRAM, Bandwidth, and How It Compares
• Morph Model Lineup
• FP8 Quantization
• LLM Inference Optimization
• LLM Cost Optimization
• GLM-5.2: 753B Open Coding Model
• Fast Apply Model: morph-v3-fast at 10,500 tok/s

Sources

• NVIDIA A100 Tensor Core GPU datasheet (40GB/80GB, HBM2/HBM2e, 1.56/2.04TB/s, 312/624 FP16 TFLOPS, 400W SXM, NVLink 3.0 600GB/s, 54.2B transistors, MIG).
• NVIDIA H100 Tensor Core GPU datasheet (80GB HBM3, 3.35TB/s, 1,979 FP16 and 3,958 FP8 TFLOPS with sparsity, 700W, NVLink 4.0 900GB/s).
• Thunder Compute, "NVIDIA A100 GPU: Full Specs" and "A100 vs H100" (July 2026 spec table, $1.09/$2.19 per GPU-hr, used and DGX pricing).
• Spheron, "NVIDIA A100 vs H100 (2026)" (spec table, Llama 3 70B training/inference benchmarks, on-demand and spot rates).
• Hyperstack, "LLM Inference Benchmark: A100 NVLink vs H100 SXM" (3,311 vs 1,148 tok/s).
• Jarvislabs, "NVIDIA A100 GPU Price in 2026" (used $4k-9k for 80GB, per-provider rental rates, A100-vs-H100 economics).
• Lyceum Technology and IntuitionLabs (fine-tuning crossover at ~30B, H100 secondary-market $15k-20k, Blackwell displacement).
• Hugging Face, LLM inference memory documentation (KV cache ~5GB for Llama 2 7B at 10,000-token context).