Best GPU for LLM Training: H100 vs A100 Cost Guide (2026)

An H100 SXM5 costs about 2.75x more per hour than an A100 80GB SXM4 on Spheron's marketplace right now. That alone tells you nothing about which is the best GPU for LLM training. The question that actually matters is whether the H100's real throughput advantage, at your model size and precision, beats that price premium once you multiply speed by hourly rate. Sometimes it does by a wide margin. Sometimes, without FP8 turned on, it doesn't beat it at all. This guide works through the spec differences that drive the gap, live throughput benchmarks for training and inference, and the actual cost math so you can pick a GPU instead of a marketing claim.

If you want the full spec sheet and precision-by-precision TFLOPS breakdown, we cover that in depth in NVIDIA A100 vs H100: specs, benchmarks, and pricing. This post stays focused on the training and fine-tuning cost decision.

What "Best GPU for LLM Training" Actually Means

There isn't one best GPU for LLM training. There's a best GPU for your model size, your precision, and your budget. A100 wins on cost for small-to-mid models and legacy CUDA stacks. H100 wins on cost and speed once FP8 is in the pipeline and the model is large enough to keep the extra compute busy. The table below is the starting point; the sections after it show where that trade-off actually lands in dollars.

Pricing fluctuates based on GPU availability. The prices above are based on 12 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Spec Differences That Matter: Memory Bandwidth, FP8, and NVLink

Three architectural differences explain almost all of the performance gap between these two GPUs. Everything else on the spec sheet is secondary.

HBM3 vs HBM2e Bandwidth and Why It Drives Decode Speed

H100 SXM ships with 3.35 TB/s of HBM3 bandwidth. A100 SXM tops out at 2.0 TB/s of HBM2e, roughly 1.6-1.7x less. During autoregressive decoding, the GPU has to stream the entire model's weights (or a large fraction of them, under quantization) from VRAM for every single generated token. That makes decode a memory-bandwidth-bound operation, not a compute-bound one, which is exactly where H100's bandwidth advantage shows up as real tokens-per-second and not just a bigger number on a spec sheet.

The Transformer Engine and FP8 (H100-Only)

This is the single biggest lever in this whole comparison. H100's Transformer Engine executes attention and feed-forward layers natively in FP8 (the E4M3 and E5M2 formats), and per-SM the Tensor Cores run FP8 matrix math at 4x the rate of A100's best 16-bit format, per NVIDIA's Hopper architecture deep dive. A100 has no FP8 hardware support at all; BF16 is its practical ceiling. If your fine-tuning or serving stack doesn't enable FP8, you're paying for H100 silicon you aren't using. If it does, the throughput gap widens dramatically, which is the crux of the cost math in the next two sections.

NVLink 4.0 vs 3.0 for Multi-GPU Training Scaling

For any training job spanning more than one GPU, NVLink bandwidth determines how much time is lost to gradient synchronization between steps. H100 SXM ships 900 GB/s of NVLink 4.0 bidirectional bandwidth across 18 links at 50 GB/s each. A100 SXM ships 600 GB/s of NVLink 3.0 across 12 links. That 50% increase reduces all-reduce stalls in 8-GPU training runs, which matters more as parameter count and gradient volume grow. For clusters spanning multiple nodes rather than a single 8-GPU box, the interconnect story gets more complicated; our FSDP, DeepSpeed ZeRO-3, and Megatron-Core multi-node setup guide covers NCCL tuning for that case specifically.

Inference Throughput and Cost-Per-Token: H100 vs A100

NVIDIA's TensorRT-LLM benchmark suite reports H100 FP8 delivering up to 4.6x max throughput and 4.4x faster first-token latency than A100 across its tested LLM serving scenarios. Their published GPT-J 6B numbers give a concrete, checkable data point: at batch size 64, H100 FP8 hits 10,907 tokens/sec against A100 FP16's 3,679 tokens/sec, a 3.0x gap. At batch size 1, H100 FP8's first-token latency is 7.1ms versus A100 FP16's 12.5ms, about 1.76x faster on that specific measurement.

Run those batch-64 numbers through live Spheron pricing and the cost-per-token math gets interesting:

H100 FP8 comes out marginally cheaper per token here, but only because its 3.0x throughput edge just clears the 2.75x price gap. Drop the batch size, or run A100 in BF16 without quantization, and the ratio flips toward A100. This is the pattern to watch for across your own benchmarks: on-demand cost-per-token is a race between throughput ratio and price ratio, and the two are close enough on these two GPUs that a small workload change can swing the winner. For a broader cross-model, cross-GPU cost-per-token dataset covering Llama, Qwen, and DeepSeek, see our GPU cost-per-token benchmark. Serving configuration matters as much as hardware here: our continuous batching and PagedAttention guide walks through how batch size changes this math in vLLM specifically.

Independent third-party benchmarks land in the same neighborhood. SemiAnalysis's InferenceX project measures H100 serving GPT-OSS-120B at around $0.13 per million tokens at 117 tokens/sec per user, rising to $0.44 per million tokens at higher per-user interactivity (216 tok/s/user), a useful outside check that these figures aren't just vendor marketing and a reminder that cost-per-token climbs fast once you demand more speed per concurrent user.

Fine-Tuning Speed and Cost by Model Size (7B, 13B, 70B)

Model size changes which GPU actually makes financial sense, mostly because it changes whether you're bottlenecked on GPU hours or on hourly rate.

LoRA/QLoRA Single-GPU Runs

A 7B or 13B LoRA or QLoRA fine-tune typically fits on a single GPU with room to spare; our VRAM sizing guide for full, LoRA, and QLoRA fine-tuning has the exact memory math by model size. At this scale, jobs are usually short enough (minutes to a few hours) that the GPU's hourly rate matters more than its peak throughput. A100 on-demand at $1.82/hr against H100 at $5.01/hr means the H100 needs to be at least 2.75x faster just to break even on total cost, and a small LoRA job on a single GPU rarely stresses either card enough to see that gap. For most 7B/13B fine-tunes, A100 is the better buy. Framework choice matters here too: Axolotl vs Unsloth vs TorchTune covers which stack extracts the most throughput per GPU at this scale, and our general fine-tuning walkthrough covers setup end to end.

Multi-GPU Full Fine-Tuning at 70B+

This is where the H100's price premium either pays for itself or doesn't, and the answer hinges entirely on FP8. Using the reference throughput ratios from our full A100 vs H100 benchmark for Llama 3 70B LoRA fine-tuning on 8-GPU NVLink clusters (18,000 tok/s on A100 BF16, 38,000 tok/s on H100 BF16, 62,000 tok/s on H100 FP8), here's what a fixed 10-billion-token fine-tuning job actually costs at current Spheron pricing:

That middle row is the trap. An 8-GPU H100 cluster running plain BF16 finishes the job in half the time of A100 but costs 30% more in total dollars, because the 2.1x speedup doesn't clear the 2.75x hourly premium. Only when FP8 is actually enabled does the H100 cluster pull ahead: 20% cheaper than A100 and 3.4x faster. If your team migrates to H100 for a training job and doesn't turn on the Transformer Engine, you're paying for hardware you aren't using and getting a worse total bill for it. For multi-node setups beyond a single 8-GPU box, see the distributed training guide for NCCL and sharding configuration that keeps this math from degrading further at scale.

Hourly Rental Price Gap Between H100 and A100

H100 comes in three form factors with meaningfully different price points, and the gap to A100 changes depending on which one you're comparing against.

Pricing fluctuates based on GPU availability. The prices above are based on 12 Aug 2026 and may have changed. Check current GPU pricing → for live rates.

Note how much closer H100 PCIe and H100 NVL price to A100 than H100 SXM5 does. If your workload doesn't need SXM's NVLink bandwidth (single-GPU inference, for instance), pricing an H100 PCIe against an A100 SXM4 shrinks the premium to under 1.5x instead of 2.75x, which changes the throughput bar H100 needs to clear. Our H100 NVL vs SXM5 vs PCIe form factor guide breaks down which variant fits which workload.

Decision Framework: Picking the Best GPU for LLM Training by Use Case

• 7B-13B LoRA/QLoRA fine-tunes where the job finishes in hours, not days, and the GPU's hourly rate outweighs its throughput ceiling.
• BF16-only pipelines where FP8 isn't wired up yet. Without the Transformer Engine active, H100 loses its main structural advantage and the price gap does the rest.
• Legacy CUDA stacks (CUDA 11.x, older PyTorch or TensorFlow builds) that don't have Hopper-tuned kernels.
• Spot-tolerant batch workloads where A100 spot at roughly $1.15/hr produces the lowest cost-per-token of any option here, as long as preemption is acceptable.
• Multi-tenant serving with MIG partitioning where you need the lowest cost per isolated instance rather than the highest peak throughput.

H100 is the right call when FP8 is enabled and actually exercised, when you're pretraining or continuing pretraining at 70B+ scale, when you're serving at consistent high concurrency where the throughput ceiling is the binding constraint, or when wall-clock time itself has a cost, shipping a model two days later than a competitor. If your models are outgrowing 80GB entirely, it's worth checking whether H200's larger HBM3e capacity changes the calculus before committing to another H100 generation.

Renting H100 and A100 on Spheron

Both GPUs are available on Spheron with per-minute billing and no contract lock-in, so the cheapest way to answer "which GPU wins for my job" is to run the actual workload on both for 30 minutes and compare measured cost-per-token or wall-clock training time, not to trust a spec sheet. Provisioning steps and API details for both GPU types are in the Spheron docs.

Current pricing has A100 GPU rental starting around $1.82/hr on-demand and $1.15/hr on spot, while H100 GPU rental on Spheron starts around $2.65/hr on-demand for the PCIe variant and $5.01/hr for SXM5. Since rates shift with regional supply and demand, check the live pricing page before budgeting a training run.

Whether your next training run needs A100's lower hourly floor or H100's FP8 throughput, both are available on Spheron with per-minute billing and no reserved commitment.

H100 GPU pricing → | Check A100 availability → | View all pricing →