GPU Cloud Benchmarks 2026: Pricing, Specs, Throughput

Choosing a GPU for your AI workload shouldn't require a spreadsheet and three hours of tab-hopping across provider websites. But that's exactly what it takes today, because no provider publishes honest side-by-side comparisons. Each one highlights the metric where they win and buries the rest. This guide helps teams navigate the landscape; for specific selection guidance, see our top 10 cloud GPU providers analysis.

This post fixes that. We compiled real specs, current pricing, and published inference throughput data across the GPUs and providers that matter most in 2026. No marketing spin, just the numbers you need to make an informed decision. For a workload-based decision guide that maps these GPUs to specific inference scenarios, see best GPU for AI inference in 2026.

For a provider-by-provider pricing breakdown across all GPU models, see our GPU cloud pricing comparison for 2026.

The Hardware Landscape: What You're Actually Choosing Between

Before comparing providers, you need to understand what separates the GPUs themselves. The differences aren't just about raw compute power; memory capacity, memory bandwidth, and interconnect speed all determine whether a GPU is right for your workload.

GPU Specifications at a Glance

FP16 and FP8 Tensor Core values are dense (non-sparse). With NVIDIA 2:4 structured sparsity: FP16: A100 ~624, H100/H200 ~1,979, L40S ~733, RTX 4090 ~330 TFLOPS; FP8: H100/H200 ~3,958, L40S ~1,466, B200 ~9,000, RTX 4090 ~1,320 TFLOPS.

¹ A100 (Ampere architecture) has no native FP8 hardware support. FP8 Tensor Cores were introduced with Hopper (H100/H200) and Ada Lovelace (L40S, RTX 4090) architectures.

A few things jump out from this table.

Memory bandwidth is the inference bottleneck. For LLM inference, the speed at which you can read model weights from memory matters more than raw FLOPS. The H200's 4.8 TB/s bandwidth is why it outperforms the H100 on inference despite having identical compute; it can feed the tensor cores faster. The B200 doubles that again to 8 TB/s. Teams making this choice should read our GPU cost optimization playbook for detailed TCO analysis.

VRAM determines what you can run. The H200's 141 GB lets you run 70B parameter models in FP16 on a single GPU. The A100 at 80 GB requires quantization or multi-GPU setups for the same model. The B200's 192 GB opens the door to running 100B+ models without sharding.

The L40S is the sleeper pick for inference. Its 1,466 FP8 TFLOPS (Ada Lovelace FP8 Tensor Cores) makes it surprisingly competitive for inference workloads that can use FP8 precision, at a fraction of the cost of an H100.

Pricing Across Providers: The Real Numbers

GPU pricing varies enormously depending on which provider you use, whether you commit to a reservation, and whether you're willing to use spot instances. Here's what the market looks like right now.

H100 SXM Pricing (per GPU, per hour)

The spread is significant: the cheapest H100 on the marketplace is less than half the price of AWS. That's the same GPU, the same CUDA cores, the same 80 GB of HBM3; just a different billing address.

Two years ago, H100s commanded $8/hr on-demand. The market has commoditized rapidly as supply caught up with demand. Current pricing has stabilized around $2.85-$3.50/hr at most providers, with marketplace rates dipping below $2/hr.

H200 SXM Pricing

The H200 carries a 15-20% premium over the H100, which is justified by its 45% inference performance improvement. For latency-sensitive inference, the cost-per-token math often favors the H200 despite the higher hourly rate.

One catch: hyperscalers currently sell H200s only in 8-GPU bundles. If you need a single H200, you're limited to specialized providers.

A100 80GB Pricing

The A100 is in its sunset phase for premium workloads, and pricing reflects that. At under $1/hr, it's become the budget option and for many workloads, it's still plenty of GPU. Fine-tuning models under 13B parameters, batch inference, and lighter training jobs don't need H100-class hardware.

L40S Pricing

The L40S sits in a compelling price-performance pocket for inference. At $0.40-$0.86/hr with 48 GB VRAM and strong FP8 throughput from its Ada Lovelace Tensor Cores, it handles most production inference workloads at a fraction of H100 pricing.

For full L40S benchmark data and workload guidance, see our L40S for AI inference guide.

B200 Pricing

B200 availability has improved significantly since early 2026, with multiple providers now offering on-demand instances. Pricing has started to stabilize at roughly 20-30% above H200 levels, though rates continue to drop as supply ramps. The B300 (Blackwell Ultra) is also now available at select providers, starting from $2.90/hr on Spheron for spot instances. See our B300 guide for full pricing details.

Inference Throughput: Where It Actually Matters

Raw specs and pricing only tell half the story. What matters is how many tokens per second you get for your dollar. Here's published benchmark data from MLPerf inference benchmarks on Llama 70B.

Llama 70B Inference Throughput

Methodology note: The A100 figure is a single-GPU result running Llama 2 70B with INT8 quantization (the model barely fits in 80 GB quantized). The H100/H200 figures are from MLPerf Inference v4.0 8-GPU server submissions using TensorRT-LLM with FP8. These are not equivalent configurations; comparing 1× A100 to 8× H100 inflates the ratio. On a per-GPU basis with the same precision, H100 SXM outperforms A100 by roughly 5-8× for Llama 2 70B throughput. The multi-GPU aggregate numbers are shown here because they reflect real deployed configurations. The $/hr column shows the full 8-GPU node rate (8 × per-GPU price); cost-per-million-tokens divides that node rate by the aggregate throughput in tokens per hour.

For a breakdown of the latest MLPerf Inference v6.0 scores and what they mean for cloud GPU selection, see our MLPerf v6.0 results guide.

For smaller models (7B-13B parameters), the gap narrows. An A100 runs Llama 3.3 8B at roughly 400-500 tok/s, while an H100 pushes 10,000+ tok/s with TensorRT-LLM. The cost-per-token advantage of newer hardware remains, but the absolute throughput is less dramatic.

Practical Inference Scenarios

Chatbot / real-time API (latency-sensitive): H200 is the best choice. The ~42% throughput improvement over H100 directly translates to lower latency at high concurrency. The 141 GB VRAM also lets you run larger models without quantization, preserving output quality. Understand the tradeoffs in our detailed nvidia-h100-vs-h200 comparison.

Batch inference / offline processing: H100 at marketplace pricing ($1.49-$2.10/hr) offers the best cost-per-token. Latency doesn't matter for batch jobs, so you're purely optimizing for throughput per dollar.

Budget inference / low-traffic endpoints: L40S or A100. If your model fits in 48 GB (L40S) or 80 GB (A100), you can serve inference at $0.40-$0.86/hr. For startups running a single model endpoint with moderate traffic, this is 3-5x cheaper than an H100 and more than sufficient.

Maximum throughput (large-scale serving): B200 when pricing normalizes. The 8 TB/s bandwidth and native FP4 support promise 11-15x throughput improvements over Hopper-generation GPUs. If you're serving millions of requests per day, the B200's higher hourly cost is offset by dramatically fewer GPUs needed.

Training Performance: How Much Faster Is Your Model Ready?

Training benchmarks are harder to standardize because they depend heavily on model architecture, batch size, precision, and optimization framework. But the relative performance between GPU generations is consistent.

Training Speedup by Generation

The H100's 2.4x training speedup over A100 translates directly to cost savings: a training run that takes 10 days on A100s finishes in roughly 4 days on H100s. Even though H100s cost 3x more per hour, the total training cost is lower because you're paying for fewer hours.

For distributed training across multiple nodes, interconnect matters enormously. NVLink bandwidth determines how fast gradients synchronize between GPUs. The B200's NVLink 5 at 1.8 TB/s per GPU is 2x the H100's NVLink 4; this means less time waiting for communication and more time computing.

The Decision Framework

With all this data, here's the practical framework for choosing a GPU and provider.

Step 1: What's your workload?

• Inference (real-time) → Optimize for latency. H200 or H100.
• Inference (batch) → Optimize for cost-per-token. H100 marketplace or L40S.
• Training (fine-tuning) → Optimize for VRAM + cost. A100 or H100 depending on model size.
• Training (pre-training) → Optimize for sustained throughput. H100/H200 clusters with InfiniBand.

Step 2: What's your budget tolerance?

• Minimum cost, flexible on availability → GPU marketplaces and spot instances.
• Predictable pricing, guaranteed availability → Reserved instances from mid-tier providers.
• Maximum reliability, budget secondary → Hyperscalers (AWS, GCP, Azure).

Step 3: Compare cost-per-output, not cost-per-hour.

A GPU that costs 2x more per hour but delivers 3x the throughput is 33% cheaper on a per-output basis. Always calculate cost per million tokens (inference) or cost per training epoch (training), not just the hourly rate. For a detailed token factory benchmarks table with live Spheron pricing across A100, H100, H200, and B200, see the token factory benchmarks guide.

What We'd Recommend

For teams evaluating GPU cloud options in 2026, the market has shifted in your favor. H100 pricing has dropped 60% from its peak, A100s are available under $1/hr, and the L40S offers a genuine budget alternative for inference. Learn which GPU suits your specific model by consulting our best NVIDIA GPUs for LLMs guide and checking current pricing.

The biggest remaining challenge isn't finding cheap GPUs; it's comparing across providers without spending a full day on it. Spheron aggregates pricing across 5+ providers into a single interface, making it practical to consistently deploy on the cheapest available option for your workload. Whether you need spot H100s for a training run or reserved L40S instances for a production endpoint, you're choosing from the full market instead of one provider's pricing.

We'll update this benchmark data quarterly as new hardware ships and pricing evolves. Bookmark this page if you want the latest numbers without the research.

Spheron aggregates GPU pricing from multiple providers into a single interface, so you can deploy on the cheapest available option for your workload. H100s from $1.33/hr, A100s from $0.76/hr, no contracts.

Compare GPU pricing on Spheron →