Two claims about FP8 circulate constantly and neither one survives contact with the actual eval data. The first is that FP8 quietly wrecks accuracy on smaller models. The second is that switching to FP8 doubles your throughput. We pulled the benchmark papers that actually measured both, and the real picture is closer to the opposite of the folklore: the accuracy case for FP8 vs BF16 is stronger than people assume, and the throughput case is weaker.
If you want the format mechanics and hardware breakdown first, start with our FP8 quantization explainer. This post picks up where that one stops: what you actually lose, measured, when you move a production model from BF16 to FP8.
What Changes When You Drop From 16 Bits to 8
BF16 gives every value 1 sign bit, 8 exponent bits, and 7 mantissa bits, which is why it trained the last several generations of frontier models without numerical drama: the exponent range matches FP32's, so gradients and activations rarely overflow or underflow. FP8 in its dominant inference format, E4M3, compresses that down to 1 sign bit, 4 exponent bits, and 3 mantissa bits. You lose both range and precision, but not equally, and the ratio of what you keep versus what you lose is the whole story of why FP8 works as well as it does.
E4M3 vs BF16: Precision vs Range Tradeoff
E4M3's 4 exponent bits cap its dynamic range at roughly 448 in magnitude, against BF16's range in the tens of thousands. That sounds disqualifying until you notice what actually varies inside a trained transformer: weight and activation tensors, once you're past training and into a converged, calibrated model, cluster in a narrow band per layer. The problem FP8 has to solve isn't spanning ten orders of magnitude, it's representing values that already sit close together with enough of its 3 mantissa bits to distinguish them. Per-tensor scaling handles the rest: instead of one fixed FP8 range for the whole model, each tensor gets its own scale factor so its actual value distribution lands inside E4M3's usable window. That's the mechanical reason FP8 recovers as much accuracy as it does, and it's also why the myth that FP8 "just doesn't have enough bits" undersells what per-tensor scaling buys back.
Hardware Support: Which GPUs Run FP8 Natively
This matters for the accuracy conversation specifically because your hardware decides which alternative you're actually comparing FP8 against. If your GPU has no FP8 path, "FP8 vs BF16 accuracy" isn't your real choice: it's BF16 vs INT8, and INT8 is the format with the bigger accuracy penalty.
Hopper and Blackwell Have Dedicated FP8 Tensor Cores, Ampere Doesn't
The short version, covered in full in our FP8 quantization explainer: NVIDIA Hopper (H100, H200, GH200) and Blackwell (B200, B300, RTX 5090, RTX PRO 6000) ship dedicated FP8 Tensor Cores with hardware per-tensor scaling built in. Ampere (A100) has no hardware FP8 path at all; INT8 is as low as its Tensor Cores go. Ada Lovelace (RTX 4090, L40S) sits in between, with CUDA-level FP8 ops but no dedicated scaling hardware, so its throughput gains fall well short of Hopper's.
The accuracy consequence of that Ampere gap shows up directly: being on A100 doesn't just mean slower FP8, it means no FP8 at all, and the fallback format costs real accuracy.
FP8 vs BF16 Accuracy: The Delta We Can Actually Measure, Across Model Sizes
A direct answer before the detail: across the model sizes that have actually been benchmarked against a full-precision baseline, FP8 (E4M3) accuracy loss doesn't scale with parameter count the way the folklore assumes. A 176-billion-parameter model and a 7-billion-parameter model from the same family can lose almost identical amounts of accuracy under FP8, and a 65-billion-parameter model can come out of quantization unchanged or slightly better. That's the opposite of the "FP8 quietly wrecks small models" story.
Large Models: The Myth Doesn't Hold Up
If FP8 broke down at scale, the largest models tested would show it first. They don't.
Small Models: The Same Story, Not a Cliff
Here's the part of the folklore that has real numbers behind it, just not the ones the story predicts.
A more recent, reasoning-focused study, InfiR2, ran a 1.5-billion-parameter Qwen2.5 model through a full FP8 training pipeline and tested it on AIME 2025 math, GPQA reasoning, and LiveCodeBench code generation. The smallest model in this entire comparison didn't show a predictable, size-driven accuracy cliff. It showed noise, in both directions, same as everything else.
Task Type Matters More Than Model Size
This is the axis the size-based folklore misses entirely.
| Model | Format | MMLU score | Recovery vs FP16 |
|---|---|---|---|
| Llama 3 8B | FP16 | 0.654 | baseline |
| Llama 3 8B | FP8 | 0.649 | 99.2% |
| Llama 3 8B | INT8 (SmoothQuant) | 0.629 | 96.2% |
| Llama 3 70B | FP16 | 0.790 | baseline |
| Llama 3 70B | FP8 | 0.787 | 99.6% |
| Llama 3 70B | INT8 (SmoothQuant) | 0.772 | 97.7% |
The same domain-over-size pattern shows up inside a single model, at a single size. On the 1.5B Qwen2.5 reasoning model above, math (AIME 25) and code generation (LiveCodeBench) held flat or improved under FP8, while GPQA, a knowledge-and-reasoning benchmark, gave up 2.46 points, the largest single move in that comparison and bigger than anything the model-size data produced.
When the FP8 Tradeoff Is Worth It, and When It Isn't
FP8 is close to a free lunch when: you're running an established LLM inference workload, whether that's an 8B model or a 176B model, on Hopper or Blackwell hardware where per-tensor scaling runs in dedicated silicon rather than emulated in CUDA. The size-based accuracy data above doesn't give you a reason to reach for BF16 at either end of that range.
FP8 needs real validation when: a meaningful share of your traffic depends on knowledge-heavy reasoning rather than math or code generation. GPQA showed the largest single drop in the data above; math and code held flat or improved on the same model. Validate against your own eval set before shipping, especially if your workload skews toward the task type that moved the most.
FP8 isn't your real choice at all if: you're on Ampere hardware. There's no dedicated FP8 Tensor Core path on A100, so the fallback is INT8, and NVIDIA's own Llama 3 benchmarking above shows INT8 (via SmoothQuant) trailing FP8 by roughly 2-3 accuracy points of recovery on the same models. Getting onto Hopper or Blackwell is what makes the FP8 accuracy case above apply to you at all, not a nice-to-have on top of it.
Cost Per Token: FP8 vs BF16 on the Same GPU
This is the part where the folklore flips from underselling FP8 to overselling it.
Production serving rarely delivers that ratio end to end. A separate 2024 benchmarking study from Argonne National Laboratory, evaluating LLaMA, Mistral, and Qwen models across NVIDIA, AMD, and Intel accelerators, found the same shape of result: "FP8 on H100 and Int8 on A100 can provide performance benefit compared to FP16," but the paper is explicit that hardware without a native FP8 path, like A100, "limits the framework's ability to leverage low precision for weights and KV cache" in the first place, which is the more fundamental gate than the raw throughput number. The decode phase that dominates inference latency runs thin, memory-bound matrix-vector operations rather than the large, compute-saturating matmuls that peak TFLOPS numbers are measured against, and that gap is why the realized gain lands well under the spec-sheet ratio.
There's a second benefit that doesn't depend on how much of that throughput number you actually capture: FP8 stores each weight and activation value in 8 bits instead of BF16's 16, so the KV cache for a given sequence length is half the size before you've measured a single token per second. That's not a benchmark result to validate, it's what 8 bits versus 16 bits means, and it holds even on hardware whose FP8 compute gains are modest, because it's a memory constraint, not a compute one, and it's what determines how many concurrent sequences you can fit in GPU memory during decoding.
Spheron's own catalog maps directly onto the hardware-gating point above: H100 on Spheron, H200, and B200 instances all carry native FP8 Tensor Cores, while A100 is in the same catalog without hardware FP8, so you can see the on-demand and spot rate for each side of that gap in one place rather than guessing at it. As of 02 Sep 2026, on-demand H100 runs $2.65/hr (spot $2.07/hr) and A100 80GB runs $1.43/hr on-demand (spot $1.19/hr) on Spheron. Prices move with availability and region, so treat those as a snapshot and check current GPU pricing → before you budget against them. Because Spheron bills per minute with no minimum commitment, running your own BF16-vs-FP8 A/B test on an eval set that matters to you costs less than an hour of GPU time rather than a procurement cycle.
Pricing fluctuates based on GPU availability. The prices above are based on 02 Sep 2026 and may have changed. Check current GPU pricing → for live rates.
What Spheron's own pages don't carry is the accuracy or throughput benchmark data itself, that has to come from papers like the ones cited above, not from a GPU rental marketplace's product page. If what you actually need is FP4 rather than FP8, or a fully managed inference API instead of raw GPU access, that's a narrower need than this post covers; see our NVFP4 vs MXFP4 guide for the 4-bit path.
For the applied side of this decision, the Transformer Engine setup guide covers the installation and code for running FP8 on H100/H200, the ModelOpt quantization guide covers the tooling for producing a calibrated FP8 checkpoint, and A100 vs H100 and L40S vs A100 both go deeper on what the FP8 hardware gap costs you on those specific GPU pairs. If you're choosing hardware from scratch rather than migrating an existing deployment, our GPU inference decision guide walks through the FP8-vs-FP4 tradeoff by workload rather than by GPU model.
The accuracy data says FP8 is close to a rounding error on most production workloads, and the hardware you pick decides whether you even get to make that trade. H100 and H200 run FP8 natively; A100 doesn't.
Frequently Asked Questions
Barely, at the model sizes that have actually been benchmarked against a full-precision baseline. In the MLSys FP8 Formats for Deep Learning study, BLOOM-176B lost 0.30% of its FP32 accuracy on the Lambada benchmark under E4M3 FP8, and LLaMA-65B scored slightly higher in FP8 than in FP32. NVIDIA's own post-training quantization testing found Llama 3 70B recovered 99.6% of its FP16 MMLU score in FP8. The gap that does exist tracks what the model is doing more than how big it is.
Not in any consistent way. In the same FP8 Formats study, BLOOM-7B1 lost 0.28% of its FP32 Lambada accuracy under FP8, statistically the same as the 0.30% drop measured on BLOOM-176B, a model 25 times larger. NVIDIA measured a similar small gap between Llama 3 8B (99.2% MMLU recovery) and Llama 3 70B (99.6%). Parameter count doesn't predict FP8 degradation the way the small-model folklore assumes.
Not in practice. NVIDIA's H100 datasheet lists FP8 Tensor Core throughput at exactly double BF16's, 3,958 versus 1,979 teraFLOPS with sparsity. But NVIDIA's own benchmarking of Llama 3 on that same H100 hardware measured real speedups of 1.26x to 1.84x depending on model size and batch size, because production decoding runs thinner, more memory-bound matrix operations than the dense matmuls the spec-sheet number is measured against.
NVIDIA Hopper (H100, H200, GH200) and Blackwell (B200, B300, RTX 5090, RTX PRO 6000) have dedicated FP8 Tensor Cores with hardware per-tensor scaling. Ampere (A100) has no hardware FP8 path at all and is limited to BF16 or INT8. Ada Lovelace (RTX 4090, L40S) has partial, CUDA-level FP8 support without the dedicated scaling hardware, so it doesn't see Hopper-level throughput gains.
