"They're both 16-bit floats, just pick whichever your framework defaults to." That line has talked more than one team into an FP16 training run that diverged somewhere past the halfway mark with no way back. Bfloat16 and float16 use the same 16 bits, but they spend them on different things, and the difference shows up exactly where a training run is most fragile: in how far a gradient can grow before the number representing it stops meaning anything.
TL;DR: Bfloat16 vs Float16, What Actually Changes
- Bit layout: BF16 spends its 16 bits as 1 sign, 8 exponent, 7 mantissa bits; FP16 spends them as 1 sign, 5 exponent, 10 mantissa bits.
- Range: BF16 matches FP32's exponent range (up to roughly 3×10^38); FP16 tops out near 65,504, so large gradients overflow to infinity.
- Evidence: BLOOM's team hit irreversible divergence training a 104B-parameter model in FP16 on V100, then trained cleanly in BF16 on A100.
- Corroboration: OPT-175B, trained in FP16 with loss scaling, logged repeated crashes where its dynamic loss scalar dropped to 0 before recovery via checkpoint restarts.
- Hardware gate: BF16 needs Ampere or newer. Every GPU in Spheron's rental catalog, from A100 up, supports it natively; V100 never did.
The Belief: "They're Both 16-bit, Pick Whichever"
The confusion is understandable, because for years half precision meant one thing: IEEE 754 float16, the format V100's first-generation Tensor Cores were built around. When Ampere shipped a second 16-bit option, plenty of training scripts just added a bf16: true flag next to the existing fp16: true one and treated them as swappable defaults, a coin flip decided by whichever the framework happened to expose first.
The name reinforces the mistake. Both formats occupy 16 bits in memory, both get called "half precision" in casual conversation, and both show up in the same PyTorch autocast context manager. Nothing about the label tells you that one of them is a demotion from FP32's dynamic range and the other one isn't. You find that out only by asking where the 16 bits actually went, or by running a large enough model that it finds out for you.
Bfloat16 vs Float16: What Actually Differs Under the Hood
A 16-bit float is a fixed budget split three ways: a sign bit, exponent bits that set how large or small a number can get, and mantissa bits that set how finely two nearby numbers can be told apart. BF16 and FP16 spend that budget on opposite priorities, and the split is the entire story.
BF16's Bit Layout: 8 Exponent Bits, 7 Mantissa Bits
Bfloat16 uses 1 sign bit, 8 exponent bits, and 7 mantissa bits, the same exponent field width as full 32-bit float, according to Intel, Google, and Facebook's joint paper introducing the format for deep learning. That's the whole design choice: BF16 is FP32 with the mantissa chopped down to 7 bits, keeping FP32's dynamic range untouched. Coarser precision within a range is a rounding error the optimizer averages out over millions of steps. Values that fall outside the representable range aren't roundable at all; they become infinity, NaN, or zero, and the training step that produces one of those doesn't recover on its own.
FP16's Bit Layout: 5 Exponent Bits, 10 Mantissa Bits
IEEE half-precision float16 uses 1 sign bit, 5 exponent bits, and 10 mantissa bits, per the format's own specification. Ten mantissa bits genuinely buys more precision than BF16's seven, which is why FP16 remains a solid choice for plenty of inference workloads where the value range is already known and bounded. But five exponent bits caps the largest representable magnitude at 65,504. That's not a rounding boundary, it's a hard wall: a gradient or activation that would need a 6-digit exponent in FP32 simply cannot exist in FP16, no matter how much mantissa precision is available to represent it once it's inside range.
Why This Split Matters for Gradients, Not Just Weights
Model weights, once a network has converged, tend to cluster in a fairly narrow, well-behaved range, which is part of why post-training quantization to even lower precision than either of these formats can work at all. Gradients during training don't behave that politely. Early layers, embedding tables, and loss spikes from a bad batch can all produce values that swing across many orders of magnitude within the same training step, and the backward pass is exactly where FP16's narrow exponent field runs out of room first. A weight rounding error costs you a small amount of precision. A gradient overflow costs you the update, and if it happens enough times in the same direction, it costs you the run.
The Measurement: BLOOM's 104B Run Diverged in FP16, Then Trained Clean in BF16
This isn't a difference you have to take on theory. The BigScience team that built BLOOM ran the same architecture at large scale on both formats, on two different Tensor Core generations, and documented what happened at each step of getting there.
What Happened on V100 in Float16
In BLOOM's preliminary experiments at the 104B-parameter scale, the team observed numerical instabilities that caused irreversible training divergences, which they attributed to float16's limited dynamic range producing overflows, according to the BLOOM paper itself. These weren't recoverable dips in the loss curve. Once the run diverged in FP16, it didn't come back on its own, and continuing to train past that point wasted compute on a model that was no longer learning anything useful. Those early experiments ran on V100, the GPU generation whose Tensor Cores only ever supported FP16, so bfloat16 wasn't an option to reach for until the hardware changed too. A100's third-generation Tensor Cores deliver 312 TFLOPS of FP16 tensor throughput on the same silicon as its BF16 path, against V100's first-generation Tensor Cores at 125 TFLOPS FP16 with no BF16 path available at any throughput.
What Changed on A100 in Bfloat16
The A100 GPUs BLOOM ultimately trained the full 176B-parameter model on support bfloat16, which carries the same dynamic range as float32, and switching final training to bfloat16 mixed precision resolved the instability problem the team had hit at 104B parameters, per the same paper. Nothing else about the recipe needed a rescue: no manual loss-scale tuning, no gradient-clipping schedule invented specifically to route around overflow. The format change alone removed the failure mode, because the values that had been overflowing FP16's exponent field simply fit inside BF16's from the start. It's the cleanest version of a controlled experiment you'll find in a training paper: same team, same target architecture, one format swapped for another, and the instability disappears with it.
A Second Data Point: OPT-175B's Loss Scalar Crashing to Zero
BLOOM's result isn't a one-off. Meta's OPT-175B trained its model weights in FP16 (with optimizer state kept in FP32) and relied on dynamic loss scaling to avoid gradient underflow, and the training team found a direct correlation between loss divergence, their dynamic loss scalar crashing to 0, and a spike in the L2-norm of the final layer's activations, according to the OPT paper. Loss scaling is the standard FP16 workaround: multiply the loss by a large factor before the backward pass so small gradients don't underflow to zero, then divide the scale back out before the optimizer step actually applies them, which is exactly what PyTorch's `GradScaler` automates for `torch.float16` autocast. It buys headroom on the underflow side, but it does nothing for the overflow side, and OPT's own logbook shows the trade wasn't free: the team logged at least 35 manual restarts and an estimated 70-plus automatic restarts from hardware failures over roughly two months, with the loss-divergence pattern tracked specifically to that scalar collapsing. Recovery meant restarting from an earlier checkpoint where the scalar was still healthy (at or above 1.0) and lowering the learning rate, and the team also had to lower gradient clipping from 1.0 to 0.3 to help stability going forward.
That's two independently run frontier-scale training efforts, on different model families and different teams, landing on the same conclusion through the same failure mode: FP16 training at this scale needs active babysitting that BF16 doesn't. PyTorch's own documentation states plainly that bfloat16 autocast doesn't require GradScaler at all, because its exponent range doesn't produce the underflow problem loss scaling exists to fix in the first place. As the Intel, Google, and Facebook researchers who introduced BF16 for deep learning put it, "maintaining the same range as FP32 is important to ensure that no hyper-parameter tuning is required for convergence," where IEEE FP16 needs exactly that tuning, in the form of loss scaling, to converge reliably.
If you're setting this up in a real training config rather than reading about it, DeepSpeed's ZeRO-3 setup shows where the flag actually goes: "bf16": { "enabled": true } sitting next to "gradient_clipping": 1.0 in ds_config.json, with no loss-scale block anywhere in the file. Compare that to what OPT's team had to do under FP16: hand-lower gradient clipping from 1.0 to 0.3 just to keep the run from diverging again. The BF16 config leaves that value at its untouched default and moves on, which is the practical difference this whole comparison comes down to.
The Verdict, and Which GPUs Even Give You the Choice
For training runs above toy scale, BF16 is the default and FP16 is the exception you reach for with a specific reason, not the other way around. The exponent range that removes an entire class of divergence is worth more than FP16's extra mantissa precision on a workload where a single bad batch can produce a gradient the format simply can't hold.
When FP16 Still Wins (Inference, Not Training)
None of this makes FP16 obsolete. Once a model is trained and its weight and activation ranges are known and stable, FP16's extra three mantissa bits over BF16 can produce measurably tighter numerics for inference, particularly on architectures tuned around it. The instability BLOOM and OPT hit was a training-time problem: unbounded gradients during backpropagation, not bounded, calibrated values during a forward pass. If your question is inference precision rather than training stability, FP8 vs BF16 accuracy loss covers the tradeoff that actually applies at serving time, where per-tensor scaling and a converged model change the calculus entirely.
Hardware Gate: BF16 Needs Ampere or Newer
The choice is also a hardware question, not just a format flag in a config file. Native BF16 Tensor Core support arrived with NVIDIA's Ampere architecture; V100's first-generation Tensor Cores never got it, which is exactly why BLOOM's early 104B experiments had no BF16 option available on V100 and had to change GPU generations to get one. Every GPU in Spheron's current rental catalog is Ampere or newer, from A100 through H100, H200, and the Blackwell generation, so a reader who decides BF16 is the right call for a new run doesn't need to audit Tensor Core generation before renting: it's a given across the catalog. What Spheron's catalog can't give you is a V100 to reproduce the Volta-era FP16 failure mode itself; that's a legacy fleet question, not a current-generation rental one. If you're choosing between two BF16-capable GPUs rather than asking whether you have one at all, A100 vs H100 and our H100 vs A100 training cost guide go deeper on that decision, and Spheron's deployment docs cover provisioning the instance once you've picked one.
Because Spheron bills per minute on both tiers, testing whether your own model diverges in FP16 or trains clean in BF16 doesn't need a procurement cycle to answer. A100 runs $1.43/hr on-demand or $1.19/hr on spot as of 10 Sep 2026, so a two-hour side-by-side run of both formats on your own architecture is a couple of GPU-hours, not a line item anyone needs to sign off on. That's a cheaper way to settle the question for your specific model than trusting BLOOM's or OPT's result to transfer exactly, since neither paper is a guarantee about a model neither team trained.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 10 Sep 2026; other providers reflect their most recent published rates and may have changed. Check current GPU pricing → for live rates.
The exponent field is what stood between BLOOM's 104B run and a training log full of unrecoverable divergences. Every Ampere-or-newer GPU in Spheron's catalog gives you the BF16 path that fixed it.
Frequently Asked Questions
Bfloat16 is the safer default for training large models. It carries the same exponent range as FP32 (8 exponent bits), so gradients and activations rarely overflow or underflow, and it needs no loss scaling. Float16 packs more mantissa precision (10 bits versus BF16's 7) but its 5-bit exponent caps its range at roughly 65,504, which is why training teams like BLOOM's and OPT's hit irrecoverable divergence in FP16 at scale and had to work around it with loss scaling, gradient clipping, or a switch to BF16 entirely.
FP16 spends its 16 bits as 1 sign, 5 exponent, and 10 mantissa bits, so its largest representable magnitude is about 65,504. Gradient and activation values in a large transformer routinely exceed that during training, so they overflow to infinity or underflow to zero. BF16 spends its 16 bits as 1 sign, 8 exponent, and 7 mantissa bits, matching FP32's exponent range in the tens of thousands of orders of magnitude, so the same values that overflow FP16 fit comfortably in BF16 without any special handling.
Loss scaling helps but doesn't fully close the gap. It multiplies the loss before the backward pass so small gradients don't underflow to zero, then divides them back out before the optimizer step, and PyTorch's GradScaler automates this. But OPT-175B's training team still logged repeated divergences where their dynamic loss scalar crashed to 0 and had to restart from checkpoints with a lower learning rate and tighter gradient clipping to recover. BF16 doesn't need GradScaler at all, because its exponent range doesn't produce the underflow problem loss scaling exists to fix.
