Here's ZeRO optimizer stages explained the way it actually matters for a rental bill: they aren't three flavors of the same trick with diminishing returns. Each one shards a different piece of what training normally keeps duplicated on every GPU, and each piece you shard away frees VRAM at the cost of more traffic on the network. ZeRO-1 and ZeRO-2 are close to free: they cut memory without adding a new communication step. ZeRO-3 is a different deal. It ships roughly 1.5 times the communication volume of standard data-parallel training in exchange for sharding the model itself, and whether that trade is worth it depends less on which stage sounds more advanced and more on whether a bigger GPU would have solved the same problem for less.
TL;DR: ZeRO Optimizer Stages Explained, ZeRO-1 vs ZeRO-2 vs ZeRO-3
- What each stage shards: ZeRO-1 shards optimizer states, ZeRO-2 adds gradients, ZeRO-3 adds parameters too, per DeepSpeed's ZeRO tutorial.
- The communication cost: ZeRO-3 moves about 1.5x the traffic of standard data-parallel training (and of ZeRO-1/ZeRO-2), per Microsoft Research.
- What hides that cost: DeepSpeed prefetches each layer's parameters ahead of the forward pass, overlapping the extra all-gather with compute.
- The real trigger: Hugging Face's own guidance is to use ZeRO-3 only when the model doesn't fit under ZeRO-2, not by default.
- Rental math: Spheron prices H100 80GB at $2.65/hr on-demand and H200 141GB at $4.80/hr on-demand as of 19 Sep 2026. Rent an H100 GPU to test your own crossover point.
What Each ZeRO Optimizer Stage Actually Shards
Every stage of DeepSpeed's Zero Redundancy Optimizer targets the same underlying problem: standard data-parallel training keeps a full copy of the optimizer states, gradients, and parameters on every single GPU, even though each GPU only ever needs its own slice of the math at any given moment. ZeRO removes that redundancy one layer at a time, and the layer it removes determines both the memory it frees and the communication it costs.
ZeRO-1 (Pos): Optimizer States Only
ZeRO-1, DeepSpeed's baseline stage, partitions only the optimizer states across the data-parallel processes, so that each process updates just its own partition rather than a full redundant copy. For an Adam optimizer, that's the momentum and variance estimates plus the fp32 master weights, which for large models is often the single biggest memory consumer in the whole training stack, since Adam alone can carry more state per parameter than the parameters themselves. Because gradients and parameters stay fully replicated, ZeRO-1 doesn't introduce a communication step that standard data-parallel training didn't already have.
ZeRO-2 (Pos+g): Optimizer States Plus Gradients
ZeRO-2 extends stage 1 by also partitioning the reduced gradients, so each process retains only the gradient shard that matches the optimizer-state partition it already owns. Gradients are naturally produced through a reduction across all ranks during backward, so ZeRO-2 essentially changes an all-reduce into a reduce-scatter, communication that was already happening, just reshaped to only keep the piece each rank needs afterward, rather than the full gradient tensor.
ZeRO-3 (Pos+g+p): Optimizer States, Gradients, and Parameters
ZeRO-3 is where the stage stops being "free." It partitions the model parameters themselves on top of the optimizer states and gradients, and then automatically gathers and re-partitions those parameters during the forward and backward passes as each layer needs its weights. No single GPU holds a full copy of the model at rest. That's what makes ZeRO-3's memory reduction scale with the data-parallel degree itself rather than topping out at a fixed multiple. The cost is a genuinely new communication step: an all-gather to reconstruct each layer's parameters before they're used, on top of the reduce-scatter ZeRO-2 already runs.
The Communication Tax: All-Gather and Reduce-Scatter Per Layer
Standard data-parallel training (and ZeRO-1/ZeRO-2 riding on top of it) moves roughly 2Ψ worth of communication per training step, where Ψ is the model's parameter count: one reduce-scatter to combine gradients across ranks, and one all-gather to give every rank the reduced result it needs. ZeRO-3 adds a third collective on top of that pattern, and the extra traffic isn't free, but it's also smaller than doubling.
Why ZeRO-3 Adds a Third Collective (3Ψ vs 2Ψ)
ZeRO-3 has to reconstruct each layer's full parameters before the forward pass can run on it, since no rank is holding the full set at rest. That's an all-gather on the way in, on top of the gradient reduce-scatter ZeRO-2 already does and the parameter re-partitioning that follows once the layer's forward or backward computation finishes. Microsoft Research's own writeup on the technique states it directly: ZeRO-3 incurs a modest 50% (1.5x) increase in communication volume versus standard data-parallel training, the tradeoff for its much larger memory savings. In plain terms, that's not triple the traffic, it's half again as much, in exchange for trading a fixed memory ceiling for one that keeps falling as you add GPUs.
The 1.5x Number, and Why Overlap/Prefetching Hides Most of It
A 50% increase in raw communication volume doesn't translate one-for-one into a 50% slower training step, because DeepSpeed doesn't wait for the all-gather to finish before doing useful work. It prefetches the next layer's parameters while the current layer is still computing, overlapping the extra collective with compute that was going to happen anyway rather than stacking it on top as pure idle time. Whether that overlap actually hides the cost depends on the interconnect: within a single node's NVLink domain, the all-gather is fast enough that it mostly disappears behind compute; across nodes on a slower network fabric, the same all-gather takes longer to complete and starts to show up as real wall-clock time. Hugging Face's DeepSpeed integration guide puts the practical guidance in one line: "ZeRO-2 shards gradients and optimizer states with lower communication overhead than ZeRO-3. Use ZeRO-3 only when your model doesn't fit across GPUs with ZeRO-2." That's not a hedge, it's the actual decision rule: ZeRO-3 is a fallback for when you've run out of memory headroom, not a default upgrade path.
ZeRO++ and Quantized Communication: Shrinking the Tax to 0.75Ψ
DeepSpeed's follow-up work, ZeRO++, goes after the 1.5x number directly rather than just hiding it behind compute. It cuts ZeRO-3's communication volume from 3Ψ down to 0.75Ψ, a 4x reduction, through three separate optimizations that target two separate collectives. On the weight all-gather side: a quantized weight step (qwZ) that compresses parameters from fp16 down to int8, paired with a hierarchical partitioning step (hpZ) that keeps a full model replica within each node so the backward pass's weight all-gather never has to leave the node at all. On the gradient side: a quantized gradient step (qgZ) that replaces the usual gradient allreduce with an all-to-all based quantized averaging. That matters most for exactly the case where prefetching runs out of headroom: multi-node ZeRO-3 over a fabric that can't keep the all-gather hidden behind compute. It's not a free lunch either, since quantization introduces its own precision tradeoff, but it's the tool DeepSpeed built specifically to make the communication tax smaller rather than just better-scheduled.
The GPU-Count Crossover Point Where ZeRO-3 Stops Costing You Speed
The question worth asking isn't "which ZeRO stage is best." It's "at what point does renting a bigger GPU to stay on ZeRO-2 stop being cheaper than renting more GPUs and paying ZeRO-3's communication tax." That crossover point moves with your model size, your interconnect, and current GPU prices, which is exactly why it has to be measured rather than assumed.
Small Clusters (Single Node, NVLink Domain): ZeRO-2 With VRAM Headroom Usually Wins
Inside a single node, where GPUs talk to each other over NVLink at hundreds of gigabytes per second rather than over a cross-node network fabric, ZeRO-2's memory ceiling is the more common constraint than ZeRO-3's communication overhead, because the interconnect is fast enough that ZeRO-3's extra all-gather barely registers. DeepSpeed's own team documents the memory ceiling directly: a single 32GB V100 with 1.5TB of CPU memory behind it can fine-tune models up to roughly 13B parameters under ZeRO-2-Offload, and ZeRO-3-Offload pushes that same hardware to models over 40B parameters, a 3x jump. That 3x jump in what fits is a memory story, not a speed one: it's evidence for when ZeRO-2 genuinely runs out of room, not evidence that ZeRO-3 is faster. Understanding NVLink's role here matters, since it's what keeps ZeRO-3's extra collective cheap in the first place; see what NVLink actually delivers in bandwidth for the numbers behind why intra-node ZeRO-3 and cross-node ZeRO-3 behave so differently. If the model still fits under ZeRO-2 with headroom to spare, the simpler stage with the lower communication overhead is the one Hugging Face's own guidance points to by default.
What the Freed VRAM Actually Buys You
ZeRO-3's freed memory rarely gets banked as a pure speed win. In practice, teams spend that headroom on a bigger microbatch or a longer sequence length instead of leaving it idle, since it's there either way once the parameters are sharded. That's consistent with the overlap DeepSpeed's prefetching relies on: on a fast, single-node interconnect, the extra all-gather mostly disappears behind compute, so a larger effective batch under ZeRO-3 can land close to ZeRO-2's wall-clock time on the same hardware without the freed VRAM ever showing up as a standalone speedup. The same logic that governs whether extra GPUs pay for themselves at all applies here too; see why doubling your GPU count rarely halves your training time for the general version of the communication-bound wall this post is a special case of.
Deciding Which Stage to Rent For: A Sizing Checklist
Work through these in order before picking a stage, and price the alternative before committing to a GPU count:
- Measure your model's ZeRO-2 footprint on the GPU you're actually pricing. Optimizer states plus gradients plus activations, on the real VRAM figure for the card, not a rounded-up estimate.
- If it fits, stop at ZeRO-2. No new collective, lower communication overhead, and Hugging Face's own DeepSpeed guide backs this as the default.
- If it doesn't fit, check whether a bigger single GPU closes the gap before reaching for ZeRO-3. Moving from an 80GB card to a 141GB one is a VRAM increase with zero communication cost; adding GPUs under ZeRO-3 is a VRAM increase that comes with a new all-gather on every layer. Gradient checkpointing is the other lever worth trying first, since it trades compute for activation memory without touching the optimizer sharding question at all.
- If ZeRO-3 is genuinely required, keep it inside a single NVLink domain where possible. That's where the 1.5x communication number is most likely to disappear behind prefetch overlap rather than show up as slower wall-clock time.
- If ZeRO-3 has to span nodes, budget for the communication tax explicitly, and look at ZeRO++'s quantized collectives if the interconnect is the bottleneck rather than compute.
- Price both paths before renting. A single H200 at $4.80/hr on-demand against two H100s at $2.65/hr each, as of 19 Sep 2026, is a real comparison you can run with live numbers rather than assumed ones; the full multi-node DeepSpeed ZeRO-3 setup guide has a working
ds_config.jsononce you've decided ZeRO-3 is the right call.
Spheron's pricing page lists H100 80GB, H200 141GB, A100 80GB, and B200 192GB side by side with live on-demand and spot rates, which is the actual lever this sizing decision turns on: a team can price "one bigger GPU on ZeRO-2" against "more GPUs on ZeRO-3" using the same marketplace before committing to either. As of 19 Sep 2026, A100 80GB runs $1.43/hr on-demand and $1.12/hr spot, and B200 192GB runs $9.36/hr on-demand and $5.37/hr spot. None of that prices the interconnect itself, though: the pricing page doesn't specify InfiniBand generation or NVLink topology per instance, and that's the variable that most changes where a multi-node ZeRO-3 crossover point actually falls, so confirm the fabric on the specific instance type before sizing a cross-node run around it. If ZeRO-3 does end up spanning multiple nodes on rented spot capacity, a preempted rank means reconstructing sharded optimizer and parameter state, not just reloading a flat checkpoint; the spot GPU checkpointing guide for FSDP and ZeRO-3 covers what that recovery path actually requires.
Pricing fluctuates based on GPU availability. Spheron rates above are live as of 19 Sep 2026; other providers and configurations may differ, and per-instance interconnect specifics should be confirmed directly. Check current GPU pricing → for live rates.
ZeRO Optimizer Stages Explained: The Bottom Line
The stage names sound like a progression, ZeRO-1 to ZeRO-2 to ZeRO-3, each one strictly better than the last. The memory numbers back that framing up. The communication numbers don't: ZeRO-2 is close to free, and ZeRO-3 is a real cost you pay for a real capability, worth taking exactly when a bigger GPU can't buy you the same headroom for less.
Sizing a training run against ZeRO-2 versus ZeRO-3 comes down to comparing rental costs for bigger single GPUs against more GPUs under real communication overhead, and an H200's extra VRAM is often the cheaper way to stay on ZeRO-2 in the first place.
Frequently Asked Questions
ZeRO-1 shards only the optimizer states across GPUs, so each rank updates just its own slice. ZeRO-2 shards the reduced gradients on top of that, so a rank only keeps the gradients matching its optimizer-state partition. ZeRO-3 goes further and shards the model parameters themselves, automatically gathering each layer's weights right before it's needed in the forward and backward pass and releasing them right after. Each stage frees more VRAM than the last, and each one after ZeRO-1 adds more inter-GPU communication.
There's no fixed GPU count, because the real trigger is whether the model plus its optimizer states and activations fit in ZeRO-2's per-GPU memory footprint on the hardware you're renting. A 7B-13B model comfortably fits under ZeRO-2 on a single 80GB H100 with room to spare. Past roughly 30B-70B parameters, or on GPUs with less VRAM, ZeRO-2's per-rank footprint stops fitting and ZeRO-3 (or a larger GPU) becomes the only way to keep training. Measure your own model's ZeRO-2 memory footprint on the GPU you're pricing before assuming you need ZeRO-3.
Not by as much as the raw communication math suggests, and not always at all. ZeRO-3 moves roughly 3 times a model's parameter count in communication per step, against roughly 2 times for standard data-parallel training and ZeRO-2, a 50% increase in communication volume that DeepSpeed's own team documents. In practice, DeepSpeed prefetches each layer's parameters ahead of when they're needed, overlapping the extra all-gather with compute from the previous layer, which hides most of that overhead as long as the interconnect keeps up. On a single NVLink-connected node, the difference between ZeRO-2 and ZeRO-3 wall-clock time is often small to negligible.
Yes. ZeRO++, DeepSpeed's follow-up optimization, cuts ZeRO-3's communication volume from roughly 3 times parameter count down to about 0.75 times, a 4x reduction, through two separate techniques: a quantized weight all-gather that compresses parameters from fp16 to int8 and keeps a full model replica per node so that traffic mostly stays intra-node, and a quantized gradient step that replaces the usual gradient allreduce with an all-to-all based quantized averaging. It's aimed specifically at the case where ZeRO-3 is communication-bound, such as multi-node training over a slower fabric.
Price both paths before deciding. Renting a single higher-VRAM GPU (moving from an 80GB H100 to a 141GB H200, for example) can keep a model on ZeRO-2 without adding any communication overhead, while renting more GPUs to spread the same model under ZeRO-3 adds the extra all-gather collective on every layer. Compare the hourly cost of the bigger single GPU against the hourly cost of the extra GPUs plus whatever wall-clock time ZeRO-3's communication tax adds on your specific interconnect, using current on-demand and spot rates rather than list prices from memory.
