GPU Interconnect Scaling Benchmark: NCCL Allreduce, H100 vs H200 vs B200 (2026)

A gpu interconnect scaling benchmark that only looks at NVLink generation misses the actual story on H100, H200, and B200. H200 shares H100's 4th-gen NVLink, 3rd-gen NVSwitch, and ConnectX-7 network cards almost pin for pin, so if the generation gap were purely about the interconnect, the two would scale identically on multi-node NCCL allreduce. The published numbers, read against B200's compute, say the real break happens somewhere else: B200's FP8 and FP4 throughput grew faster than the ConnectX-7 400Gb/s scale-out fabric most HGX B200 nodes still ship with, and that mismatch is what pulls the node count where allreduce starts eating the step budget earlier for B200 than it does for H100.

If you haven't run nccl-tests against your own cluster yet, our guide to benchmarking a GPU cluster's interconnect with nccl-tests covers building all_reduce_perf, reading busbw against algbw, and the acceptance thresholds practitioners hold a fabric to. This post assumes you already know how to run that test and focuses on a different question: what the measured numbers actually look like once you compare H100, H200, and B200 against each other, and where each generation's scaling curve starts to bend.

TL;DR: Where Multi-Node Scaling Efficiency Breaks Down by GPU

Multi-node NCCL allreduce scaling breaks down earlier on B200 than on H100 or H200, because B200's FP8/FP4 compute outgrew the same ConnectX-7 400Gb/s scale-out fabric most HGX B200 nodes still ship with.

• Shared stack: H100 and H200 SXM both run 4th-gen NVLink at 900GB/s per GPU behind a 3rd-gen NVSwitch, and both typically ship on ConnectX-7 NDR400 scale-out NICs at 400 Gb/s per GPU.
• H100 at scale: Microsoft Azure measured training scaling efficiency above 97% for both FP8 and BF16 on a 314B-parameter model across 512 to 1,024 H100 GPUs, confirmed 4 Nov 2026.
• B200 tuned vs. untuned: a B200 cluster achieved 98% of theoretical AllReduce bandwidth at roughly 1,024 GPUs; an untuned fabric of the same class fell to 60% of theoretical AllReduce bandwidth, per the same Hedgehog Cloud measurement, confirmed 4 Nov 2026.
• Spheron: publishes on-demand and spot H100, H200, and B200 instances, plus a Custom & Reserved tier scaling to 512+ GPUs with InfiniBand configurations on request, as of 4 Nov 2026. Rent a B200 node to baseline your own allreduce numbers before scaling out.

NCCL Allreduce Bandwidth: H100 vs H200 vs B200 in a GPU Interconnect Scaling Benchmark

Start with what's actually different in the silicon. H100 and H200 SXM both run 4th-gen NVLink at 900GB/s bidirectional per GPU behind a 3rd-gen NVSwitch. B200 moves to 5th-gen NVLink at 1.8TB/s per GPU, behind a newer NVSwitch generation, roughly double H100 and H200's per-GPU NVLink bandwidth.

That's the number most comparisons stop at, and it's the wrong one for a multi-node benchmark. NVLink and NVSwitch only cover traffic between the 8 GPUs inside one node. The moment an allreduce has to cross node boundaries, it's handled by a different piece of hardware entirely: the scale-out network interface card. HGX H100, HGX H200, and most HGX B200 nodes all ship with the same ConnectX-7 NDR400 adapters at 400 Gb/s per GPU for scale-out networking, which works out to about 3.2 Tb/s per 8-GPU node. The newer ConnectX-8-class 800G networking that would actually raise that ceiling arrives with GB200, not with a standard HGX B200 node.

Lay the three out this way and H100 and H200 are the same row with a different memory spec attached. B200's NVLink column jumps; its scale-out column doesn't, on a standard node. That's the whole thesis of this post in table form.

Real measurements back up that the ceiling, not the GPU generation, is what caps multi-node allreduce. Microsoft Azure measured scaling efficiencies above 97% for both FP8 and BF16 precision when training a Grok-1-scale 314B-parameter model across 512 to 1,024 H100 GPUs, which at 8 GPUs per node is somewhere between 64 and 128 nodes, well into the range where a weak interconnect would show up as lost throughput. On the B200 side, Hedgehog Cloud discussed a cluster achieving 98% of theoretical AllReduce bandwidth efficiency at a scale of roughly 1,024 GPUs, both confirmed 4 Nov 2026. Tuned correctly, a B200 fleet on the same fabric class as H100 gets close to the same percentage of its ceiling. The problem shows up when the fabric isn't tuned, or isn't matched to what B200 actually needs, which the next two sections cover.

Why NVSwitch Generation Changes the Curve, Not Just Raw Bandwidth

It's tempting to read B200's 1.8TB/s NVLink and conclude its multi-node scaling should simply be better than H100's across the board. That's backwards for the part of the curve that actually matters at scale.

NVSwitch generation changes how fast GPUs inside a node can reduce their local contribution before anything touches the network. NCCL's collective algorithms are hierarchical: within a node, partial reductions happen over NVLink through the NVSwitch, which is fast and gets faster with each generation; only the node's combined result then has to cross the scale-out fabric to reach every other node. B200's 5th-gen NVLink and newer NVSwitch generation genuinely speed up that local step. At small scale, where most of an allreduce's traffic stays inside a handful of nodes, that advantage is real and visible.

The curve bends as node count grows, because the share of total traffic that has to leave the node and cross the scale-out NIC increases with every additional node you add. Past some cluster size, the operation is gated almost entirely by that inter-node hop, and that's exactly the hop where B200 inherited H100 and H200's ConnectX-7 400 Gb/s ceiling rather than getting its own upgrade. The NVSwitch generation changed the shape of the curve at small scale; it didn't move the asymptote that the curve settles into once scale-out traffic dominates. A faster local reduction step that still feeds into the same 400 Gb/s pipe out of the node buys you less than it looks like it should once you're running hundreds of GPUs instead of eight.

This is also why generation comparisons that stop at the spec sheet get the direction of the risk backwards. NVIDIA's GB200 NVL72 architecture is the clearest illustration of what actually fixes this: it puts 72 GPUs behind a single NVLink domain with roughly 130 TB/s of aggregate NVSwitch bandwidth and more than 1 PB/s total, which moves far more of what used to be "the network" behind NVLink instead of a scale-out NIC. That's a structural change to where the node boundary sits, not just a faster version of the same 8-GPU box. A standard HGX B200 node doesn't get that; it gets the faster local reduction step and the same old pipe out the door.

The Node Count Where Scaling Efficiency Drops Below 80%

There's no single node count where every B200 cluster crosses below 80% efficiency and every H100 cluster doesn't; the published numbers don't support a clean universal threshold like that, and anyone who gives you one without naming a specific fabric configuration is guessing. What the numbers do support is a mechanism, and a sense of how much margin for error each generation has left.

Our own acceptance-testing writeup on nccl-tests covers the two-tier bar practitioners actually use: greater than 90% of theoretical bandwidth for intra-node NVLink, and greater than 80% of theoretical bandwidth for inter-node InfiniBand or RoCE, drawn from the AI-HPC Alliance's acceptance framework. That 80% line is the one worth watching, because it's the inter-node hop where H100, H200, and standard B200 nodes all share the same ConnectX-7 ceiling.

Hedgehog Cloud's analysis discusses how wide the gap between a tuned and an untuned B200 fabric actually is at the same scale, as confirmed 4 Nov 2026: a well-tuned cluster achieved 98% of theoretical AllReduce bandwidth, while a misconfigured one fell to about 60% of theoretical AllReduce bandwidth. That 60% is nowhere near the 80% inter-node bar, and it's a configuration failure, not a generation failure; the same misconfiguration would hurt an H100 cluster on the same fabric about as badly. The difference is what that lost efficiency costs. Hedgehog's own analysis puts an untuned DIY RoCEv2 fabric on a B200-class cluster at a 15% model-FLOPs-utilization penalty versus a tuned fabric, working out to roughly $8.07 million a year in lost throughput across 1,024 GPUs. Run that same percentage of lost bandwidth on an H100 cluster, where the GPU's own FLOPs are smaller to begin with, and the dollar cost of the identical misconfiguration is lower, because there's less compute sitting idle behind the same stalled network.

That's the real shape of the risk: B200's higher per-GPU compute doesn't make its interconnect worse, but it does make the cost of any given efficiency drop, whether that's a scaling-curve bend from node count or a tuning failure, more expensive per percentage point lost. The margin for sloppy fabric work shrinks as the GPU behind it gets faster.

What a GPU Interconnect Scaling Benchmark Means for How Many GPUs to Rent

None of this changes anything for a single node. If you're training or serving on 8 GPUs, NVLink and NVSwitch handle the entire collective and the scale-out NIC never gets involved, so H100, H200, and B200 all perform to their local interconnect's spec regardless of generation. The scaling story only starts once you add a second node, and it only becomes a serious buying question somewhere in the tens-to-hundreds-of-GPUs range, which is also where best NVIDIA GPUs for LLMs stops being purely a VRAM-per-card question and starts being a cluster-topology one.

Three practical takeaways follow from the mechanism above:

1. Don't assume an H100-era interconnect number carries over to B200 without checking. The compute changed; the scale-out NIC on a standard node usually didn't. Confirm whether you're on ConnectX-7 (400 Gb/s per GPU) or an upgraded 800G-class fabric before planning a multi-hundred-GPU B200 training run around a theoretical scaling curve.
2. A misconfigured fabric is a bigger risk than the wrong GPU generation. The gap between Hedgehog Cloud's tuned (98%) and untuned (as low as 60%) B200 analysis dwarfs anything attributable to NVLink or NVSwitch generation on its own. Run your own nccl-tests pass, hierarchically, from single node up through the full cluster, before trusting either a vendor's spec sheet or a generation-based assumption.
3. Translate scaling efficiency into a rental decision, not just a benchmark number. A cluster running at 80% busbw efficiency instead of 97% isn't failing outright, but it's effectively renting more GPU-hours than it's using productively. Our GPU cluster profit margin calculator walks through turning utilization and efficiency gaps like this into an actual dollar figure, and the same compute-to-communication tradeoff shapes GPU sizing in multi-node RLHF training infrastructure, where rollout and training roles scale across nodes differently.

Spheron publishes on-demand and spot rates for H100, H200, and B200 directly, which is the cheapest way to run a single-node baseline before committing to anything larger: rent one node, run all_reduce_perf yourself, and compare it against the ceilings in the table above rather than trusting a sales deck. For the class of cluster this post's node-count math actually applies to, Spheron's Custom & Reserved tier scales from 8 to 512+ GPUs with InfiniBand configurations available on request. InfiniBand there is a request, not a default on every tier, so confirm the NIC generation for your specific deployment before assuming a ceiling, and note that custom multi-node sourcing runs on a 24-48 hour turnaround rather than instant self-serve, which makes it the wrong fit if you need a validated large cluster the same day. For current rates across the catalog, including how H100, H200, and B200 compare against AWS and Azure list prices, see our GPU cloud pricing comparison.

Scaling efficiency is a fabric question before it's a GPU-generation question. Confirm which NIC generation is actually wired into a cluster before trusting any vendor's spec sheet, including ours.

Check current GPU pricing →