NCCL Tests: Benchmark Your GPU Cluster Interconnect (2026)

A vendor's spec sheet says 400 Gb/s InfiniBand. Nothing about that number tells you what your cluster actually delivers once the cabling, the switch firmware, and the NIC placement are real instead of theoretical. Before you accept delivery or renew a multi-node contract, the gap between "advertised" and "measured" is the only number that matters, and a spec sheet can't produce it.

nccl-tests is NVIDIA's own microbenchmark suite for NCCL collective operations, and it's the tool that turns "up to 400 Gb/s" into a measured figure you can hold a provider to. This post covers building it, running it correctly from a single node up through a full cluster, reading algbw and busbw without confusing the two, and the 92%-of-theoretical-max threshold practitioners use as a pass bar. If you already operate a cluster and want to tune NCCL for a workload you know is healthy, that's a different job covered in our NCCL tuning guide for multi-GPU training; this post is about verifying the fabric before you trust it with anything.

TL;DR: How to Benchmark a GPU Cluster's Interconnect with NCCL Tests

• Acceptance bar: all_reduce_perf busbw near 92% of the fabric's theoretical max, per Together AI's practitioner's guide.
• Report busbw, not algbw. algbw inflates with GPU count; busbw corrects for that, staying comparable to the fabric's real ceiling.
• Where it fits: Spheron's Custom & Reserved tier scales to 512+ GPUs with InfiniBand on request, the build class this test is for. More on cluster health →

Why You Should Never Trust an Advertised Bandwidth Number

An "up to 400 Gb/s" spec describes the NIC, not the cluster you were handed. It says nothing about whether every fiber in the MPO trunk is seated correctly, whether the switch firmware is running the routing profile the vendor tested with, or whether a GPU is silently running its PCIe link at Gen3 x8 instead of the Gen5 x16 it was rated for.

Real gaps between advertised and measured bandwidth show up for mundane reasons. Leviathan Systems points to cabling as the other common culprit: "The most common failure mode in field deployments is a marginal MPO connector (dust, damage, or misalignment) causing link errors that reduce effective bandwidth by 10-50% without a full link drop." Since an MPO trunk of that size carries its capacity across 12 individual fibers, losing even one to damage or misalignment works out to at least an 8.3% cut on its own, on top of whatever else the surrounding fibers are also degraded by, even while the link stays up and every port shows green.

None of that is visible from a data sheet, a nvidia-smi idle check, or a ping test. It's visible from an actual collective operation moving real bytes across every GPU and every hop in the fabric, which is exactly what nccl-tests does. NVIDIA's own guidance on this is blunt: "A GPU cluster can pass every health check and still fail to run an AI workload," as its technical blog on cluster readiness puts it. Individual GPUs can report healthy, every node can boot, and the interconnect can still be the reason a training run stalls on step one.

This is the step to run before you sign, before you accept delivery, and before you scale a pilot deployment into a production commitment. This post is what gives you the number to write into that contract; later on we cover exactly where the bandwidth clause belongs in the agreement itself.

Building and Running nccl-tests: From Single Node to Full Cluster

nccl-tests is NVIDIA's official suite for checking both the performance and correctness of NCCL collective operations. It ships a separate binary per collective (all_reduce_perf, all_gather_perf, broadcast_perf, reduce_scatter_perf, alltoall_perf, and a few more), and all_reduce_perf is the one to reach for first: all-reduce is the collective every data-parallel and tensor-parallel training job actually depends on, so it's the fairest proxy for whether the fabric will hold up under a real workload.

A single-node run only exercises NVLink or PCIe between local GPUs and tells you nothing about the network fabric between nodes, which is usually where the vendor's advertised number and reality diverge. To validate a multi-node cluster you need MPI support compiled in.

Building nccl-tests with MPI support

git clone https://github.com/NVIDIA/nccl-tests.git
cd nccl-tests
make MPI=1 MPI_HOME=/path/to/mpi CUDA_HOME=/path/to/cuda NCCL_HOME=/path/to/nccl

MPI_HOME, CUDA_HOME, and NCCL_HOME need to point at real install paths on your system, typically /usr/local/mpi, /usr/local/cuda, and wherever your NCCL build lives (system NCCL is often already present under /usr on GPU cloud images). Without MPI=1, the build only produces single-node binaries, which is fine for a quick sanity check but not for validating the interconnect a multi-node contract is paying for.

The all_reduce_perf flags that matter

A handful of flags cover nearly every acceptance run you'll do:

A representative run looks like:

mpirun -np 16 -host node1:8,node2:8 \
  ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 1 -n 20 -w 5

That sweeps message sizes from 8 bytes up to 8 GB, doubling each step, across 16 GPUs spread over two 8-GPU nodes. The large end of that range (multi-GB messages) is the one that matters for acceptance testing: it's the size that actually saturates the fabric, where NIC and switch limitations show up. Small messages are latency-bound and will look fine on a broken fabric.

Scaling up: single node, node pairs, then the full cluster

Together AI's own practice for testing large training clusters, described in its practitioner's guide to testing and running large GPU clusters, runs all_reduce_perf hierarchically rather than jumping straight to the full fleet: single nodes first, then node pairs, then larger groups, working up to the entire cluster. A slow result is the most common NCCL failure signature, and it usually points to a specific InfiniBand fabric problem rather than a GPU problem. Testing hierarchically is what lets you localize it.

The practical reason to do this in order: if the full-cluster run comes back slow, you have no way to tell whether the problem is one bad node, one bad switch port, or a systemic fabric issue, without also having a clean single-node baseline to compare against. Run it in this order:

1. Single node. Confirms intra-node NVLink/NVSwitch bandwidth is healthy before the network is even involved. If this fails, the problem is local (bad GPU, bad NVLink, wrong driver), not the interconnect.
2. Node pairs. The first hop onto the actual network fabric. Test several different pairs, not just one, since a single bad cable or NIC will only show up on the pairs that include it.
3. Larger groups (4, 8, 16 nodes). Scaling world size in powers of 2 exercises more of the switch fabric and more possible routing paths. SemiAnalysis's ClusterMAX methodology follows this same power-of-2 scaling approach when benchmarking provider clusters, alongside ib_write_bw/ib_read_bw for isolated point-to-point rail bandwidth outside of NCCL entirely, per its ClusterMAX 3.0 methodology writeup.
4. Full cluster. The number you actually report against the contract.

If any stage regresses relative to the one before it, stop and isolate before scaling further. A cluster that looks fine at 8 nodes and drops sharply at 16 has told you where to look.

Reading nccl-tests Output: algbw, busbw, and Where the 92% Threshold Comes From

all_reduce_perf prints a table with columns including size, time, algbw, and busbw for every message size in the sweep. The two bandwidth columns measure different things, and conflating them is the most common mistake in reading this output.

algbw (algorithm bandwidth) is simply bytes moved divided by time: how fast the operation completed, in the units NCCL actually transferred. busbw (bus bandwidth) applies a correction factor derived from the specific collective's communication pattern so the result is comparable to the fabric's actual hardware peak, independent of how many GPUs are in the test.

How busbw is calculated, and why it's the number you report

NCCL's own performance documentation lays out the correction factors per collective, as detailed in nccl-tests' PERFORMANCE.md:

• AllReduce: busbw = algbw x 2(n-1)/n
• AllGather / ReduceScatter / AlltoAll: busbw = algbw x (n-1)/n
• Broadcast / Reduce: busbw = algbw x 1

The reason this correction exists: an all-reduce moves more total bytes across the fabric than the size of the tensor being reduced, because of the reduce-scatter-then-all-gather pattern under the hood, and that overhead ratio changes with GPU count n. Without the correction, algbw for the same physical fabric would report a different number at 8 GPUs than at 64 GPUs, even though nothing about the underlying hardware changed. busbw cancels that out, which is what makes it comparable to a NIC's rated bandwidth and comparable across test runs of different sizes.

That's why every acceptance threshold in this post, and every one you'll see cited elsewhere, is stated in terms of busbw, never algbw. If a report only shows algbw, ask for busbw before drawing any conclusion from it.

Where the 92% number comes from

Together AI's practitioner's guide states the target directly: "we are looking for the all_reduce_perf test to show bandwidth around 92% of the theoretical maximum of the fabric: so around 370 GB/s on a 400GB/s fabric." That's the reference number this post anchors on, and it's a reasonable single figure to hold a full-cluster InfiniBand fabric to.

A separate acceptance-testing framework from the AI-HPC Alliance splits the bar by hop type rather than using one number for everything: greater than 90% of theoretical bandwidth for intra-node NVLink, and greater than 80% of theoretical bandwidth for inter-node InfiniBand, per its NCCL performance testing and tuning guide. The two frameworks aren't contradictory so much as scoped differently: NVLink inside a node is a shorter, simpler path with less protocol overhead, so a tighter bar makes sense there, while a full multi-node InfiniBand fabric carries more hops and more room for legitimate overhead before something is actually wrong. Use whichever matches what you're actually testing, intra-node or inter-node, and don't apply the tighter NVLink bar to a full-cluster InfiniBand result.

NVIDIA's own cluster-readiness framework (NVCRE) goes further still, defining four validation stages, bring-up, burn-in, preproduction, and production, and running five separate NCCL variants (all-reduce, all-gather, all-to-all, loopback, and loopback across NVSwitch) at intra-node, intra-rack, full-scale, and diagnostic scope, with thresholds like bus bandwidth at or above 900 GB/s for GB200 NVL72-class systems, according to NVIDIA's own cluster readiness blog post. That level of rigor is what a hyperscaler runs before handing a cluster to an internal team; a single hierarchical all_reduce_perf sweep against the 92% bar is the version a buyer can run in an afternoon before signing.

What 92% of Theoretical Max Actually Looks Like on H100 vs B200

The 92% figure is only useful once you know the fabric's actual theoretical ceiling, and that ceiling depends on which NIC generation is wired into the node in front of you, not just the GPU model printed on the spec sheet.

A B200 cluster's advertised network speed is not automatically faster just because the GPU is newer.

GB200 NVL72 is where this stops being a safe assumption. Its back-end scale-out network keeps the same 1:1 NIC-per-GPU ratio, but runs at either 400G or 800G per GPU depending on the specific deployment's NIC generation, according to SemiAnalysis's writeup on NVL72's InfiniBand scale-out design. Two GB200 NVL72 clusters can carry the identical GPU model and a genuinely different theoretical ceiling. Confirm which NIC generation actually shipped in your rack, from the vendor directly or from ibstat output on the node itself, before you compute what 92% should be. Applying an 800G assumption to a fabric that's actually wired at 400G will fail a healthy cluster; applying a 400G assumption to an 800G fabric will pass a cluster that's running at half its real capacity.

For context on how these numbers compare across provider fabrics in the wild, SemiAnalysis's ClusterMAX benchmark found real spread even among optimized providers: on a 16-node, 16 GiB all-to-all test, Google Cloud's gIB plugin reached 99.5 GB/s versus 95.3 GB/s from another provider on the same test. Neither number is broken. It's the kind of gap that separates "good" from "optimized," and it's only visible once you've run the actual test rather than compared spec sheets.

If you need the intra-node picture too, NVLink bandwidth is a separate topology from the scale-out fabric nccl-tests is validating here; our NVLink explainer covers NVLink generations and how they factor into single-node collective performance before you get to the inter-node number.

When the Number Comes in Low: Common Failure Modes

A busbw result well below the 92% target (or below 80% for inter-node InfiniBand under the split framework) points to a small set of usual suspects, roughly in order of how often they show up in practice:

• PCIe link width mismatch. A GPU rated for Gen5 x16 silently running at Gen3 x8 loses 75% of its host bandwidth. Crusoe's burn-in process checks this on every node specifically because it's common and easy to miss without an explicit check. Confirm with sudo lspci -vv | grep -A 20 "NVIDIA" and look for LnkSta reporting a lower speed or width than LnkCap.
• Damaged or marginal cabling. A single bad fiber inside an MPO trunk causes at least an 8.3% reduction on its own, and dust or misaligned connectors more broadly can cut effective bandwidth by 10-50% without the link ever dropping. This is Leviathan Systems' stated most common field failure mode, and it's invisible to a port-up check.
• Wrong or missing HCA pinning. If NCCL is routing traffic through a NIC that isn't in the same NUMA domain as the GPUs generating the traffic, you'll see a bandwidth hit that looks like a fabric problem but is actually a software misconfiguration, fixable with NCCL_IB_HCA and the topology discovery steps in the tuning guide linked above.
• Switch firmware or routing profile mismatch. A provider's advertised number was measured with a specific adaptive-routing configuration; if your cluster's switches shipped with a different default profile, you won't see the vendor's own benchmark number until it's corrected.
• Isolate before you escalate. Because you tested hierarchically, from single node up through node pairs and larger groups, a low full-cluster number should already point you at roughly which stage it appeared. Escalate to the vendor with that specific result, not just the aggregate one.

If the fabric genuinely can't clear the InfiniBand-class threshold and the workload doesn't strictly need it, that's a real tradeoff worth making deliberately rather than accepting silently. Our guide to multi-node training without InfiniBand covers when a lower-bandwidth Ethernet fabric is still the right call and what it costs you in wall-clock time.

Making Interconnect Verification Part of the Contract, Not an Afterthought

The pattern that actually protects a buyer isn't running nccl-tests once out of curiosity. It's writing the acceptance threshold into the delivery process before money changes hands, the way Crusoe's burn-in pipeline runs NCCL over InfiniBand and RCCL over RoCE across multiple nodes on every single node before customer handoff, alongside an RDMA bandwidth and error-rate check and a PCIe link-width check. That's a pre-delivery gate, not a post-hoc audit.

For a reader negotiating a reserved multi-node cluster, that same idea belongs in the contract itself: a specific busbw threshold, measured with all_reduce_perf at a specified message size, checked before the commitment period starts and with a defined remedy if the number doesn't clear it. Our guide to negotiating GPU cluster reservation contracts covers where that clause fits alongside minimum commitments and capacity guarantee SLAs.

Bare-metal access matters here in a way it's easy to underrate: running ibstat and nvidia-smi topo -m to see the real NIC-to-GPU wiring, and getting an unfiltered all_reduce_perf result rather than one shaped by a hypervisor layer, requires the kind of direct node access a virtualized instance doesn't give you. Spheron's standard on-demand and bare-metal marketplace instances give you that visibility for whatever fabric ships with the node. For the class of cluster this post is actually about, a multi-node build large enough that a contract and an SLA are on the table, Spheron's Custom & Reserved tier scales from 8 to 512+ GPUs with InfiniBand configurations available on request, which is exactly the kind of deployment worth running through this acceptance workflow before you sign. InfiniBand there is available on request rather than a given on every tier, so confirm the fabric configuration for your specific deployment rather than assuming it, and note that this is a verification method you run yourself: it isn't a published benchmark or burn-in step Spheron performs on your behalf.

Once the fabric passes acceptance, tuning it for your actual training workload is a separate job with its own environment variables and topology decisions, covered in the tuning guide linked earlier. And if the interconnect isn't the bottleneck you're chasing this quarter, the same discipline of measuring instead of trusting a spec sheet applies to compute too: our reproducible GEMM benchmark with the CUTLASS profiler is the same idea applied to raw matmul throughput instead of the network.

Verifying a cluster's real interconnect bandwidth before you commit to it is the same discipline as checking any other spec you're paying for. Explore Custom & Reserved GPU clusters on Spheron and confirm the fabric you're getting before you sign.

Get started on Spheron →