Every AI cluster commissioning process runs nccl-tests before handover to production. It is the standard benchmark suite that validates the full stack: NCCL configuration, NIC drivers, RDMA transport, switch fabric, and topology -- simultaneously.
Reading nccl-tests output — click any column header to understand it
This is allreduce_perf output for a 32-node (256 GPU) BasePOD cluster at 400G
| size (B) | time (µs) | algbw (GB/s) | busbw (GB/s) ◀ | #wrong |
|---|---|---|---|---|
| 8388608 | 412.5 | 20.3 | 38.1 | 0 |
| 16777216 | 743.7 | 22.6 | 42.4 | 0 |
| 134217728 | 4891.2 | 27.4 | 51.4 | 0 |
| 536870912 | 9814.2 | 54.7 | 102.6 | 0 |
| 1073741824 | 17234.8 | 62.3 | 116.8 | 0 |
| 8589934592 | 109842.1 | 78.2 | 146.6 | 0 |
algbw × 2(N-1)/N. This corrects for the ring algorithm's communication volume. Each DGX H100 node has 8 × 400G NICs (~400 GB/s aggregate per node). The busbw in a 32-node / 256-GPU test reflects per-node aggregate throughput across all 8 NICs. Compare busbw to 8 × 50 GB/s = 400 GB/s per node max. A healthy 32-node cluster shows 130–160 GB/s busbw at large tensor sizes (33–40% of theoretical max, expected due to ring overhead and fabric hops).
Latency-dominated. Low busbw is normal.
Bandwidth-dominated. Compare busbw to NIC line rate.
STOP. Data corruption. Do not run training.
Running allreduce_perf
# On DGX node A -- run across all nodes in the cluster
mpirun --hostfile /etc/mpi/hostfile \
-np 256 \
--map-by ppr:8:node \
-x NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1 \
-x NCCL_DEBUG=INFO \
-x NCCL_IB_GID_INDEX=3 \
./build/all_reduce_perf \
-b 8 \ # Start at 8 bytes
-e 8G \ # End at 8 GB
-f 2 \ # Double message size each step
-g 1 \ # 1 GPU per process
-c 1 # Check results for correctness
Flags explained:
-b 8 -e 8G -f 2: Sweeps from 8 bytes to 8 GB, doubling each step -- reveals performance across all tensor sizes-g 1: Each MPI process owns 1 GPU-c 1: Enables the#wrongcorrectness check -- critical, must be zero
Reading the output table -- field by field
# out-of-place
# size count type redop root time algbw busbw #wrong
8388608 2097152 float sum -1 412.5 20.3 38.1 0
16777216 4194304 float sum -1 743.7 22.6 42.4 0
134217728 33554432 float sum -1 4891.2 27.4 51.4 0
536870912 134217728 float sum -1 9814.2 54.7 102.6 0
1073741824 268435456 float sum -1 17234.8 62.3 116.8 0
8589934592 2147483648 float sum -1 109842.1 78.2 146.6 0
size: Message size in bytes (this is what one GPU is contributing -- the total AllReduce processes N x this).
time: Elapsed microseconds for the operation. Includes all N GPUs completing the full AllReduce.
algbw (algorithm bandwidth): size / time. Raw throughput -- bytes moved per second per GPU.
busbw (bus bandwidth): algbw x 2(N-1)/N. This is the number that matters. It corrects algbw by the mathematical efficiency of the ring algorithm, giving you the effective utilisation of the NIC bandwidth. At large N, busbw ~= 2 x algbw. This is what you compare against line rate.
#wrong: The number of gradient values that did not match after the AllReduce. This must always be zero. A non-zero value means data corruption -- training would produce silently wrong results. Causes: ECC error, RDMA memory corruption, fabric bit-flip. Any non-zero value requires immediate investigation before the cluster is used for training.
What "expected busbw" means for different scales
The theoretical maximum busbw equals the NIC line rate (400 Gb/s = 50 GB/s per NIC). In practice, protocol overhead, algorithm efficiency, and multiple NICs sharing a path reduce this. Healthy targets:
| Cluster size | Topology | Expected busbw (large tensors) | Alarm threshold |
|---|---|---|---|
| 8 GPUs, 1 node | NVLink only | 800–880 GB/s (NVLink) | < 600 GB/s |
| 32 GPUs, 4 nodes | 2-stage leaf-spine | 350–390 GB/s | < 280 GB/s |
| 256 GPUs, 32 nodes (BasePOD) | 2-stage leaf-spine | 130–160 GB/s | < 100 GB/s |
| 256 GPUs, SuperPOD | 3-stage | 280–340 GB/s | < 200 GB/s |
| 2048 GPUs, large cluster | 5-stage | 200–280 GB/s | < 150 GB/s |
Why busbw decreases with cluster size: More GPUs means more hops, more protocol overhead, and more QP management. A 256-GPU SuperPOD will always show lower busbw than a 32-GPU BasePOD -- this is expected and does not indicate a problem.
The small-message / large-message distinction
The output table covers 8 bytes through 8 GB. These are not equivalent:
- Small tensors (< 1 MB): NCCL uses LL (Low Latency) protocol. Latency dominates. busbw is low -- not because the fabric is slow, but because the per-message overhead is large relative to the message. This is normal.
- Large tensors (> 128 MB): NCCL uses SIMPLE protocol. Bandwidth dominates. This is where you compare against line rate expectations.
When an ML engineer says "training is slow," first ask: what size are the gradient tensors_2 If the model produces many small gradients (small layer sizes), you may be in the latency-dominated regime and load balancing / hop count improvements will not help as much as algorithmic changes.