Every AI cluster commissioning process runs nccl-tests before handover to production. It is the standard benchmark suite that validates the full stack: NCCL configuration, NIC drivers, RDMA transport, switch fabric, and topology -- simultaneously.

Running allreduce_perf

# On DGX node A -- run across all nodes in the cluster
mpirun --hostfile /etc/mpi/hostfile \
  -np 256 \
  --map-by ppr:8:node \
  -x NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1 \
  -x NCCL_DEBUG=INFO \
  -x NCCL_IB_GID_INDEX=3 \
  ./build/all_reduce_perf \
    -b 8 \           # Start at 8 bytes
    -e 8G \          # End at 8 GB
    -f 2 \           # Double message size each step
    -g 1 \           # 1 GPU per process
    -c 1             # Check results for correctness

Flags explained:

• -b 8 -e 8G -f 2: Sweeps from 8 bytes to 8 GB, doubling each step -- reveals performance across all tensor sizes
• -g 1: Each MPI process owns 1 GPU
• -c 1: Enables the #wrong correctness check -- critical, must be zero

Before you believe any number: pin the clocks

A busbw figure that moves by ten percent between runs is not telling you about the fabric. It is telling you the GPUs boosted differently.

Pin the application clocks before the first run and unpin them afterwards:

# Pin memory and SM clocks (values are per-generation; these are H100)
sudo nvidia-smi -ac 2619,1980
# Applications clocks set to (MEM 2619, SM 1980)

# ...run the benchmark...

# Return to the default boost behaviour
sudo nvidia-smi -rac

Without this, run-to-run variance is thermal and power behaviour, and it will be attributed to the network -- because the network is what everyone is looking at. Pin the clocks, and any remaining variance is real.

Two related preconditions, for the same reason:

• Run on a whole node. Under Slurm, --exclusive. A busbw number measured while another job shares the node is not a fabric measurement.
• Health-check the node first. Run the node's own health gate (Chapter 3, Act 1) before benchmarking it. Benchmarking a node with an open hardware alert wastes the run and, worse, produces a number somebody will quote later.

Isolating the network from everything else: the loopback test

Here is the test that is missing from most runbooks, and it needs exactly one node.

all_reduce_perf on a single node normally never touches the NIC -- NCCL routes the traffic over NVLink or shared memory because those are faster. That is the right behaviour for a real job and the wrong behaviour for a diagnosis, because it means a single-node result tells you nothing about the network stack.

You can force the traffic out of the NIC and back:

srun -N 1 --exclusive \
  --container-image=nvcr.io/nvidia/pytorch:24.10-py3 \
  bash -c '
    export NCCL_P2P_DISABLE=1      # no peer-to-peer over PCIe
    export NCCL_SHM_DISABLE=1      # no shared-memory transport
    export NCCL_NVLS_ENABLE=0      # no NVLink SHARP
    export NCCL_MNNVL_ENABLE=0     # no multi-node NVLink
    export UCX_TLS=tcp
    export NCCL_DEBUG=INFO
    ./build/all_reduce_perf -b 8 -e 8G -f 2 -g 8
  '

With every faster path disabled, NCCL has nowhere to go but the NIC. On an HDR (200 Gb/s) fabric a healthy node returns roughly 175 Gbps on this test.

What that buys you is a clean bisection:

That third row is the valuable one. It tells you to stop opening switch counters and start on the host, using one node, with no switch access and no maintenance window. It is the difference between an afternoon of fabric archaeology and a ten-minute answer.

A note on containers

Every example here launches through Slurm's container integration, which arranges host networking and IPC for you. If you run nccl-tests under plain Docker instead, you have to arrange it yourself:

docker run --rm --net=host --uts=host --ipc=host --pid=host \
  --device=/dev/infiniband --cap-add=IPC_LOCK \
  --gpus all <image> ...

Omitting --net=host or --ipc=host is a common and confusing failure: NCCL initialises, the job runs, and the numbers are terrible -- because the container's network namespace hid the RDMA devices and NCCL quietly fell back to TCP. The NCCL_DEBUG=INFO output will name the transport it chose. Read it before blaming the fabric.

Reading the output table -- field by field

#                                                              out-of-place
#       size         count    type    redop    root     time   algbw   busbw  #wrong
      8388608       2097152   float     sum      -1    412.5    20.3    38.1      0
     16777216       4194304   float     sum      -1    743.7    22.6    42.4      0
    134217728      33554432   float     sum      -1   4891.2    27.4    51.4      0
    536870912     134217728   float     sum      -1   9814.2    54.7   102.6      0
   1073741824     268435456   float     sum      -1  17234.8    62.3   116.8      0
   8589934592    2147483648   float     sum      -1 109842.1    78.2   146.6      0

size: Message size in bytes (this is what one GPU is contributing -- the total AllReduce processes N x this).

time: Elapsed microseconds for the operation. Includes all N GPUs completing the full AllReduce.

algbw (algorithm bandwidth): size / time. Raw throughput -- bytes moved per second per GPU.

busbw (bus bandwidth): algbw x 2(N-1)/N. This is the number that matters. It corrects algbw by the mathematical efficiency of the ring algorithm, giving you the effective utilisation of the NIC bandwidth. At large N, busbw ~= 2 x algbw. This is what you compare against line rate.

#wrong: The number of gradient values that did not match after the AllReduce. This must always be zero. A non-zero value means data corruption -- training would produce silently wrong results. Causes: ECC error, RDMA memory corruption, fabric bit-flip. Any non-zero value requires immediate investigation before the cluster is used for training.

What "expected busbw" means for different scales

The theoretical maximum busbw equals the NIC line rate (400 Gb/s = 50 GB/s per NIC). In practice, protocol overhead, algorithm efficiency, and multiple NICs sharing a path reduce this. Healthy targets:

Why busbw decreases with cluster size: More GPUs means more hops, more protocol overhead, and more QP management. A 256-GPU SuperPOD will always show lower busbw than a 32-GPU BasePOD -- this is expected and does not indicate a problem.

The small-message / large-message distinction

The output table covers 8 bytes through 8 GB. These are not equivalent:

• Small tensors (< 1 MB): NCCL uses LL (Low Latency) protocol. Latency dominates. busbw is low -- not because the fabric is slow, but because the per-message overhead is large relative to the message. This is normal.
• Large tensors (> 128 MB): NCCL uses SIMPLE protocol. Bandwidth dominates. This is where you compare against line rate expectations.

When an ML engineer says "training is slow," first ask: what size are the gradient tensors_2 If the model produces many small gradients (small layer sizes), you may be in the latency-dominated regime and load balancing / hop count improvements will not help as much as algorithmic changes.