ProDeploy — Learn the fabric that keeps AI clusters alive

Lab 5 models the most common real-world NCCL failure: the cluster passes all hardware checks, fabric is correctly configured, but nccl-tests shows 3 GB/s busbw instead of 380 GB/s. The ML team is blocked. Training has not started.

The scenario

A 16-node DGX H100 cluster has just been handed over by the infrastructure team. They confirm:

All links active (ibstat shows all 8 rails ACTIVE on all nodes)
PFC and ECN configured (show dcb pfc and show dcb ets both green)
Spine utilisation even during test traffic
#wrong: 0 in the nccl-tests output

But busbw for 1GB tensors is 3.1 GB/s -- roughly 1% of the 380 GB/s expected for a 16-node cluster.

Your task: diagnose why NCCL is achieving 1% of expected bandwidth despite all hardware appearing healthy. Fix the configuration. Re-run and verify.

The diagnostic path

Step 1: Check NCCL transport selection

# On DGX terminal (green prompt)
nccl-debug --transport    <- ProDeploy simulator command
                          # Real equivalent: NCCL_DEBUG=INFO ./all_reduce_perf ... 2>&1 | grep "transport selected"

Output shows:

[0] NCCL INFO Channel 00: transport selected: socket
[0] NCCL INFO No IB devices found matching NCCL_IB_HCA
[0] NCCL INFO Falling back to socket transport

NCCL found no RDMA devices. It is using TCP socket transport -- hence 3 GB/s.

Step 2: Check what RDMA devices actually exist

rdma link show

Output:

link mlx5_0/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth0
link mlx5_1/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth1
link mlx5_2/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth2
link mlx5_3/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth3
link mlx5_4/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth4
link mlx5_5/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth5
link mlx5_6/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth6
link mlx5_7/1 state ACTIVE physical_state LINK_UP  type RoCE  netdev eth7

All 8 NICs are active. But NCCL said "No IB devices found matching NCCL_IB_HCA."

Step 3: Check what NCCL_IB_HCA is set to

show nccl env    <- ProDeploy simulator command
                 # Real equivalent: env | grep NCCL

Output:

NCCL_IB_HCA:     mlx5_bond_0
NCCL_SOCKET_IFNAME: eth0
NCCL_IB_GID_INDEX: 3

NCCL_IB_HCA=mlx5_bond_0 -- this is an InfiniBand bonded interface name used on IB deployments. These DGX nodes are in RoCEv2 mode with individual NICs named mlx5_0 through mlx5_7. NCCL scanned for mlx5_bond_0, found nothing, and fell back to TCP.

Additionally: NCCL_SOCKET_IFNAME=eth0 -- this is using the RDMA training interface for control messages, which will cause QP establishment failures under load.

Step 4: Fix both environment variables

set nccl ib-hca mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7    <- ProDeploy simulator command
set nccl socket-ifname eno1                                                      <- ProDeploy simulator command

In production, these are environment variables set before launching the training job:
export NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1
export NCCL_SOCKET_IFNAME=eno1
# Then relaunch your training job
The simulator uses shorthand set nccl commands to model the state change. The environment variables themselves work identically -- they are read by NCCL at process startup.

Step 5: Re-run nccl-tests and verify

run nccl-tests

Output:

[0] NCCL INFO Channel 00: transport selected: NET
[0] NCCL INFO Using 8 channels, Double Binary Tree
[0] NCCL INFO 128 GPUs connected, all paths verified

  1073741824   float   sum   -1   18243.1    58.8   110.2      0
  8589934592   float   sum   -1  115822.3    74.1   138.9      0

busbw of 138 GB/s on a 128-GPU cluster (16 nodes x 8 GPUs) is within expected range for this topology. #wrong: 0.