NCCL has no configuration file. Everything is controlled via environment variables set in the job launcher before the training process starts. A single wrong variable -- or a missing one -- is the most common cause of "NCCL is running at 1% of expected bandwidth" and produces no obvious error.

The variables that matter for network engineers

NCCL_IB_HCA -- which RDMA HCA devices NCCL should use.

If not set, NCCL scans all available devices. On a DGX H100 with 8 ConnectX-7 NICs in Ethernet/RoCEv2 mode, those NICs present as mlx5_0 through mlx5_7 at the RDMA layer. NCCL should find all 8.

The failure mode: if NCCL_IB_HCA is set to an IB device name like mlx5_bond_0 (a bonded InfiniBand interface name used on some systems) but your NICs are in Ethernet mode presenting as mlx5_0, NCCL finds nothing and falls back to TCP socket transport. Training continues -- at 2-4 GB/s instead of 400 GB/s. No error. The only sign is NCCL_DEBUG=INFO output showing transport selected: socket.

# Correct for a DGX H100 in RoCEv2 mode -- specify all 8 NICs explicitly
export NCCL_IB_HCA=mlx5_0:1,mlx5_1:1,mlx5_2:1,mlx5_3:1,mlx5_4:1,mlx5_5:1,mlx5_6:1,mlx5_7:1

# The :1 suffix specifies port 1 of that HCA (ConnectX-7 is single-port per NIC)

# Verify what NCCL sees by checking RDMA device names on the DGX:
rdma link show
# Each line shows: link mlx5_N/1 state ACTIVE ...
# The mlx5_N names are what NCCL_IB_HCA should reference

NCCL_SOCKET_IFNAME -- which network interface to use for control messages (not data).

NCCL uses a TCP socket for out-of-band coordination between processes -- exchanging QP information during initialisation, synchronising barriers. This must point to a management interface (typically the DGX management ethernet, eno1 or similar), not to the RDMA training interfaces.

If this points to an RDMA interface, control messages compete with data traffic and QP establishment can fail unpredictably.

export NCCL_SOCKET_IFNAME=eno1     # management ethernet
# Do NOT set to eth0 (RDMA interface)

NCCL_DEBUG -- the most important troubleshooting variable.

export NCCL_DEBUG=INFO    # Shows transport selected, NIC count, algorithm, QP establishment
export NCCL_DEBUG=WARN    # Default -- only shows warnings and errors
export NCCL_DEBUG=TRACE   # Shows every message -- extremely verbose, for deep bugs only

INFO level is what you always enable when diagnosing NCCL performance issues. The key lines to look for:

[0] NCCL INFO transport selected: SHM    <- using shared memory (intra-node only)
[0] NCCL INFO transport selected: NET    <- using network (this is what you want)
[0] NCCL INFO Channel 00: 0 [mlx5_0]    <- NIC mlx5_0 assigned to channel 0
[0] NCCL INFO Using 8 channels, Ring    <- 8 NICs active, ring algorithm

[0] NCCL INFO transport selected: socket <- FALLBACK -- no RDMA found

If you see socket transport, NCCL found no RDMA devices. Fix NCCL_IB_HCA.

NCCL_IB_QPS_PER_CONNECTION -- Queue Pairs per GPU-to-GPU connection.

Default 1. Each QP creates one RDMA flow. With RoCEv2 and per-packet load balancing (RSHP), more QPs means more flows, which means better ECMP distribution across spine links. On Spectrum-X fabrics with RSHP enabled:

export NCCL_IB_QPS_PER_CONNECTION=4
# Creates 4 QPs per GPU pair = 4 distinct ECMP flows per pair
# On a 32-node cluster: 32 x 31 x 4 = 3,968 active flows across 8 rails
# Dramatically better path diversity than the default 1 QP per connection

NCCL_IB_GID_INDEX -- which Global Interface Identifier to use for RoCEv2.

On a ConnectX-7 in Ethernet mode, multiple GIDs exist (one per IP configuration). The RoCEv2 GID is typically at index 3. If the wrong GID index is selected, NCCL may attempt to use an IPv4-mapped IPv6 address (::ffff:x.x.x.x) that is not routable on your fabric.

export NCCL_IB_GID_INDEX=3    # Standard for RoCEv2 on Ethernet-mode ConnectX-7
# Verify with: ibv_devinfo -d mlx5_0 | grep GID

NCCL_P2P_DISABLE=1 -- disables NVLink peer-to-peer.

Forces all GPU-to-GPU communication through the host. Use only for debugging when NVLink is suspected faulty. Never set in production -- it reduces intra-node bandwidth from 900 GB/s to ~30 GB/s.

NCCL_ALGO and NCCL_PROTO -- algorithm and protocol override.

export NCCL_ALGO=RING      # Force ring algorithm
export NCCL_ALGO=TREE      # Force tree algorithm
export NCCL_PROTO=LL       # Low Latency -- best for small tensors
export NCCL_PROTO=LL128    # Default -- optimised for most tensor sizes
export NCCL_PROTO=SIMPLE   # Lowest overhead -- best for very large tensors