This is the act that connects the previous seven chapters to the application layer. A specific NCCL symptom maps to a specific diagnostic path in the fabric.

Decision tree: from NCCL symptom to fabric command

Symptom A: busbw is < 5% of expected (e.g., 2 GB/s on a 400G fabric)

Root cause: NCCL selected socket transport (TCP fallback). RDMA was not found.

# Step 1: Confirm transport selection
NCCL_DEBUG=INFO mpirun ... ./all_reduce_perf ...
# Look for: "transport selected: socket"

# Step 2: Check RDMA device names on DGX
rdma link show
# Expected: link mlx5_0/1 state ACTIVE ... type RoCE

# Step 3: Verify NCCL_IB_HCA matches what rdma link show shows
echo $NCCL_IB_HCA
# If unset or wrong -> fix it

# Step 4: Confirm NICs are in the correct mode
ibstat | grep "Link layer"
# Must show: Link layer: Ethernet (for RoCEv2) or InfiniBand (for IB)

Fix: Set NCCL_IB_HCA to the correct device names from rdma link show. Re-run nccl-tests.

Symptom B: busbw is 40-70% of expected, NCCL using NET transport (RDMA active)

Root cause: Fabric congestion or load balancing failure. RDMA is working but something is slowing it down.

# Step 1: Check spine link utilisation (fabric-level check first)
# On spine switch terminal:
show interface counters
# Compare link utilisation: is it even across all ports_2
# If some spine links at 85% and others at 5% -> load balancing problem (Ch6 diagnosis)
# If spine links are even -> congestion problem (Ch5 diagnosis)

# Step 2: Check for drops or sustained pauses
# On leaf switch terminal:
show interface counters
# Output drops: 0_2 If non-zero -> PFC or ECN misconfiguration
# PFC pause frames: growing rapidly_2 -> sustained congestion

# Step 3: Verify PFC and ECN are correctly configured
show dcb pfc
# PFC enabled priorities: 3 (must match DSCP marking)

show dcb ets
# ECN marking: enabled_2 DCQCN: active_2

# Step 4: Check NIC-level experience
# On DGX terminal:
ethtool -S eth0
# rx_pfc_pause_frames growing constantly -> sustained PFC storm
# rx_ecn_marked: 0 despite congestion -> ECN not configured
# tx_dropped: non-zero -> drops occurring despite PFC

# Step 5: Verify DCQCN is not over-reacting
show roce
# Check DCQCN state and DSCP marking

Fix: Based on which step reveals the issue -- apply the Chapter 5, 6, or 7 fix. Re-run nccl-tests after each fix.

Symptom C: busbw is 80-95% of expected (small degradation)

Root cause: Normal for large clusters. Protocol overhead and DBT algorithm efficiency.

# Confirm this is expected, not a problem
# Check cluster size and topology match the expected busbw table above
# Verify #wrong is 0 (no corruption)

# If you want to improve:
# Try forcing RING algorithm for comparison (valid only for small clusters)
NCCL_ALGO=RING mpirun ... ./all_reduce_perf -b 512M -e 8G ...
# If RING gives higher busbw -> cluster is small enough that ring is more efficient
# Set NCCL_ALGO=RING in production job launcher

# Check QPS_PER_CONNECTION
echo $NCCL_IB_QPS_PER_CONNECTION
# If 1 (default) -> try 4 with RSHP-enabled Spectrum-X fabric
NCCL_IB_QPS_PER_CONNECTION=4 mpirun ... ./all_reduce_perf ...
# More QPs = more ECMP flows = better path diversity

Symptom D: NCCL timeout errors mid-training (QP error state)

Root cause: A packet was dropped, causing a PSN gap, causing a QP to enter error state. As established in Chapter 3: a dropped packet causes a PSN gap, the QP enters error state, and the GPU on that rail cannot participate in AllReduce until the QP is reset.

# Step 1: Check for QP errors
ibstat | grep -A 5 "State:"
# Look for: State: Error -- the rail that suffered the packet drop

# Step 2: Find the root cause of the packet drop
# On DGX terminal:
ethtool -S eth0
# tx_dropped: non-zero_2 -> drops are occurring
# rx_pfc_pause_frames: 0 despite drops -> PFC is not configured or on wrong priority

# Step 3: Check switch logs for the time of the timeout
# On UFM server:
show ufm events --severity ERROR --since "2025-03-23 14:00"
# Look for: link flap, symbol error spike, or port error-disabled events

# Step 4: Correlate timing
# QP timeout at 14:23:15 in NCCL log_2
# UFM shows link flap on Leaf-Switch-3 port 18 at 14:23:14_2
# These are the same event seen from different layers.

# Step 5: Check symbol errors
show ib counters
# SymbolErrors > 0 and growing -> physical layer issue (cable, transceiver, SFP)

Fix: Replace the faulty cable or transceiver on the identified rail. If software-caused (PFC misconfiguration causing drops), fix PFC. Reset the QP by restarting NCCL (restart the training job).

Symptom E: busbw good on nccl-tests, degraded during actual training

Root cause: nccl-tests uses controlled message sizes and clean traffic. Real training has mixed tensor sizes, concurrent pipeline parallelism traffic, and variable timing.

# Step 1: Profile tensor sizes during actual training
NCCL_DEBUG=INFO training_script.py 2>&1 | grep "AllReduce size"
# What sizes are actually being transferred_2

# Step 2: Run nccl-tests at the actual tensor sizes your training uses
./all_reduce_perf -b <actual_min_size> -e <actual_max_size> -f 1
# Does the degradation reproduce here_2

# Step 3: Check DCQCN rate control
mlnx_qos -i eth0
# Is the NIC injecting at full rate_2 Or has DCQCN throttled it_2
# rp_rate: should be near 400G during AllReduce
# If significantly below -> DCQCN is over-reacting

# Step 4: Check concurrent traffic
# Is pipeline parallelism traffic competing with AllReduce on the same links_2
show interface counters  # on leaf switch, compare uplink utilisation during training
# AllReduce is bursty; pipeline is steady. If both peak simultaneously,
# the fabric may be temporarily overloaded even though nccl-tests (which only runs
# AllReduce) looks clean.