Chapter 5 taught DCQCN mechanics. This act connects that theory to what the knobs actually do when NCCL is running AllReduce.

When DCQCN is your problem

DCQCN is working correctly if:

• ethtool -S eth0 shows rx_ecn_marked growing (switch is signalling congestion)
• ethtool -S eth0 shows tx_dropped: 0 (DCQCN is preventing drops)
• busbw is within 85-100% of expected

DCQCN may be over-reacting if:

• rx_ecn_marked is growing rapidly
• busbw is below 70% of expected
• show interface counters shows no drops and no congestion
• The NICs are being told to slow down even though the fabric is not congested

The DCQCN over-reaction pattern: a brief AllReduce burst triggers ECN marks, the NIC reduces its injection rate, the burst passes, but the NIC stays at reduced rate for rp_time_reset milliseconds. If the next AllReduce burst begins before the NIC recovers, it starts already rate-limited. Over many training steps this compounds into sustained underutilisation.

The two parameters that matter

rp_time_reset -- time at reduced rate before beginning recovery. Default 300ms.

For AllReduce workloads with a burst every 5-50ms (typical for large models), a 300ms recovery window means the NIC may never fully recover between barriers. Reducing to 50-100ms allows the NIC to recover fully between AllReduce steps.

# Check current value
mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep RP_TIME_RESET
# Output: RP_TIME_RESET  300

# Reduce for AllReduce workloads
mlxconfig -d /dev/mst/mt4129_pciconf0 set RP_TIME_RESET=100
# Apply to all 8 NICs on the DGX

rp_byte_reset -- bytes transmitted at reduced rate before beginning recovery.

Works alongside rp_time_reset -- whichever triggers first initiates recovery. For AllReduce tensors in the 1-8 GB range, the default byte reset value may allow the NIC to recover mid-transfer rather than waiting for the time threshold. Tuning this to match your typical tensor size can help.

mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep RP_BYTE_RESET
mlxconfig -d /dev/mst/mt4129_pciconf0 set RP_BYTE_RESET=131072  # 128 KB

When NOT to tune

If:

• ethtool -S eth0 shows rx_ecn_marked is low (switch is not signalling congestion frequently)
• Spine links show even utilisation with show interface counters
• busbw is within 85% of expected

Then DCQCN is not your problem. Do not tune. The degradation is elsewhere -- check topology, check algorithm selection, check NCCL_IB_HCA.