Understanding the algorithm is one thing. Knowing which config file to edit at 2am when the training job is degraded is another. This act is the field engineer's reference.

ConnectX-7 DCQCN knob map

All DCQCN parameters on ConnectX-7 are accessible via mlxconfig. The key parameters and their operational impact:

# Read all DCQCN parameters:
mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep -E "DCQCN|CNP|ECN"

# Parameter reference:
# DCQCN_RL_AI      (default:  1)  Rate Additive Increase per timer tick
#                                  Too low = slow recovery after congestion clears
#                                  Production value: 50–100 MB/s
#
# DCQCN_RL_HAI     (default: 50)  Hyper-Additive Increase (fast phase)
#                                  Kicks in after dcqcn_rl_bc_time_period
#
# DCQCN_GD         (default:  7)  Alpha update weight denominator: g = 1/(2^GD)
#                                  Higher GD = smoother alpha (less reactive)
#
# DCQCN_NP_MIN_TIME_BETWEEN_CNPS  (default: 50µs)
#                                  CNP rate limiter. Too low = CNP storm.
#                                  Too high = congestion undetected for 50µs windows.
#
# DCQCN_NP_CNP_DSCP (default: 48) DSCP value in CNP packets
#                                  MUST match nv show qos trust dscp-map 48 on switch

# Apply a corrected Rate Additive Increase value:
mlxconfig -d /dev/mst/mt4129_pciconf0 s DCQCN_RL_AI=50

Spectrum-X ECN threshold tuning

The ECN thresholds are the switch-side knobs that complement the NIC parameters:

# View current ECN profile:
nv show qos ecn profile roce

# Set thresholds (500KB min / 1500KB max is a good starting point for 400GbE):
nv set qos ecn profile roce min-threshold 500KB
nv set qos ecn profile roce max-threshold 1500KB
nv config apply

# Verify ECN marking is happening:
nv show interface swp1 counters | grep ecn
# ecn_marked_packets should be non-zero under load

Threshold sizing rule: min_threshold should be approximately:

min_threshold ≈ BDP/4 = (bandwidth × RTT) / 4
             ≈ (400 Gbps × 1.6µs) / 4  ≈  80 KB

For 400GbE with 1.6µs RTT, 80 KB is the BDP/4 target. Many deployments use 500 KB to accommodate multiple simultaneous flows — but if your flows are small (NCCL messages < 1 MB), 500 KB may be too high: the queue fills and starts dropping before ECN ever marks.

The CNP trust chain — the most common misconfiguration

In production, the single most common DCQCN failure mode is the CNP trust chain breaking. Here is the complete chain with the failure points labelled:

Sender NIC: generates data packets with DSCP 26 (RoCE traffic)
    ↓
[FAILURE POINT A]: Switch doesn't trust DSCP 26 → reclassifies to TC0 (lossy queue)
    ↓ (if A is correct)
Switch queue fills → ECN marks CE bit in packets at DSCP 26
    ↓
Receiver NIC: sees CE bit → generates CNP with DSCP 48
    ↓
[FAILURE POINT B]: Switch doesn't trust DSCP 48 → CNP lands in lossy TC0
    ↓ (CNP gets dropped or paused in lossy queue)
    ↓ (No CNP arrives at sender → sender never reduces rate)
    ↓ (Congestion escalates → PFC triggers → potential pause storm)

Fix:

# Verify DSCP trust on leaf switch:
nv show qos trust dscp-map 26    # Should be TC3 (priority 3, lossless)
nv show qos trust dscp-map 48    # Should be TC6 (priority 6, high priority)

# If missing, set them:
nv set qos trust dscp-map 26 traffic-class 3
nv set qos trust dscp-map 48 traffic-class 6
nv config apply

Hardware architecture: where CC code runs

It is worth being precise about the execution context of each algorithm:

The operational implication: any CC algorithm that runs in kernel software (TIMELY with software timestamps, early DCQCN implementations) has a floor latency 100–1000x higher than hardware implementations. For sub-5 µs AI fabric RTTs, kernel-software CC is not viable.