DCQCN (Data Center Quantized Congestion Notification) is the dominant congestion control algorithm in production RoCEv2 deployments today. It was jointly developed by Microsoft and Mellanox (now NVIDIA) and is described in RFC 8938 and the original SIGCOMM 2015 paper. If you operate a NVIDIA Spectrum-X or ConnectX-7 deployment, DCQCN is what is running.

The feedback loop: three actors

DCQCN involves three participants that must be correctly configured simultaneously:

Switch ──ECN mark──► Receiver NIC ──CNP──► Sender NIC ──rate reduce──► (lower TX rate)

Step 1 — ECN marking at the switch. The switch egress queue maintains a running byte count. When the queue depth crosses the ECN min_threshold, it begins marking packets with the CE (Congestion Experienced) bit in the IP header using a probabilistic WRED curve. At max_threshold, every packet is marked. The switch does not drop packets at this stage — it marks them. This is the critical distinction from traditional TCP drop-based CC.

The marking happens in hardware at line rate. No packet inspection, no CPU involvement. On Spectrum-4, this runs at wire speed across all 128 ports simultaneously.

# Spectrum-X / NVUE: verify ECN profile
nv show qos ecn profile roce

# Expected output:
#   min-threshold:  500KB
#   max-threshold:  1500KB
#   probability:    100  (at max-threshold)

Step 2 — CNP generation at the receiver. The receiver NIC (ConnectX-7) inspects incoming packets. When it sees a packet with CE=1, it generates a CNP (Congestion Notification Packet) and sends it back to the sender. One CNP is generated per flow per 50 microseconds by default — not one per marked packet — to prevent CNP storms under heavy congestion.

The CNP carries the QP (Queue Pair) identifier of the congested flow. The CNP itself must be placed in the highest-priority queue (TC6, DSCP 48 by default) so it is never held by PFC pause frames. If CNP gets paused in a lossless queue, DCQCN cannot function.

# ConnectX-7: verify CNP DSCP
mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep CNP_DSCP
# Must match the switch DSCP trust profile:
nv show qos trust dscp-map 48

Step 3 — Rate reduction at the sender NIC. Upon receiving a CNP, the sender NIC applies the DCQCN rate reduction algorithm:

α(t) = (1 - g) × α(t-1) + g × 1    # when CNP received: alpha increases
Rate = Rate × (1 - α/2)              # multiplicative decrease

Where α is the congestion estimate (0 = no congestion, 1 = maximum congestion) and g is the exponential weight (default: 1/256 on ConnectX-7). The alpha filter ensures that a single CNP does not cause a drastic rate cut while persistent congestion progressively reduces the rate toward the minimum.

Rate recovery uses two mechanisms operating in parallel:

• Byte Counter (BC): After sending a fixed number of bytes (dcqcn_bc) without a CNP, the rate increases by a fixed additive increment Rai (Rate Additive Increase).
• Timer (RT): A timer fires every dcqcn_rl_timer microseconds, also incrementing rate by Rai. This ensures recovery even in low-bandwidth flows where the byte counter rarely fires.

# Key DCQCN parameters on ConnectX-7:
mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep -i dcqcn

# Critical parameters:
# DCQCN_RL_AI       = Rate Additive Increase (default: 5 MB/s, too low = slow recovery)
# DCQCN_RL_HAI      = Hyper-Additive Increase (fast rate increase phase)
# DCQCN_NP_CNP_DSCP = DSCP value placed on CNP (must match switch trust)
# DCQCN_MIN_RATE    = Floor rate even under maximum congestion (default: 4 MB/s)
# DCQCN_GD          = Denominator of alpha update factor g

The two-RTT feedback problem

DCQCN has an inherent delay between congestion onset and sender response:

Time 0:   Queue at switch builds up
Time +1RTT: CE-marked packets arrive at receiver; CNP is generated
Time +2RTT: CNP arrives at sender; rate is reduced

Under a sharp incast (all 8 GPUs in a DGX H100 transmitting simultaneously), 2 RTTs at 800ns each = 1.6 microseconds. In that 1.6µs, a 400GbE port can deliver: 400 Gbps × 1.6µs = 80 KB of data. With 8 sources incast, 640 KB lands before the rate reduction begins. This is why ECN min_threshold is typically set to 500 KB — the algorithm cannot react fast enough to prevent the first fill.

This fundamental two-RTT delay is what motivates Swift and HPCC.