What happens when the NIC receives a CNP

DCQCN (Data Center Quantized Congestion Notification) is the rate control algorithm implemented in ConnectX-7 NIC firmware. It runs entirely in the NIC -- no kernel involvement, no CPU cycles. When a CNP arrives at the NIC:

Phase 1 -- Rate decrease The NIC immediately reduces the current rate of the congested RDMA flow by a factor alpha. Alpha is maintained as a running average of how often CNPs are received -- frequent CNPs -> higher alpha -> more aggressive rate reduction.

New rate = Current rate x (1 - alpha/2)

The minimum rate is bounded -- the NIC will not reduce below a configured floor to prevent complete starvation.

Phase 2 -- Recovery After a timer expires with no new CNPs (no congestion signal), the NIC begins recovering rate:

• For the first few recovery stages: rate increases by a fixed additive increment (fast recovery)
• After sufficient recovery stages: rate increases multiplicatively up to the target rate (hyper-active increase)
• If a CNP arrives during recovery: immediately return to Phase 1

This additive-increase/multiplicative-decrease (AIMD) behaviour is the same principle as TCP congestion control, but implemented in NIC hardware at microsecond timescales rather than milliseconds.

New rate = Current rate × (1 − alpha/2)
Alpha update: alpha = (1 − g) × alpha + g × 1
where g = 1/256 (typical), alpha starts at 1/64

The DCQCN parameters and where they live

The DCQCN parameters are configured via mlnx_qos on the DGX host and via switch ECN threshold configuration. The most important parameters:

On the DGX host (mlnx_qos):

# DGX host terminal (green prompt)

# View current DCQCN configuration
mlnx_qos -i eth0 --dcqcn-params

# Key parameters:
# rp_clamp_tgt_rate     : whether to clamp the target rate on CNP receipt
# rp_time_reset         : time interval for rate increase after no CNPs
# rp_byte_reset         : byte count for rate increase after no CNPs
# rp_threshold          : CNP threshold for alpha increase
# rp_alpha_to_factor    : alpha scaling factor
# rp_initial_alpha_value: initial alpha on first CNP

On the switch (ECN thresholds):

# Leaf switch terminal (blue prompt)

show dcb ets

Interface swp1 -- ETS Configuration
  Traffic class  Priority  Bandwidth  Algorithm
  TC0            0,1,2     30%        ETS
  TC3 (RoCE)     3         50%        Strict Priority
  TC7 (mgmt)     7         20%        ETS

  ECN marking:    enabled (DSCP 26 in this deployment)
  ECN min-thresh: 150000 bytes
  ECN max-thresh: 1500000 bytes
  ECN probability: 100%
  DCQCN:          active

The ECN min-thresh and ECN max-thresh values are the parameters you are reading in this output. A common mistake is deploying a fabric with the default ECN thresholds from a different product -- Spectrum-X defaults differ from older Spectrum switches, and using wrong thresholds can cause either too-early marking (constant CNPs, senders spending all their time in rate-reduction mode) or too-late marking (ECN fires too close to the PFC threshold, giving no time for rate reduction to work).

How to verify DCQCN is working

The rx_ecn_marked counter in ethtool -S eth0 shows how many packets the NIC has received with the CE bit set. If DCQCN is active and congestion is present, this counter should be growing:

# DGX host terminal
ethtool -S eth0 | grep ecn
  rx_ecn_marked:  2947   <- CNPs received, DCQCN is reacting
  tx_ecn_marked:  0      <- NIC is not itself congested

If congestion is present (buffer utilisation elevated on the switch) but rx_ecn_marked is zero, ECN marking is not configured on the switch side -- this is the Lab 2 fault scenario. The fabric is running on PFC alone, which means every congestion event triggers pauses rather than rate reductions.