ECN marking on Spectrum-4 happens entirely in hardware at wire speed. When a packet arrives on an ingress port destined for TC3, the ASIC checks the current TC3 egress queue depth against two thresholds: min_threshold and max_threshold. Between these thresholds, marking probability rises linearly from 0% to 100%. Above max_threshold, every packet is marked.

WRED vs pure ECN: WRED (Weighted Random Early Detection) both marks and drops packets in the congestion zone ”” it was designed for TCP, which relies on dropped packets as a congestion signal. For RoCEv2 with DCQCN, the correct mode is pure ECN: mark only, never drop in the lossless queue. Cumulus Linux default for RoCE is pure ECN on TC3.

ECN threshold sizing:

The standard sizing rule is: min_threshold ≈ BDP/4

Where BDP = bandwidth × RTT (bandwidth-delay product).

At 400GbE (50 GB/s) with a fabric RTT of 1.6µs:

BDP = 50,000,000,000 bytes/s × 0.0000016 s = 80,000 bytes = ~80 KB
BDP/4 = 20 KB

In practice, most Spectrum-X deployments use min_threshold = 500 KB and max_threshold = 1.5 MB. The reasoning: a single DGX H100 node runs 8 RoCEv2 flows simultaneously (one per ConnectX-7 port). With 8 concurrent flows, queue build-up can be 8× per-flow. Too-low thresholds cause ECN to fire on transient micro-bursts. Too-high thresholds mean ECN never fires and PFC becomes the only congestion mechanism.

Verification commands:

leaf-01# nv show qos ecn profile roce
Profile: roce
  Min Threshold:  500000 bytes (488 KB)
  Max Threshold:  1500000 bytes (1.46 MB)
  Mark Probability: linear
  Mode: ECN only (no drops)
  Applied to:     TC3 egress on all RoCE interfaces

leaf-01# ethtool -S swp1 | grep ecn
  tx_ecn_marked_pkts:     2847392
  rx_ecn_ce_pkts:         0