You have read the symptoms. Now you look at the configuration to understand why.
show dcb pfc -- reading PFC state
PFC (Priority Flow Control) is the lossless mechanism. show dcb pfc shows whether it is configured, on which priorities, and whether it is functioning.
show dcb pfc — three states to know
Interface eth0 Priority Flow Control: enabled PFC enabled priorities: 3 (cos3) Pause quanta: 0xffff Watchdog: enabled Watchdog interval: 200ms
No action needed. This is the target configuration for lossless RoCEv2.
When PFC is correctly enabled:
$ show dcb pfc
Interface swp1
Priority Flow Control: enabled
PFC enabled priorities: 3 (cos3)
Pause quanta: 0xffff
Watchdog: enabled
Watchdog interval: 200ms
Reading the output:
Priority Flow Control: enabled -- PFC is active on this interface. When disabled, any congestion will result in drops rather than pauses.
PFC enabled priorities: 3 (cos3) -- PFC is applied to traffic class 3. In this deployment, RoCEv2 traffic is marked to land on priority 3. The exact priority used for RoCEv2 is a site configuration decision -- common values are 3 and 5 depending on the deployment guide followed. What matters is that the PFC-enabled priority here matches the CoS marking your RoCEv2 traffic actually carries. If your traffic lands on priority 5 but PFC is only enabled on priority 3, you get no protection.
Pause quanta: 0xffff -- The duration of each PAUSE frame, measured in pause quanta (512-bit times). 0xffff is the maximum value -- the receiver should stop sending for the maximum possible duration. This is standard for storage and HPC traffic.
Watchdog: enabled -- The PFC watchdog monitors for deadlocks. If a port has been continuously paused for longer than the watchdog interval, it kills the PFC pause to break the deadlock. Always keep the watchdog enabled. It is your last line of defence against permanent pause storms.
Watchdog interval: 200ms -- If this port is continuously paused for 200ms, the watchdog fires.
When PFC is disabled:
$ show dcb pfc
Interface swp1
Priority Flow Control: disabled
PFC enabled priorities: none
Pause quanta: N/A
Watchdog: disabled
This is the fault state in Lab 1. Disabled PFC on a RoCEv2 port means any congestion causes drops. Training jobs will experience retransmissions. NCCL will log timeout errors. AllReduce operations will stall.
The PFC priority mismatch trap:
One of the most common PFC misconfigurations is enabling PFC on the wrong priority. If your RoCEv2 traffic is marked DSCP 26 (landing on CoS 3 in this deployment) but PFC is configured for priority 0, PFC provides no protection to your training traffic. This does not show up as "PFC disabled" -- PFC reports as enabled. You need to verify that PFC enabled priorities matches the actual CoS priority your RoCEv2 traffic uses. Verify the DSCP-to-priority mapping on both the NIC (mlnx_qos -i eth0 --dscp) and the switch (show qos dscp-map) -- they must agree.
show dcb ets -- reading ETS and ECN configuration
ETS (Enhanced Transmission Selection) controls how bandwidth is allocated between traffic classes. ECN (Explicit Congestion Notification) is configured within the ETS framework.
$ show dcb ets
Interface swp1 -- ETS Configuration
Traffic class Priority Bandwidth Algorithm
TC0 0,1,2 30% ETS
TC3 (RoCE) 3 50% Strict Priority
TC7 (mgmt) 7 20% ETS
ECN marking: enabled (DSCP 26)
DCQCN: active
Reading the output:
TC3 (RoCE) -- Strict Priority -- RoCEv2 traffic (priority 3) uses strict priority scheduling. This means TC3 traffic always gets served before TC0 or TC7 traffic. This is correct for latency-sensitive RDMA traffic.
ECN marking: enabled (DSCP 26) -- ECN is active. The switch will mark packets with the CE (Congestion Experienced) bit when queues exceed the ECN threshold, signalling senders to reduce their rate before buffers overflow. This is DCQCN in action.
DCQCN: active -- The Data Center Quantized Congestion Notification algorithm is running. The switch is marking ECN, the NIC firmware is reacting by reducing injection rate, congestion is being managed proactively.
When ECN is disabled:
ECN marking: disabled
DCQCN: inactive -- congestion unmanaged
This is the fault state in Lab 2. Without ECN, the only congestion mechanism is PFC (if configured). Congestion builds until PFC pauses the sender or until buffers overflow and drops occur. DCQCN is the more elegant solution -- it signals congestion before buffers fill.
show roce -- the RoCEv2 summary
show roce gives you a consolidated view of the RoCEv2 configuration on a specific interface. It is the first command to check when validating a newly configured port.
$ show roce
RoCE Configuration -- eth0
RoCE version: RoCEv2
State: active
PFC: enabled (priority 3)
ECN: enabled -- DCQCN active
DSCP marking: 26 (RoCE traffic)
MTU: 9000
GID: fe80::506b:4b03:00a1:b200
RoCE version: RoCEv2 -- Confirms this is RoCEv2 (UDP/IP encapsulation), not RoCE v1 (Ethernet-only, non-routable).
DSCP marking: 26 -- Traffic is being marked DSCP 26 for correct QoS treatment. If this shows "not configured", your RoCEv2 traffic will not receive priority scheduling.
MTU: 9000 -- Jumbo frames are enabled. This is required for efficient RDMA. If MTU shows 1500, your RDMA performance will be severely degraded -- every large RDMA transfer will be fragmented into thousands of small packets, each with its own header overhead.
GID: fe80::... -- The Global Identifier for this RDMA interface. This is the routable address that other nodes use to establish RDMA connections. If this is missing, RDMA connections cannot be established.