PFC operates on IEEE 802.3x PAUSE frame mechanics extended to per-priority operation by IEEE 802.1Qbb. When a switch's TC3 egress queue depth crosses the xoff threshold, the switch generates a priority-specific PAUSE frame and sends it to the upstream link partner.

Per-priority enable/disable ”” critical rule: Only TC3 (RoCEv2) should be lossless. Enabling PFC on multiple priorities causes buffer starvation cascades. The correct configuration: PFC on priority 3 only. All other priorities: PFC disabled.

PFC watchdog ”” deadlock detection: The PFC watchdog monitors how long each lossless queue has been continuously paused. Default timer on Cumulus Linux: 200ms. If TC3 remains in a PAUSE state for longer than 200ms, the watchdog fires.

Two watchdog actions:

• Drop mode (correct for RoCEv2): The watchdog starts dropping frames from the paused queue, breaking the deadlock at the cost of retransmissions.
• Forward mode: Not recommended for RoCEv2 because it defeats the lossless guarantee.

# Configure PFC watchdog
leaf-01# nv set interface swp1-32 qos pfc watchdog action drop
leaf-01# nv set interface swp1-32 qos pfc watchdog timeout 200

# Verify PFC counters
leaf-01# nv show interface swp1 counters pfc
Interface swp1 PFC Counters:
  Priority 3:
    TX pause frames:       142
    RX pause frames:       0
    Pause duration (ms):   12.4 avg / 38.1 max
    Watchdog fires:        0
  Priority 0-2,4-7:       not enabled

PFC headroom calculation: With a 5m passive DAC cable between DGX H100 and leaf switch:

cable propagation delay:    5m × 5ns/m = 25ns
switch processing delay:    ~300ns (Spectrum-4 cut-through)
total: ~325ns

headroom = 325ns × 400Gbps = 325e-9 × 400e9 = 130,000 bits = ~16 KB

# NIC-side PFC verification (DGX node)
dgx-01:~$ ethtool -S mlx5_0 | grep pfc
  rx_pfc_xon_frames_priority_3:   142
  rx_pfc_xoff_frames_priority_3:  142
  tx_pfc_xon_frames_priority_3:   0
  tx_pfc_xoff_frames_priority_3:  0