What PFC actually is

Classic Ethernet pause (IEEE 802.3x) was designed in 1997 for full-duplex links. When a receiver's buffer fills, it sends a 64-byte PAUSE frame to the sender. The sender stops transmitting for the duration specified -- measured in pause quanta, where one pause quanta equals 512 bit-times.

Classic pause has one fatal flaw for HPC: it pauses all traffic on the link. Management traffic, storage traffic, and training traffic all stop simultaneously. In a modern cluster running multiple traffic classes, pausing everything to protect one class is unacceptable.

Priority Flow Control (IEEE 802.1Qbb) solves this. Instead of one pause frame for the entire link, PFC defines up to eight independent pause frames -- one per CoS (Class of Service) priority. A PAUSE frame for priority 3 pauses only priority 3 traffic. Priorities 0, 1, 2, 4, 5, 6, and 7 continue flowing unaffected.

This is why RoCEv2 traffic is placed on a specific priority -- CoS 3 in standard deployments. PFC can protect the RDMA traffic class while allowing management and storage traffic to pass unimpeded.

The pause frame anatomy

A PFC PAUSE frame is a 64-byte Ethernet frame with:

• Destination MAC: 01:80:C2:00:00:01 (reserved multicast -- link-local, never forwarded)
• EtherType: 0x8808 (MAC Control)
• Opcode: 0x0101 (Priority Pause)
• Enable vector: 8 bits, one per priority -- which priorities are being paused
• Pause quanta: 8 x 16-bit values -- pause duration for each priority

Pause quanta is the key parameter. One pause quanta = 512 bit-times. At 400 Gb/s:

512 bits / 400,000,000,000 bits/second = 1.28 nanoseconds per quanta

Maximum quanta value: 0xFFFF = 65,535 quanta
Maximum pause duration at 400G: 65,535 x 1.28ns ~= 83.9 microseconds

So when you see Pause quanta: 0xffff in show dcb pfc, you are seeing the maximum possible pause duration -- approximately 84 microseconds at 400G. This is intentionally set to maximum because in RDMA workloads, you want the sender to stop completely and stay stopped until the congestion clears, not resume prematurely.

The PFC buffer threshold

The switch does not wait until its buffer is full before sending a PAUSE frame. It sends the PAUSE frame when the buffer reaches a headroom threshold -- typically 60-80% full. Why? Because it takes time for the PAUSE frame to travel to the sender, and it takes time for the sender to stop. During that propagation delay and reaction time, packets keep arriving. The headroom ensures there is still buffer space available to absorb in-flight packets before the sender actually stops.

At 400G with a 200ns round-trip propagation delay:

Packets arriving during 200ns pause propagation:
  400 Gb/s x 200ns = 80,000 bits = 10,000 bytes ~= 1 maximum-size jumbo frame

The headroom must be at least 10 KB to absorb packets in flight during pause propagation. Production deployments typically configure 64-128 KB of headroom to account for variable latency.

The three ways PFC misconfiguration shows up

Misconfiguration 1: PFC disabled Output: Priority Flow Control: disabled Effect: Any congestion causes tail drop. RDMA queue pairs error. AllReduce stalls on every packet drop event. This is the Lab 1 scenario.

Misconfiguration 2: PFC on wrong priority Output: Priority Flow Control: enabled, PFC enabled priorities: 0 (cos0) Effect: PFC is protecting best-effort traffic (CoS 0). RoCEv2 traffic on CoS 3 gets no protection -- drops occur under congestion exactly as if PFC were disabled. This is the most deceptive misconfiguration because show dcb pfc shows "enabled."

Misconfiguration 3: PFC watchdog disabled Output: Watchdog: disabled Effect: If a pause storm develops (pause frames propagating in a loop), nothing breaks the deadlock. The fabric freezes. Training jobs hang indefinitely. Only a manual intervention (disabling PFC on the affected port or rebooting the switch) clears it.

Configuring PFC on a Spectrum-X switch

On the switch (Cumulus Linux / NVUE):

# Leaf switch terminal (blue prompt)
# Note: Spectrum-X switch ports use swp* naming (swp1-swp32 for front-panel)
# not eth* -- eth* names are the DGX host-side NIC interfaces

# Enable PFC on priority 3 for a specific switch port
nv set interface swp1 qos pfc enable on
nv set interface swp1 qos pfc priority 3

# Set PFC watchdog
nv set interface swp1 qos pfc watchdog action drop
nv set interface swp1 qos pfc watchdog interval 200

# Apply to all server-facing ports at once (swp1-swp32 typical)
nv set interface swp1-32 qos pfc enable on
nv set interface swp1-32 qos pfc priority 3

nv config apply

On the DGX host (mlnx_qos):

# DGX host terminal (green prompt)
# eth0 here is the DGX NIC netdev name -- different from the switch's swp1

# Enable PFC on priority 3 for the RDMA interface
mlnx_qos -i eth0 --pfc 0,0,0,1,0,0,0,0
# Arguments are priorities 0-7: 0=disabled, 1=enabled
# Position 3 = 1 means priority 3 is PFC-enabled

# Verify
mlnx_qos -i eth0

The NIC side and switch side must agree -- both must have PFC enabled on the same priority. A mismatch (PFC enabled on priority 3 on the switch but priority 0 on the NIC) means pause frames are sent on one priority but the NIC is not watching for pauses on that priority. Also verify that the DSCP-to-priority mapping is consistent: the RoCEv2 DSCP value (commonly 26 or 46 depending on your deployment's QoS policy) must map to the same priority on both the NIC and the switch.