The hop-by-hop propagation

PFC was designed to stop a single sender from overrunning a single buffer. It works well in that scenario. The problem emerges in multi-hop fabrics where the paused sender is itself a switch with multiple upstream connections.

Consider this path: DGX Node A -> Leaf Switch 1 -> Spine Switch -> Leaf Switch 2 -> DGX Node B

Node B's port is congested. Leaf Switch 2 sends a PAUSE frame to Spine Switch. Spine Switch pauses its output toward Leaf Switch 2. But Spine Switch is still receiving traffic from Leaf Switch 1. Spine Switch's buffer fills. Spine Switch sends a PAUSE frame to Leaf Switch 1. Leaf Switch 1 pauses its output toward Spine Switch. Leaf Switch 1's buffer fills. Leaf Switch 1 sends a PAUSE frame to DGX Node A. Node A stops sending.

This is the intended behaviour -- backpressure propagating to the source. But in a mesh fabric with multiple flows, the pause can propagate to flows that have nothing to do with the original congestion event. A pause originating from one congested destination can pause traffic destined for an entirely different, uncongested destination if that traffic shares a switch buffer with the paused flow.

Figure: A pause storm is not one hot port. It is a chain of queues waiting on each other, with the truly dangerous case being a closed loop that only the watchdog can break.

The deadlock scenario

The truly dangerous scenario is a circular pause dependency. Imagine a simplified fabric:

Node A -> Switch X -> Node B
Node B -> Switch Y -> Node A

Node A is sending to Node B (through Switch X). Node B is simultaneously sending to Node A (through Switch Y). Both paths are congested. Switch X sends PAUSE to Node A. Switch Y sends PAUSE to Node B. Node A is paused, so its buffer fills with traffic it cannot send. Node B is paused, so its buffer fills. Both switches are waiting for the nodes to drain their buffers. Neither node can drain because both are paused. The fabric is deadlocked.

In a real training cluster with hundreds of nodes doing simultaneous AllReduce, creating a circular pause dependency requires an unfortunate combination of traffic patterns and switch buffer pressure. It is rare under normal operation -- but it happens. The signature: multiple ports on multiple switches showing sustained PFC pause frames counters growing indefinitely, with zero drops but near-zero throughput.

The watchdog: the circuit breaker

The PFC watchdog is the mechanism that breaks deadlocks. It monitors each port's pause state. If a port has been continuously paused for longer than the watchdog interval (typically 200ms), the watchdog takes action:

• Action: drop -- The watchdog breaks the pause by dropping packets from the paused queue. Throughput recovers at the cost of some packet loss and RDMA retransmissions. The training job experiences a brief stall but resumes.
• Action: warn -- The watchdog logs the event but does not break the pause. Used for debugging, not production.

Always configure watchdog: action drop in production. A 200ms pause is not a problem -- it is an expected brief backpressure event. A sustained pause is a deadlock, and the only recovery is to break it. The watchdog does this automatically. Without it, a deadlock requires manual intervention -- SSHing into switches and disabling PFC while the training job is hanging, then re-enabling after draining the buffers.

The watchdog timer should be longer than the maximum expected legitimate pause duration. At 400G with pause quanta: 0xffff, the maximum legitimate pause is approximately 84 microseconds. A 200ms watchdog timer gives 2,380x headroom for legitimate pauses before declaring a deadlock. This is deliberate -- brief bursts can generate rapid sequences of pause frames that look like a sustained pause to a naive detector.