A Spectrum-X leaf switch has a 400G port connected to a DGX NIC. During AllReduce, multiple DGX nodes simultaneously send gradient tensors through this port to the same destination. For a brief moment, the input rate exceeds the output rate. Traffic is arriving faster than it can leave.

The port has a hardware buffer -- a block of SRAM in the switch ASIC. When output is slower than input, packets queue in the buffer. The buffer fills. What happens next defines the difference between a lossless fabric and a best-effort one.

Without any mechanism: tail drop

When the buffer is full, the next arriving packet is discarded. This is called tail drop -- the packet at the tail of the queue is thrown away. The sender does not know the packet was dropped until a timeout fires (typically hundreds of milliseconds to seconds). The RDMA queue pair enters an error state. The entire tensor must be retransmitted. Every GPU in the AllReduce is stalled at the synchronisation barrier waiting.

At 400 Gb/s, a 9 KB jumbo frame takes 180 nanoseconds to transmit. A congestion event lasting 10 milliseconds means tens of thousands of packets have arrived while the buffer filled. The retransmission stall compounds across the entire cluster.

With PFC only: pauses instead of drops

PFC (Priority Flow Control) is a backpressure mechanism. When the switch buffer reaches a threshold, the switch sends a PAUSE frame upstream -- telling the sender to stop transmitting. The sender freezes its output queue for the duration specified in the PAUSE frame. No packets are dropped. The network is lossless.

But pauses propagate. The paused sender's own upstream switch now has traffic queuing because its downstream is paused. It sends its own PAUSE frames upstream. The pause propagates hop by hop through the fabric. In a worst case, the pause reaches a node that is also in a circular dependency -- and the fabric deadlocks. Nothing moves.

With PFC and ECN: early warning prevents the cliff

ECN (Explicit Congestion Notification) is a marking mechanism. Before buffers reach the drop threshold, the switch marks packets with a signal -- "congestion experienced here." The receiver sees the mark and sends a notification back to the sender. The sender reduces its injection rate. Traffic slows before buffers fill. PFC pauses become rare rather than constant. Deadlocks become nearly impossible.

This is the correct architecture for a lossless RDMA fabric: ECN handles normal congestion proactively, PFC is the emergency backstop for bursts that ECN could not catch in time.

Understanding this interaction is the key to everything that follows.

But there is a layer before both: proactive congestion prevention

PFC and ECN are both reactive mechanisms -- they respond after congestion has begun. ECN marks packets when buffers start to fill. PFC pauses senders when buffers approach overflow. Both are essential, but they are last resorts.

Load balancing is proactive -- it distributes traffic across available paths before buffers fill, preventing congestion from forming in the first place. When the fabric correctly spreads AllReduce flows across all spine links, no single link becomes a bottleneck. ECN thresholds are never crossed. PFC frames are never sent. Training proceeds without interruption.

This layering defines how you diagnose problems in production:

If you see PFC pause frames growing constantly -- the reactive layer is working (PFC is preventing drops) but congestion is sustained. Ask why load balancing is not preventing the congestion from forming. Check spine link utilisation: are some links at 90% while others are at 10%? If yes, the load balancing layer has a problem, and tuning PFC thresholds will not fix it.

If you see PFC pause frames occasionally during burst periods -- this is expected and healthy. AllReduce bursts can briefly overwhelm any load balancer. PFC is providing the safety net it was designed for. No action needed unless pauses become sustained.

If you see drops despite PFC being configured -- either ECN is not configured (so rate reduction never happened before PFC was needed) or the oversubscription ratio is too high for the traffic volume. PFC alone cannot save an oversubscribed fabric.

The complete congestion management stack has three layers: proactive load balancing (distributes before buffers fill), ECN (signals senders to slow down as buffers start filling), PFC (pauses senders when buffers approach overflow). Chapters 5 and the upcoming Chapter 6 on load balancing cover each layer. Understanding the stack as a whole is what separates an engineer who can diagnose production congestion from one who just runs the same commands repeatedly.