The congestion timeline

Here is what actually happens during a congestion event on a correctly configured fabric with both PFC and ECN active:

T=0ms: Congestion begins Multiple DGX nodes simultaneously send AllReduce gradient data. One switch port receives 420G of input destined for a 400G output. Buffer depth begins increasing.

T=0.1ms: ECN threshold crossed Buffer depth crosses ECN min-threshold (150 KB in our example). The switch begins marking a fraction of packets with CE. Receivers send CNPs back to the senders. DCQCN in each sender's NIC begins reducing injection rate.

T=0.5ms: Rate reduction takes effect Senders have reduced their injection rates. Input traffic drops toward 400G. Buffer depth stabilises and begins decreasing. ECN marking frequency decreases. DCQCN enters recovery phase.

T=2ms: Congestion resolved by ECN alone Buffer drains below ECN min-threshold. No PFC frames were ever sent. No traffic was paused. From the application's perspective, AllReduce completed with slightly reduced throughput for 2ms -- completely invisible at the training step level.

Alternative: ECN insufficient (burst too large)

T=0.1ms: ECN marking begins (same as above) T=0.3ms: PFC threshold crossed The burst is too large and too sudden for DCQCN rate reduction to drain the buffer fast enough. Buffer depth crosses the PFC headroom threshold. The switch sends a PAUSE frame for CoS 3. Senders freeze immediately.

T=0.35ms: Pause propagates In-flight packets continue arriving for approximately 200ns (propagation delay). Buffer absorbs them -- the headroom was sized for exactly this.

T=0.4ms: Buffer stabilises No new packets arriving. Buffer begins draining as the output port continues forwarding.

T=1.2ms: Buffer drains below PFC threshold Switch sends a RESUME frame (or simply stops sending PAUSE renewals). Senders resume at the DCQCN-reduced rate, not the original full rate.

T=3ms: Congestion resolved AllReduce completes. The whole event took 3ms with one 800us pause -- well within acceptable limits for a training cluster.

When both mechanisms fail

Both PFC and ECN fail when congestion is sustained and severe -- when multiple flows are simultaneously sending at maximum rate and the aggregate input rate persistently exceeds the output rate by a large margin.

In this scenario:

• ECN marks packets continuously, but senders cannot reduce rate fast enough to drain the buffer
• PFC pauses senders, but new flows from other nodes fill the buffer immediately
• Buffer reaches maximum depth -- drops occur despite PFC

This is not a configuration failure -- it is a fabric capacity failure. The solution is more bandwidth (higher-speed links, more parallel paths), better traffic engineering (spreading flows across more paths), or reducing the all-to-all communication volume (model architecture changes, gradient compression).

A third failure mode exists: both PFC and ECN are configured correctly, but load balancing is placing too much traffic on one path. This looks like a capacity failure but the fix is not more bandwidth -- it is better path distribution. The diagnostic signature: drops and pauses on specific ports while adjacent ports with the same hardware have low utilisation.

The counters you look for to distinguish a configuration failure from a capacity failure:

• Configuration failure: drops with ECN marking zero -> ECN not configured
• Configuration failure: sustained pause storms -> PFC priority mismatch or missing ECN
• Capacity failure: drops despite active ECN marking and valid PFC -> fabric is undersized for the traffic volume