You have spent two chapters understanding how a 400GbE switch fabric moves packets. Now let's talk about what happens when it can't move them fast enough — and why the algorithm that responds to that condition has a measurable dollar cost at scale.
Job Completion Time sensitivity
A GPU training job does not complete until the last GPU finishes its gradient synchronisation. That property — the job is only as fast as its slowest participant — is called the straggler effect, and it makes GPU cluster throughput brutally sensitive to tail latency.
Consider a 10,000-GPU All-Reduce. The ring-AllReduce algorithm requires every node to both send and receive across 2*(N-1) steps. Any one link that is congested and slowing one node ripples across the entire collective. If congestion doubles the completion time on a single 400GbE link for just 200 microseconds, that delay propagates backward through the synchronisation barrier and stalls 9,999 other GPUs.
The cost is not abstract. At $10/GPU-hour for H100 capacity:
- A 10,000-GPU cluster running 24 h/day = $2.4M/day in GPU compute
- A 5% JCT degradation from persistent congestion = $120K/day in lost compute value
- A 20% JCT degradation — common with misconfigured DCQCN — = $480K/day
This is why hyperscalers spend engineering years tuning congestion control algorithms. It is not a feature on a feature list. It is the primary operational lever available to a network engineer inside a running AI cluster.
Why standard TCP congestion control fails here
TCP Cubic or DCTCP work adequately for latency-insensitive workloads. They have two fatal properties for GPU collectives:
-
Kernel-bypass incompatibility. RoCEv2 operates entirely in NIC hardware, bypassing the kernel network stack. TCP's congestion control lives in the kernel. When you use RoCEv2 (which you must for RDMA performance), TCP CC is not in the picture — you need a CC scheme implemented in NIC firmware or hardware.
-
Reaction latency. TCP Cubic's backoff is designed for WAN round-trips (10s–100s of milliseconds). A 400GbE AI fabric link has an RTT of roughly 1–5 microseconds. TCP Cubic reacts many RTTs after queue build-up begins. In that time a 51.2 Tbps switch can fill its entire per-port buffer (1–2 MB) from a single incast event. By the time TCP backs off, the damage — PFC pause propagation, dropped packets, retransmits — has already occurred.
The algorithms covered in this chapter operate at NIC hardware speed, react within 1–2 RTTs, and are specifically designed for the incast patterns generated by GPU collectives.