Skip to content

Chapter 2: Why HPC Networking Is Different · Part 3 of 6

Act 2 -- The AllReduce barrier

AI model training splits computation across many GPUs simultaneously. After each training step, all GPUs must agree on updated model parameters. This requires an operation called AllReduce: every GPU sends its gradient updates, they are aggregated, and the result is distributed back to all GPUs before the next step can begin.

That synchronisation point is a barrier. No GPU can proceed to the next step until every other GPU has completed the AllReduce. If one GPU's network path has a retransmission due to a dropped packet, every other GPU sits idle waiting for it.

AllReduce Barrier

The barrier enforces synchronisation: no GPU advances until every GPU completes. One straggler idles all others.

All GPUs computing gradients...
GPU 0
compute
GPU 1
compute
GPU 2
compute
GPU 3
compute
GPU 4
compute
GPU 5
compute
GPU 6
compute
GPU 7
compute
← barrier
Computing
Sending (fabric)
Waiting at barrier
Straggler (GPU 5)
Barrier passed

The simulation above makes the consequence concrete. GPU 5 is slightly slower -- a single marginal switch port, a single congested link. The result is not that GPU 5 runs at 80% efficiency while the others run at 100%. The result is that all eight GPUs run at GPU 5's speed. The barrier enforces this.

At 1,000 training steps per hour on a 256-GPU cluster, one degraded switch port can idle 255 GPUs for hours. The GPU-hour cost accumulates at whatever your cluster's hourly rate is -- typically hundreds of dollars per hour for an H100 cluster.

This is why Job Completion Time (JCT) -- the elapsed time from when a training job starts to when it finishes -- is the primary KPI for AI infrastructure. Every conversation about network design, topology, congestion control, and cabling eventually comes back to: what does this do to JCT?

JCT = f(slowest thread) = f(worst network path in the cluster)