AI model training splits computation across many GPUs simultaneously. After each training step, all GPUs must agree on updated model parameters. This requires an operation called AllReduce: every GPU sends its gradient updates, they are aggregated, and the result is distributed back to all GPUs before the next step can begin.
That synchronisation point is a barrier. No GPU can proceed to the next step until every other GPU has completed the AllReduce. If one GPU's network path has a retransmission due to a dropped packet, every other GPU sits idle waiting for it.
AllReduce Barrier
The barrier enforces synchronisation: no GPU advances until every GPU completes. One straggler idles all others.
The simulation above makes the consequence concrete. GPU 5 is slightly slower -- a single marginal switch port, a single congested link. The result is not that GPU 5 runs at 80% efficiency while the others run at 100%. The result is that all eight GPUs run at GPU 5's speed. The barrier enforces this.
At 1,000 training steps per hour on a 256-GPU cluster, one degraded switch port can idle 255 GPUs for hours. The GPU-hour cost accumulates at whatever your cluster's hourly rate is -- typically hundreds of dollars per hour for an H100 cluster.
This is why Job Completion Time (JCT) -- the elapsed time from when a training job starts to when it finishes -- is the primary KPI for AI infrastructure. Every conversation about network design, topology, congestion control, and cabling eventually comes back to: what does this do to JCT?
JCT = f(slowest thread) = f(worst network path in the cluster)