Tail latency is the formal term for the straggler problem. It refers to the high-percentile latency responses -- the slow outliers that dominate worst-case system performance.
In a single-server application, p99 latency is a useful metric. If 99% of requests complete in under 10ms, the experience is generally acceptable. The 1% that take longer are a manageable trade-off.
In distributed AI training, the math inverts completely.
Consider a fabric that loses 1 in 1,000 packets -- excellent performance by most enterprise networking standards. A single training step across 256 GPUs involves thousands of RDMA operations. The probability that at least one hits the bad case is:
P(at least one failure) = 1 - (999/1000)^N
For N = 1,000 operations: ~= 63% chance of a stall per step. At 256 GPUs x 1,000 ops each: the stall is near-certain on every step.
Tail Latency — Why Scale Breaks Everything
Adjust the parameters to see how a small per-operation failure rate becomes a near-certain stall at training scale.
Single-server / 1 GPU — 1,000 ops
P(stall) = 1 − (1 − 0.100%)^1,000
Already problematic
256-GPU cluster — 256,000 total ops
P(stall) = 1 − (1 − 0.100%)^256,000
Near-certain stall on every training step.
The calculator above makes this concrete. Try setting GPU count to 256 and failure rate to 0.1% -- the default. Then reduce the failure rate toward 0.01% and watch the stall probability drop. The required target is not "low loss". It is zero loss, sustained, at every hop, across every link the job touches simultaneously.
This is the mathematical reason that TCP -- which tolerates packet loss and recovers via retransmission -- cannot be the transport protocol for AI training at scale. Every retransmit is a stall event. The transport must deliver without loss, which requires the fabric beneath it to be lossless. That is what Priority Flow Control (PFC) and ECN are for -- covered in Chapter 5.
Training vs inference
Figure: Training and inference react differently to the same slow event. In training, one lagging path can hold the whole synchronized step at the barrier; in inference, one slow request usually degrades only that request's latency.
Training and inference have different relationships with tail latency. In training, tail latency extends JCT -- the whole job takes longer due to AllReduce stalls. In inference (where a deployed model serves live user queries), there is no AllReduce barrier. Each request is independent. The networking concern shifts to per-request p99 latency -- how long does one user's query take? Inference values consistent low latency per request rather than aggregate throughput across a synchronised computation. This curriculum focuses on training infrastructure, where JCT and the barrier define the requirements.