Your first instinct might be: just use fast Ethernet. We have 400GbE. That should be enough.

The bandwidth is not the problem. The problem is how TCP/IP handles failure.

When a packet is dropped on a TCP connection, TCP detects the loss after a timeout, retransmits the packet, and continues. For a web server or a database, this is acceptable -- the user notices a slight slowdown at most.

For an AllReduce operation across 256 GPUs:

• One packet drop causes a GPU's RDMA queue pair to enter an error state
• The RDMA QP enters error state and must retransmit all unacknowledged packets from the point of the drop (Go-Back-N from the dropped sequence number onwards) -- which, for a large in-flight tensor, means retransmitting the vast majority of the transfer
• Every other GPU is sitting idle at the synchronisation barrier waiting
• At 400 Gb/s, "a tensor" might mean several gigabytes
• This happens thousands of times per hour during training. Optical transceivers -- the pluggable modules that terminate every 400G fibre run -- are the single most frequent hardware failure point in production AI clusters. A marginal transceiver can cause intermittent packet drops without fully going down, making it far harder to diagnose than a clean link failure.

A single packet drop can stall a 256-GPU job for hundreds of milliseconds. At the scale of a large training run -- weeks of continuous GPU time -- even a 1% slowdown from network issues costs millions of dollars.

This is why AI clusters need lossless networking. Not low-loss. Zero loss.

Packet loss was not the only reason vanilla Ethernet struggled. Even when packets were delivered correctly, the load balancing mechanism itself caused congestion.

Standard Ethernet uses ECMP (Equal-Cost Multi-Path) routing: all packets in a flow follow the same path, chosen by hashing the 5-tuple (source IP, destination IP, protocol, source port, destination port). For ECMP to distribute traffic well, flows need to produce different hash values -- in other words, flows need entropy.

RoCEv2 traffic is structurally low-entropy. Consider what the 5-tuple looks like for two DGX nodes communicating during AllReduce:

• Source IP: always one of eight DGX node addresses
• Destination IP: always one of eight DGX node addresses
• Protocol: always UDP
• Destination port: always 4791 (the RoCEv2 port -- unchangeable)
• Source port: derived from the queue pair number (a small range)

Unlike web traffic -- where thousands of different clients connecting to thousands of different services produce richly varied 5-tuples -- AI training traffic between a small number of GPU servers producing a fixed destination port produces headers that are nearly identical. The ECMP hash function sees the same input and produces the same output: every packet from the same GPU pair takes the same path.

This becomes a flowlet problem. A flowlet is an individual low-entropy flow that a switch treats as part of a larger flow because it cannot distinguish them at the header level. Two types of flowlets cause specific problems in AI fabrics:

Simultaneous flowlets: Multiple GPU pairs communicating at the same moment during AllReduce. From the switch's perspective, packets from GPU 0 on Node A to GPU 0 on Node B, and packets from GPU 0 on Node C to GPU 0 on Node B, may look identical at the IP/UDP header level. The switch hashes them to the same path. One link is overloaded while others sit idle -- even though the cluster has enough aggregate bandwidth for all flows.

Sequential flowlets: Job 1 completes between GPU A and GPU B. Job 2 starts between the same pair. The headers are identical. The switch treats Job 2's traffic as a continuation of Job 1's flow and keeps it pinned to the same path -- even if that path is now congested. From the switch's perspective, this looks like one long-lived elephant flow. The path rebalancing that might help never triggers because the switch never detects a new flow.

The cluster has plenty of bandwidth in aggregate -- it is just distributed incorrectly, and ECMP has no mechanism to detect or fix this mid-flight.

Why this matters for your diagnostic work: When you see uneven link utilisation on spine switches -- some links at 90% while adjacent links are at 10% -- during an AllReduce workload, the cause is almost always the flowlet problem. The fix is not more bandwidth on the hot links. The fix is per-packet load balancing (Spectrum-X RSHP or InfiniBand adaptive routing) so the switch distributes individual packets across all paths rather than pinning flows.

This is addressed in Act 6 below. For now, the point is: Ethernet had two independent problems -- packet loss under congestion, and inefficient load balancing under high fan-out traffic. AI training exposed both simultaneously. The combination is particularly damaging because low-entropy traffic makes the load balancing problem worse at exactly the moment congestion is most likely -- when all GPUs are synchronising gradients.