Static ECMP has served data centre networks well for decades. The principle is simple: hash five-tuple fields (source IP, destination IP, source port, destination port, protocol) into a bucket, and send every packet in a flow down the same spine link. Consistency is the virtue - packets within a flow arrive in order, TCP stays happy, and the forwarding hardware is dead simple.
But AI training fabrics are not web traffic. When 512 GPUs execute an allreduce collective - the synchronization barrier at the end of every backward pass - they all begin transmitting at roughly the same microsecond. The flow table at the leaf switch explodes with hundreds of simultaneous RDMA QPs, each assigned to a spine by hash. On a fabric with four spine switches, the ideal outcome is equal distribution: 25% of QPs per spine. Reality is different.
Hash-based ECMP was designed for statistically independent flows arriving at random intervals. Collective operations are not statistically independent - they are synchronized by design. The NCCL ring-allreduce algorithm creates wave patterns where every node in the ring starts new QPs at nearly the same time. The five-tuple hash of these flows is dominated by the GPU rail IP addresses, which are assigned sequentially (10.100.0.1, 10.100.0.2, ...). Sequential addresses produce hash collisions with alarming regularity.
Quantifying the imbalance. In a 512-GPU cluster with four spines, each spine should carry exactly 25% of aggregate fabric bandwidth. Under hash-based ECMP with synchronized collective traffic, empirical measurements show variance of 18-25% across spines. In practice this means one spine at 91% utilization while adjacent spines sit at 22%. The overloaded spine becomes a bottleneck: ECN marks trigger DCQCN, send rates are throttled, and all-reduce completion time stretches because every node must wait for the slowest QP pair.
The variance compounds with scale. At 128 GPUs the hash collision probability is manageable. At 512 GPUs you statistically guarantee severe collisions on every barrier. At 1024 GPUs (a full Spectrum-X SuperPOD) hash ECMP without adaptive routing is simply not viable for collective workloads.
Why per-packet spraying is theoretically ideal. If every packet is independently forwarded to the least-loaded spine, load balances to within a few percent of perfect. The problem: if packet 1 takes spine-01 (4 us latency) and packet 2 takes spine-03 (4.1 us latency due to transient queue), the receiver sees out-of-order delivery. For RoCEv2 using Go-Back-N error recovery, receiving a packet out-of-sequence triggers a NAK - and the entire window retransmits. Per-packet spraying without receiver-side reordering collapses throughput worse than hash ECMP.
Adaptive Routing on Spectrum-X solves this problem through two cooperating mechanisms: queue-depth-driven forwarding decisions in the Spectrum-4 ASIC, and a hardware reorder buffer in the BlueField-3 SuperNIC on DGX B200 hosts. Together they enable per-packet distribution with in-order delivery.