Static ECMP has done honest work in data centre networks for decades. Hash the five tuple -- source and destination IP, source and destination port, protocol -- into a bucket, then pin every packet of that flow to the same spine link. Consistency is the whole point: packets stay in order, TCP is content, and the forwarding hardware stays trivially simple.
AI training traffic is not web traffic. When 512 GPUs hit an allreduce -- the synchronisation barrier that closes every backward pass -- they all start transmitting inside the same microsecond. The flow table at the leaf switch explodes with hundreds of simultaneous RDMA QPs, each assigned to a spine by hash. On a fabric with four spine switches, the ideal outcome is equal distribution: 25% of QPs per spine. Reality is different.
Hash-based ECMP assumes statistically independent flows turning up at random. Collectives are the opposite of that: they are synchronised on purpose. The NCCL ring-allreduce algorithm creates wave patterns where every node in the ring starts new QPs at nearly the same time. The five-tuple hash of these flows is dominated by the GPU rail IP addresses, which are assigned sequentially (10.100.0.1, 10.100.0.2, ...). Sequential addresses produce hash collisions with alarming regularity.
Quantifying the imbalance. In a 512-GPU cluster with four spines, each spine should carry exactly 25% of aggregate fabric bandwidth. Under hash-based ECMP with synchronized collective traffic, empirical measurements show variance of 18-25% across spines. In practice this means one spine at 91% utilization while adjacent spines sit at 22%. The overloaded spine becomes a bottleneck: ECN marks trigger DCQCN, send rates are throttled, and all-reduce completion time stretches because every node must wait for the slowest QP pair.
The variance compounds with scale. At 128 GPUs the hash collision probability is manageable. At 512 GPUs you statistically guarantee severe collisions on every barrier. At 1024 GPUs (a full Spectrum-X SuperPOD) hash ECMP without adaptive routing is simply not viable for collective workloads.
Why per-packet spraying is theoretically ideal. If every packet is independently forwarded to the least-loaded spine, load balances to within a few percent of perfect. The problem: if packet 1 takes spine-01 (4 us latency) and packet 2 takes spine-03 (4.1 us latency due to transient queue), the receiver sees out-of-order delivery. For RoCEv2 using Go-Back-N error recovery, receiving a packet out-of-sequence triggers a NAK - and the entire window retransmits. Per-packet spraying without receiver-side reordering collapses throughput worse than hash ECMP.
Adaptive Routing on Spectrum-X solves this problem through two cooperating mechanisms: queue-depth-driven forwarding decisions in the Spectrum-4 ASIC, and a hardware reorder buffer in the SuperNIC on the host. On B200 that SuperNIC is BlueField-3; on HGX B300 it is ConnectX-8, which delivers 800 Gb/s per port and performs the same direct data placement - writing packets into the receive buffer in sequence order as they arrive. Together the two mechanisms enable per-packet distribution with in-order delivery.