Given that RoCEv2 over Ethernet achieves similar bandwidth numbers in benchmarks, why does InfiniBand still dominate large-scale AI training clusters?
Three reasons:
1. Lossless by design, not by configuration InfiniBand uses credit-based flow control built into the protocol. Packets are only sent when the receiver has confirmed it has buffer space. Drops are architecturally impossible. RoCEv2 achieves losslessness through PFC (Priority Flow Control) and ECN -- mechanisms that work well but require careful configuration and can fail in complex ways (PFC deadlocks, ECN misconfiguration). InfiniBand simply cannot lose packets.
2. SHARP -- in-network compute InfiniBand switches can participate in the AllReduce computation itself. Rather than all gradients travelling to GPUs and back, the switch performs the summation as data flows through it. SHARP (Scalable Hierarchical Aggregation and Reduction Protocol) can reduce AllReduce communication time by up to 50% at scale. Ethernet switches cannot do this -- but there is an important nuance: the 3rd-generation NVSwitch inside each DGX H100 node supports SHARP-equivalent in-network reduction for intra-node AllReduce over NVLink. The IB SHARP advantage described here applies to inter-node AllReduce across the external fabric, which NVSwitch cannot reach. For training runs that span multiple nodes -- which is most large-scale training -- IB SHARP remains a meaningful advantage.
3. The subnet manager InfiniBand has a centralised subnet manager that knows the complete topology, computes optimal paths, and reroutes traffic in microseconds when a link fails. Ethernet relies on distributed protocols (BGP, OSPF) that converge over seconds to minutes. For a training job that cannot tolerate even brief interruptions, the InfiniBand model is superior.
4. Adaptive routing vs ECMP
Standard Ethernet routes entire flows per-path using ECMP hash. InfiniBand makes routing decisions per-packet, measuring actual congestion on each output port and forwarding each packet on whichever path is least loaded at that instant. When multiple large AllReduce operations run simultaneously -- which is constant in a training cluster -- InfiniBand distributes them evenly across all available paths. ECMP creates hot links and idle links simultaneously.
ECMP collisions vs adaptive routing — the load balancing gap
Spine link utilisation during simultaneous AllReduce operations
Two flows hashed to the same path → 200% load on 2 links, near-zero on others. Total available bandwidth unused: ~60%.
How it works
ECMP (Equal-Cost Multi-Path) routes entire flows — all packets in a TCP or RDMA flow — down the same path, chosen by hashing the 5-tuple (src IP, dst IP, src port, dst port, protocol). Once a flow is assigned to a path, it stays on that path for the entire duration.
The mechanism
When two massive 400G AllReduce flows hash to the same spine link, that link is instantly saturated at 800G while trying to handle 400G capacity. Meanwhile other spine links sit completely idle. The cluster has plenty of total bandwidth — it is just distributed wrong. You cannot rebalance without dropping and re-establishing flows, which causes retransmissions.
Result: hot spots, packet drops, retransmissions, stalled AllReduce operations. This was one of the primary reasons vanilla Ethernet was inadequate for large-scale AI training before Spectrum-X.
This load balancing gap was one of the primary reasons Ethernet was inadequate for AI training at scale. NVIDIA's Spectrum-X closed this gap by introducing RSHP (Resilient Hashing and Per-packet reordering) -- per-packet routing on Ethernet with NIC-side reorder buffering. Traditional Ethernet switches without RSHP still have the ECMP problem.
The counterarguments for RoCEv2 are equally real: lower cost, operational familiarity, existing tooling, multi-tenancy support. There is no universal answer -- which is why you need to understand both, which is what this course teaches.