Every fabric technology produces a different packet, different switch lookup, different congestion control, different latency profile. This act consolidates the full comparison.
Layer-by-layer comparison
Layer 1 -- Physical
All modern AI fabrics use 400G OSFP with PAM-4 modulation. This is the same physical spec across ConnectX-7 RoCEv2, ConnectX-7 InfiniBand, Gaudi 3, and MI300X with Elba. Slingshot also uses 200G or 400G optical -- the physical signalling is compatible but the protocol above it is not.
The optics may differ: DGX H100 ships with QSFP-DD-to-OSFP breakout cables for some rack configurations; Frontier uses SMF-based optical links between Rosetta ASICs. These differences are at the DAC/transceiver level, not visible to any network protocol.
Layer 2 -- Data link
| Fabric | L2 header | Addressing | Rewritten each hop? |
|---|---|---|---|
| RoCEv2 / Spectrum-X | Ethernet II (14B) | 48-bit MAC | Yes -- full rewrite |
| InfiniBand NDR | LRH (8B) | 16-bit LID | No -- LID unchanged end-to-end |
| Slingshot | Ethernet + Slingshot ext | MAC + routing tag | Yes -- MAC rewritten |
Layer 3 -- Network
| Fabric | L3 header | Addressing | Lookup type |
|---|---|---|---|
| RoCEv2 | IPv4 (20B) + UDP (8B) | 32-bit IP | ECMP hash -> next-hop (IP route table) |
| InfiniBand | GRH (40B, optional) | 128-bit GID | LFT array[DLID] = port -> O(1) |
| Slingshot | IPv4 + Slingshot ext | IP + routing tag | Slingshot routing table |
Congestion control
| Fabric | Mechanism | Granularity | Propagation |
|---|---|---|---|
| RoCEv2 | PFC pause + ECN marks + DCQCN rate reduction | Per priority class | Hop-by-hop (PFC) + end-to-end (ECN) |
| InfiniBand | Credit-based flow control (per-VL per-port) | Per virtual lane | Per-link (no backward propagation) |
| Slingshot | HRP credit-based + adaptive routing | Per destination | Per-link (no PFC storms possible) |
Load balancing
| Fabric | Method | Entropy source | OOO handling |
|---|---|---|---|
| RoCEv2 + SLB | ECMP hash | 5-tuple + optional QPair | None -- retransmit on OOO |
| RoCEv2 + RSHP | Per-packet spray | All flows treated independently | RSHP reorder buffer in ConnectX-7 |
| InfiniBand | Adaptive routing (per-packet, congestion-aware) | No hash -- real-time congestion measurement | Native IB transport |
| Slingshot | Adaptive routing (per-flow, per-credit) | Flow tag in Slingshot header | HRP credits maintain order within flow |
Latency benchmark (2-hop fabric, 128B message)
| Fabric | MPI latency | Notes |
|---|---|---|
| InfiniBand NDR 400G | ~0.6 us | Credit-based FC, per-packet adaptive routing, O(1) LFT lookup |
| RoCEv2 + Spectrum-X RSHP | ~1.4-1.6 us | Added UDP/IP overhead, PFC latency jitter, IP route lookup |
| Slingshot 200G | ~1.1-1.3 us | Credit-based FC, adaptive routing, Slingshot header overhead |
| Gaudi 3 (100GbE RoCEv2) | ~2.0-2.5 us | 100G limit, standard Ethernet, no Spectrum-X RSHP |
When would you choose each?
InfiniBand NDR remains the lowest-latency, highest-throughput choice for pure GPU training workloads where the cluster is homogeneous NVIDIA hardware. The premium is the proprietary switch stack (QM9700 vs commodity Ethernet) and the reliance on the Subnet Manager.
RoCEv2 over Spectrum-X is the choice when the cluster must support mixed workloads (GPU training + inference + storage on the same fabric), when EVPN multi-tenancy is needed, or when the organisation has strong Ethernet operational expertise and wants BGP-based routing with familiar tooling. The latency penalty (~1 us per hop) is acceptable for most AllReduce patterns.
Slingshot is the choice when building at scientific HPC scale (tens of thousands of nodes) where the traffic pattern is mixed HPC + AI and where the credit-based flow control is a hard requirement due to the scale of MPI collective operations.
Gaudi 3 (100GbE integrated) is the choice when the budget requires it or when the workload is inference-oriented (lower bandwidth requirements, more sensitivity to per-GPU cost than raw training throughput).