The journey to today's AI networking infrastructure went through several stages.
Industry evolution — click any era
InfiniBand adoption begins
Supercomputer tech enters AI clusters
Models outgrew single GPUs. Researchers discovered InfiniBand — which had existed in supercomputing since the early 2000s — and adopted it for GPU clusters. InfiniBand's credit-based flow control made it naturally lossless. GPUDirect RDMA allowed GPU memory to be directly exposed to the InfiniBand fabric. The V100 (2017) had 300 GB/s NVLink and full InfiniBand support.
2010-2015: Early deep learning clusters Researchers used standard Ethernet with TCP/IP. Training was slow but the models were small. The network was not yet the bottleneck.
2016-2018: The GPU cluster era begins Models got larger. Multi-GPU training became necessary. Teams started using InfiniBand -- a technology that already existed in supercomputing -- for GPU-to-GPU communication. InfiniBand was designed from the start to be lossless, with credit-based flow control built into the protocol.
2019-2020: NVIDIA acquires Mellanox In March 2020, NVIDIA completed its $6.9 billion acquisition of Mellanox Technologies -- the dominant maker of InfiniBand switches and NICs. This was a strategic move: NVIDIA controlled the GPU, and now it controlled the network that connected GPUs. The DGX reference architecture became an integrated product.
2021-2023: RoCEv2 emerges as an alternative RoCEv2 (RDMA over Converged Ethernet v2) lets you run RDMA -- the same kernel-bypass technology as InfiniBand -- over standard Ethernet. For enterprises that already had Ethernet infrastructure, this was attractive. But it required making Ethernet lossless using PFC and ECN -- which is exactly what this course teaches.
2024-present: Two camps Today the industry has effectively split:
- InfiniBand (Quantum-2 NDR, Quantum-X800 XDR) -- used for hyperscale AI training clusters, cloud providers building dedicated AI infrastructure, and any deployment where absolute maximum performance matters
- Spectrum-X (RoCEv2 over Ethernet) -- used where Ethernet infrastructure already exists, for enterprises wanting multi-tenant GPU clusters, or where the operational simplicity of Ethernet outweighs the raw performance of InfiniBand
Neither is "wrong." The choice depends on workload, budget, and existing infrastructure.