Ethernet was never designed with GPU collectives in mind. Run an NCCL All-Reduce across 1,024 GPUs on a conventional Ethernet fabric and three failure modes arrive together. ECMP hash collisions place multiple elephant flows onto the same spine link while adjacent links sit idle — the hash function has no visibility into queue depth, so it cannot self-correct. Head-of-line blocking in switch port buffers causes a single large flow to stall smaller flows sharing the same egress queue. Most critically, PFC storm risk: when a receiver asserts PAUSE on a priority class, the pause propagates upstream hop by hop, eventually freezing unrelated flows in a deadlock cascade. Together they push effective utilisation well under half of the bandwidth you paid for, precisely during the All-Reduce synchronisation barrier where it matters most.
Spectrum-X is NVIDIA's answer: a vertically integrated, AI-optimised Ethernet platform that addresses all three pathologies simultaneously through hardware co-design rather than software workarounds. The platform has four components that must be present together to unlock its full capability. The Spectrum-4 switch ASIC provides the hardware adaptive routing engine that eliminates ECMP hash collisions at the packet level. The SuperNIC — BlueField-3 on DGX B200, ConnectX-8 on HGX B300, and never on H100 or H200 — provides the in-NIC reorder buffer that makes per-packet adaptive routing viable without triggering RoCEv2 Go-Back-N retransmit storms. The DOCA SDK exposes programmable packet processing on the BF3 ARM cores. NetQ provides fabric-wide telemetry with nanosecond-resolution event capture.
The "AI factory" framing is not marketing decoration -- it changes how you cable and how you draw the topology. A production build carries three separate fabrics with genuinely different traffic characteristics. The training fabric carries GPU-to-GPU collective traffic — bandwidth-sensitive, bursty, synchronous, and absolutely intolerant of head-of-line blocking. This is where Spectrum-4 leaf and spine switches live. The storage fabric carries checkpoint and dataset traffic — throughput-sensitive but more tolerant of latency jitter, typically terminated on dedicated SN4600C storage leaf switches. The OOB management fabric carries BMC, console, and orchestration traffic — lowest bandwidth, completely isolated from training and storage. Spectrum-X components span all three fabrics but in different roles.
A standard Spectrum deployment — same Spectrum-4 ASIC, same Cumulus Linux — without a SuperNIC at the endpoint is just a high-bandwidth switch. The Spectrum-X designation means the full stack is present and the adaptive routing engine is configured to use per-packet spraying with the SuperNIC's reorder buffer absorbing out-of-order delivery at the endpoint.
Where HGX B300 sits in this. The reference build this course uses is HGX B300: eight Blackwell Ultra GPUs per node, each with its own ConnectX-8 SuperNIC port at 800 Gb/s. That doubles per-node scale-out attachment relative to HGX B200 and pushes the leaf tier onto 800 GbE OSFP cages rather than 400 GbE. Every mechanism in this chapter applies unchanged; the port speeds, cage counts and buffer pressure all move up one notch.
# Verify you are looking at a Spectrum-X deployment (not standard Spectrum)
leaf-01# decode-syseeprom | grep -E "Product|Part"
Product Name: MSN5600-CS2FO
Part Number: 920-9N110-00R7-0N0
# SN5600 = Spectrum-4 ASIC confirmed
# Check that adaptive routing is in spectrum-x mode (not standard ECMP)
leaf-01# nv show router adaptive-routing
operational applied
------------------ ----------- -------
enable on on
mode spectrum-x spectrum-x
reorder-mode endpoint endpoint