The phrase "rail-optimised topology" sounds like a software choice. It is actually a physical wiring constraint -- and that constraint is deliberate.
In a DGX SuperPOD, every GPU 0 across every node connects to Leaf Switch 0. Every GPU 1 across every node connects to Leaf Switch 1. And so on for all 8 rails. The cables are physically run this way from day one and never changed.
The reason: AllReduce traffic between GPU Ns only ever needs one switch hop.
When 32 DGX nodes run data-parallel training, the GPU 0s on all 32 nodes need to AllReduce their gradients. In rail-optimised topology, every one of those GPU 0s is directly connected to Leaf Switch 0. The AllReduce traffic flows down from each GPU 0, through Leaf Switch 0, and back up to all other GPU 0s. The spine switches are never involved. One hop each way.
Rail-optimised topology — why AllReduce never touches the spine
AllReduce stays within one leaf switch
In a rail-optimised topology, every GPU 0 across every DGX node connects to Leaf Switch 0. When GPU 0s perform AllReduce — sending gradient updates to all other GPU 0s — that traffic flows down from each GPU 0, into Leaf Switch 0, and back up to all other GPU 0s. It never touches the spine. One switch hop in each direction. The spine layer is completely bypassed for this traffic pattern.
The insight
This is the key insight: by wiring all GPU Ns to the same leaf switch, AllReduce traffic between GPU Ns is topologically local — it never competes for spine bandwidth. You get dedicated, non-blocking throughput for the most critical communication pattern in AI training.
Compare this to a conventional fat-tree where GPUs connect to whichever leaf switch is in their rack regardless of GPU index. AllReduce traffic from GPU 0s, GPU 1s, and GPU 7s all mix on the same spine uplinks, competing for bandwidth. Rail optimisation eliminates this competition by making the wiring match the traffic pattern.
Oversubscription and why AI fabrics cannot tolerate it
Oversubscription ratio is the ratio of potential total server bandwidth to available fabric bandwidth. In a conventional enterprise data center, 4:1 or even 8:1 oversubscription is common -- network architects assume that not all servers will send at full rate simultaneously. The oversubscription ratio is an intentional cost optimisation.
AI training breaks this assumption completely. During AllReduce, every GPU on every node sends at full line rate simultaneously. There is no statistical multiplexing. The "not everyone sends at once" assumption -- the entire justification for oversubscription -- is false for AI training traffic.
A 2:1 oversubscribed fabric in an AI training cluster does not give you 50% of potential throughput. It creates a congestion event on every single synchronisation barrier, causing drops or PFC stalls that compound across thousands of training steps. This is why AI fabric designs target 1:1 (non-blocking) from leaf to spine. Some aggressive designs even use undersubscription (more uplinks than server-facing ports) to ensure each leaf port has dedicated egress buffer capacity during peak burst.
The tradeoff: cabling a rail-optimised SuperPOD is complex and rigid. You cannot repatch cables without redesigning the fabric. But at 400G per port, you want every advantage you can get -- and a single additional spine hop adds microseconds that compound across millions of training steps.
Three physically separate networks, each with different requirements:
Compute fabric (InfiniBand NDR or RoCEv2) The GPU training network. Every DGX node has 8 ConnectX-7 NICs, each connecting to a different leaf switch. This is the rail-optimised topology. The leaf switches connect to spine switches to form a fat-tree fabric. This network must be lossless and run at full bandwidth to all nodes simultaneously.
Storage fabric (Ethernet) A separate Ethernet network connecting DGX nodes to parallel storage systems. Uses 400GbE Spectrum switches. GPUDirect Storage (GDS) allows GPUs to read training data directly from NVMe storage systems -- bypassing the host CPU and system RAM entirely. Separate from the compute fabric so storage traffic cannot interfere with training communication.
Management network (1GbE / 10GbE) Out-of-band management. BMC access, OS provisioning, NVIDIA UFM (Unified Fabric Manager), Base Command Manager. Standard enterprise Ethernet switches. Completely separate from compute and storage. This is the network you use when the other two networks are misbehaving.