A single 400G switch with 64 ports can connect 64 DGX NICs directly. If each DGX node has 8 NICs, that is 8 nodes connected through one switch. During AllReduce, all 8 nodes send to all other nodes -- 8 x 400G = 3.2 Tb/s of total traffic, all staying within one switch. No spine needed. Perfect bisection bandwidth. No bottlenecks.
This is the 1-stage topology -- a single switch connecting all servers. It works beautifully for small clusters. The problem is radix.
Radix is the number of ports on a switch. Even the largest commercially available switches top out at 128 ports at 400G or 64 ports at 800G. With 128 x 400G ports and 8 ports per DGX node, a single switch can serve 16 DGX nodes -- 128 GPUs. That covers a BasePOD. For anything larger, you need more switches and a way to connect them.
Connecting switches to each other requires uplinks. Every port used as an uplink is a port not used for a DGX node. A switch with 64 ports that dedicates 16 to uplinks only has 48 ports for servers. The fundamental tension in fabric design: every uplink port is a server port you cannot use.
How topology stages determine cluster scale
Adjust switch radix to see how port count drives cluster capacity at each stage
The solution -- and the architecture that all large AI clusters use -- is the fat-tree: multiple layers of switches where lower layers connect to servers and upper layers interconnect the lower-layer switches. As you add layers, you can connect more servers. As you widen each layer, you get more bandwidth between the layers.
Real deployment: A DGX SuperPOD production rack. The image below shows the physical reality: DGX nodes stacked in compute racks on the left, switch racks in the centre carrying the leaf and spine layers, and the cable plant connecting them. Every cable you see represents one of the logical connections described in this chapter.
![DGX SuperPOD rack layout -- compute racks, switch racks, and structured cabling plant] Source: NVIDIA DGX SuperPOD Reference Architecture documentation Image placeholder: Search "NVIDIA DGX SuperPOD rack reference architecture" for the official NVIDIA documentation diagram showing the physical rack layout.