NVIDIA publishes two reference architectures for DGX clusters. Understanding their exact port allocation and switch counts gives you a concrete framework for topology calculations.
NVIDIA reference designs — BasePOD vs SuperPOD
DGX BasePOD
Scale: Up to 32 DGX H100 nodes = 256 GPUs
Compute fabric structure (2-stage, non-blocking):
- 8 leaf switches (one per rail), each: 32 downlinks to DGX NICs + 32 uplinks to spine
- 32 spine switches, each: 8 downlinks to leaf switches (one per leaf)
- Switch hardware: Spectrum-X SN5600 (64 ports x 400G each)
How the port math works: Each SN5600 has 64 ports. A leaf switch uses:
- 32 ports facing DGX nodes (4 DGX nodes x 8 NICs each? No -- 32 nodes x 1 NIC per rail = 32 downlinks)
- 32 ports facing spine switches
A spine switch uses all 8 ports to connect to the 8 leaf switches. All 32 spine switches do the same, giving each leaf switch 32 uplink paths -- one to each spine switch.
Bisection bandwidth check: Split 32 nodes into two halves of 16. Each node has 8 NICs at 400G = 3.2 Tb/s per node. 16 nodes = 51.2 Tb/s of potential traffic crossing the bisection. How much spine bandwidth crosses the cut? Each leaf switch contributes 32 uplinks x 400G = 12.8 Tb/s to the spine layer. 8 leaf switches x 12.8 Tb/s = 102.4 Tb/s total spine bandwidth. Split evenly = 51.2 Tb/s per half. Full bisection bandwidth achieved.
AllReduce on a BasePOD can run at exactly line rate, bounded only by NIC speed.
Real deployment: The NVIDIA DGX BasePOD wiring diagram. Shows 32 DGX H100 nodes across 4 compute racks, connected to 8 leaf switches, which connect to 32 spine switches in a separate switch rack.
![DGX BasePOD reference wiring diagram -- 32 nodes, 8 leaf, 32 spine] Source: NVIDIA DGX BasePOD Reference Architecture document, available at developer.nvidia.com/dgx-basepod Image placeholder: The NVIDIA BasePOD documentation contains an official topology diagram. Search "NVIDIA DGX BasePOD reference architecture PDF" for access.
DGX SuperPOD
Scale: Up to 256 DGX H100 nodes = 2,048 GPUs (standard configuration)
Compute fabric structure (3-stage):
- 64 leaf switches (8 rails x 8 groups of 8 nodes per rail)
- 64 spine switches (connecting leaf to core)
- Additional core layer for inter-group traffic
Why 3 stages are needed: With 256 nodes and 8 NICs per node, you have 2,048 NIC connections. Even with 64-port switches, 2,048 / 32 (downlinks per switch) = 64 leaf switches. 64 leaf switches need 64 uplink ports each = 4,096 uplinks. With 64 ports per spine switch and all ports used for downlinks = 64 switches handle only 64 x 64 = 4,096 connections. This works for a 2-stage design, but the bandwidth at the spine layer would be oversubscribed if all 256 nodes tried to AllReduce simultaneously.
The SuperPOD uses a 3-stage design with a core layer to maintain non-blocking bandwidth across the full 256-node cluster.
| BasePOD | SuperPOD | |
|---|---|---|
| DGX H100 nodes | 32 | 256 |
| Total GPUs | 256 | 2,048 |
| Total NICs | 256 | 2,048 |
| Fabric stages | 2 | 3 |
| Leaf switches | 8 | 64 |
| Spine/agg switches | 32 | 64+ |
| Core switches | 0 | 64 |
| Max bisection bandwidth | Full | Full (with correct config) |
| AllReduce hop count | 2 | 4 |
The SuperPOD's extra stages double the AllReduce hop count -- 2 hops in BasePOD (down to leaf, up through spine), 4 hops in SuperPOD (down to leaf, up to spine, up to core, back down). Each hop adds microseconds of latency. For a training run that executes millions of AllReduce steps, the latency difference compounds.
This is why NVIDIA's reference designs use the highest-radix switches available -- to minimise the number of fabric stages needed and keep AllReduce hop counts low.