The NVLink Switch System extends the scale-up fabric across the node boundary. The hardware consists of external NVLink Switch modules -- rack-mounted chassis containing 3rd-generation NVSwitch chips, connected to DGX nodes with OSFP cables.

Physical connectivity

The NVLink Switch module uses the same OSFP form factor as the DGX H100's compute ports. The ConnectX-7 HCAs on a DGX node have a firmware mode that determines which fabric they connect to: InfiniBand, RoCEv2, or NVLink. When configured for NVLink, those same 400G OSFP ports carry NVLink traffic instead of IB/Ethernet traffic.

This means a DGX H100 node connected to a NVLink Switch System has its compute ports committed to scale-up. The IB/RoCEv2 scale-out fabric is either absent or served by a separate set of ports -- DGX H100 nodes in NVLink Switch configurations typically have dedicated IB connectivity for storage and management, but the primary GPU-to-GPU traffic moves on NVLink.

Topology: 2:1 tapered fat-tree

The NVLink Switch System uses a 2:1 tapered fat-tree: two tiers of NVLink Switch modules, with more bandwidth at the leaf tier than the spine tier. This is intentional -- AllReduce traffic is largely within groups of 8 GPUs (one DGX node), and the tapered design provides sufficient bandwidth for inter-node AllReduce while keeping switch module count manageable.

Maximum scale: 32 DGX H100 nodes connected to a single NVLink Switch System, yielding 256 GPUs.

The switch module count scales with node count:

• 8 nodes (64 GPUs): 1 tier of NVLink Switch modules
• 32 nodes (256 GPUs): 2 tiers (leaf + spine NVLink Switch modules)

Cabling

Same physical cable: 400G OSFP with the same DAC/AOC options covered in Chapter 9. The NVSwitch firmware running in the switch module negotiates the NVLink protocol over the OSFP physical layer. From a physical layer perspective, a NVLink Switch cable is indistinguishable from a ConnectX-7 IB cable -- same connector, same reach constraints, same signal integrity considerations.