Before we reach the external network, there is a problem to solve within a single server: how do multiple GPUs talk to each other efficiently?

The answer NVIDIA built is NVLink -- a proprietary high-speed interconnect that directly connects GPUs to each other, bypassing the CPU and PCIe bus entirely.

The DGX H100 uses 4th-generation NVLink, connected through 4 x 3rd-generation NVSwitch chips inside the chassis. This gives each GPU 900 GB/s of bidirectional bandwidth to communicate with all other GPUs in the same node -- roughly 18 times faster than the external fabric connections we will see shortly.

The naming is a common source of confusion: NVLink and NVSwitch use separate generation counters. The H100 has 4th-gen NVLink but 3rd-gen NVSwitch. For reference: the DGX A100 used 3rd-gen NVLink (600 GB/s) with 2nd-gen NVSwitch. The Blackwell B200 uses 5th-gen NVLink (1.8 TB/s). This course focuses on H100 -- 4th-gen NVLink, 3rd-gen NVSwitch, 900 GB/s.

3rd-generation NVSwitch: more than a crossbar

The 3rd-generation NVSwitch in the DGX H100 is not just a high-speed crossbar switch. It adds hardware acceleration for collective operations -- specifically multicast and in-network reduction. During an intra-node AllReduce, the NVSwitch can aggregate gradient data as it flows through the chip, rather than requiring all 8 GPUs to exchange data pairwise. This reduces the work that the intra-node AllReduce requires and lowers the latency of gradient synchronisation within the chassis.

This is the NVSwitch equivalent of what SHARP does on InfiniBand switches -- in-network compute that reduces communication volume by doing part of the mathematical work inside the switch fabric itself. The difference: NVSwitch SHARP applies to intra-node AllReduce (the 900 GB/s NVLink domain). IB SHARP applies to inter-node AllReduce (the external fabric). Chapter 0 Act 6 covers IB SHARP. Chapter 11 covers the NVLink Switch System, which extends this capability across nodes.

The aggregate NVSwitch throughput for the DGX H100 is 13.6 Tb/s (compared to 7.2 Tb/s in the A100's 2nd-generation NVSwitch), matching the bandwidth increase from NVLink 3 -> NVLink 4.

This bandwidth ratio is the reason that tensor parallelism -- splitting a single model layer across multiple GPUs -- runs within a single DGX node. You use the fast path for the most communication-intensive operations.

But NVLink only works within one chassis. The moment you need more than one server -- which happens quickly when you are training a 70B parameter model -- you need an external network. And that is where the real engineering challenge begins.