The reason to deploy a NVLink Switch System is the bandwidth gap between scale-up and scale-out. This section makes that gap concrete.
Bandwidth Comparison
Logarithmic scale — the gap is larger than it looks. Toggle between per-GPU and full 256-GPU cluster views.
256-GPU all-to-all bandwidth
In a 256-GPU NVLink Switch System:
- Each GPU contributes 900 GB/s bidirectional NVLink bandwidth
- Aggregate all-to-all bandwidth: 57.6 TB/s
- This is the bandwidth available for AllReduce, tensor parallel operations, and any collective communication
In a 256-GPU IB NDR fabric (8 x ConnectX-7 per node, 32 nodes):
- Each GPU contributes 400 Gbps = 50 GB/s via IB
- Aggregate bandwidth: 32 x 8 x 400 Gbps = 102.4 Tbps = 12.8 TB/s
- But this is bisection bandwidth across the fat-tree -- actual per-GPU AllReduce bandwidth is approximately 3.2 Tb/s per node = 400 Gbps per GPU in practice, accounting for the fat-tree topology
The NVLink Switch System delivers approximately 18x more all-to-all bandwidth than the equivalent IB scale-out fabric at the same GPU count.
20.5 TB unified HBM pool
256 H100 SXM5 GPUs x 80 GB HBM3 per GPU = 20.5 TB of GPU memory, addressable as a single flat pool.
This number matters because of how large language models are trained. A 70B parameter model in BF16 requires approximately 140 GB of parameter storage. With 20.5 TB of unified memory, the entire model fits in HBM with room for activations, gradients, and optimizer state -- without any tensor parallelism sharding. Tensor parallelism (splitting weight matrices across GPUs) exists specifically to distribute a model that is too large for a single GPU's memory. When the memory pool is 20.5 TB, that constraint disappears for all but the very largest models being trained today.
The practical consequence: jobs that would require complex 4-way or 8-way tensor parallelism on an IB cluster run without tensor parallelism on a NVLink Switch System. Tensor parallel communication is all-to-all -- eliminating it removes the most bandwidth-intensive collective from the training loop entirely.
Why AllReduce busbw is a misleading metric here
In Chapter 8, you learned to read NCCL busbw output as the primary performance indicator for AllReduce. On an IB or RoCEv2 cluster, busbw is the right metric because AllReduce is the dominant inter-GPU operation.
On a NVLink Switch System, the picture changes. AllReduce busbw will be dramatically higher (scaling with NVLink bandwidth), but the more important metric is end-to-end training throughput -- measured as training samples per second or FLOP/s utilisation. The NVLink Switch System can make AllReduce so fast that other bottlenecks (data loading, compute, checkpoint I/O) become the limiting factor instead. Optimising busbw on a system where AllReduce takes microseconds is the wrong target.