NVLink Switch hardware: External chassis containing 3rd-generation NVSwitch chips, connected via OSFP to DGX nodes. Same physical cable as IB/RoCEv2 -- firmware mode selects the protocol. Maximum scale: 32 DGX H100 nodes, 256 GPUs, 2:1 tapered fat-tree topology.
57.6 TB/s at 256 GPUs: The all-to-all bandwidth of a 256-GPU NVLink Switch System -- approximately 18x greater than the equivalent IB fabric. This comes from the 900 GB/s per-GPU NVLink bandwidth extended across node boundaries.
20.5 TB unified HBM pool: 256 x 80 GB, addressable as a single flat space. Eliminates tensor parallelism overhead for all but the largest models in production today.
NVLink Network address space: Hardware address translation that enables tenant isolation at the NVSwitch ASIC -- not at the Subnet Manager level. More tamper-resistant than IB P_Keys but less mature in management tooling.
SHARP at 256-GPU scale: 192 TFLOPS of in-network compute for gradient aggregation. Reduces GPU stall time during AllReduce on NVLink Switch -- complementary to, not a replacement for, IB SHARP on the scale-out fabric.
Fabric Manager: The NVLink fabric's control plane, and the structural counterpart to the InfiniBand Subnet Manager -- it routes among NVSwitch ports, registers GPUs into the fabric, and monitors for NVLink errors. Its signature failure is cudaErrorSystemNotReady, which surfaces as an application error rather than a network one. Pre-installed on DGX; installed by hand on HGX.
nvbandwidth: The scale-up fabric's ib_write_bw. An all-pairs matrix at roughly 670 GB/s per pair on a healthy node; a single low row and matching column localises a degraded endpoint. Measures throughput, where nvidia-smi nvlink --status only reports state.
Scale-up vs scale-out decision: NVLink Switch wins for <=256 GPU jobs with tensor-parallel or MoE workloads. IB wins for >256 GPUs and pipeline-parallel jobs. Most production clusters use both.
Diagnostics: No SM, no adaptive re-routing. nvidia-smi nvlink --status and DCGM NVLink counters replace ibstat and UFM. A failed lane means degraded bandwidth -- there is no automatic rerouting around it.