The H100-generation NVLink Switch System is an optional tier bolted onto a cluster of otherwise independent nodes. The rack-scale Grace-Blackwell systems invert that relationship: the NVLink domain is the machine, and the scale-out fabric is what hangs off it. If you are evaluating GB300 NVL72, this act is the difference that matters; otherwise it is optional reading.
The shape of the rack. A GB300 NVL72 holds 72 Blackwell Ultra GPUs and 36 Grace CPUs across 18 compute trays. Nine NVLink switch trays, each carrying two NVSwitch ASICs, close a fully connected 72-GPU domain -- every GPU has a direct NVLink path to every other GPU in the rack -- at 130 TB/s of aggregate NVLink bandwidth. Roughly 20 TB of HBM3e and 17 TB of LPDDR5X are addressable inside that domain, and the whole rack is liquid-cooled with a budget of up to 142 kW.
What that does to the scale-out fabric. Each GPU still presents 800 Gb/s to the outside world through a ConnectX-8 SuperNIC -- four per compute tray, 72 per rack -- into Quantum-X800 InfiniBand or Spectrum-X Ethernet. But a collective that fits inside 72 GPUs never leaves the rack. Compared with an 8-GPU node, the ratio of scale-up to scale-out bandwidth widens by an order of magnitude, and the traffic that does cross the spine is fewer, larger and burstier flows. That profile stresses adaptive routing harder and raw bisection bandwidth less.
Three operational consequences. The unit of maintenance becomes the rack, not the node -- there is no draining one GPU out of an NVLink domain the way you drain a node out of a partition. The health checks in Chapter 29 acquire a whole extra domain: NVLink link state, error counters and fabric-manager status across nine switch trays, verified before any scale-out test is meaningful. And validation splits in two, because an all-reduce at 72 GPUs measures NVLink rather than your fabric, and will produce a number so much larger than a node-based cluster's that it is easy to mistake for a passing scale-out result. Chapter 30 works through how to keep the two apart.
Looking ahead. NVLink 6 raises per-GPU scale-up bandwidth to 3.6 TB/s and the rack-scale domain's aggregate to 260 TB/s in the announced Vera Rubin generation, with in-network reduction on the switch trays. The direction of travel is consistent: each generation moves more collective traffic inside the rack and leaves the scale-out fabric carrying a smaller but harsher share.