When PyTorch calls torch.save() on a checkpoint, the call stack eventually reaches the kernel NVMe-oF
initiator or the cuFile library. Either way, a block of GPU memory needs to land on a storage appliance.
The DGX H100 does not have a dedicated storage processor. The two dual-port ConnectX-7 NICs in Slot1
and Slot2 carry both storage traffic and in-band management. The host CPU (Intel Xeon) runs the NVMe-oF
initiator stack -- it is the CPU that maps namespaces, allocates queue pairs, and posts work requests.
This matters because the initiator is not close to the data. The model weights live in HBM, attached to the GPU over NVLink. To get those weights onto the wire, the data must cross PCIe from GPU to the CX7 NIC. The CX7 can DMA directly from GPU memory via GPUDirect, which is the GDS path covered in Act 6. The baseline path -- the one you learn first -- is the host CPU-mediated path where data goes GPU HBM -> system DRAM -> CX7 NIC -> storage switch -> appliance.
The storage fabric is a separate physical network from the compute fabric. In a BasePOD, the compute fabric uses QM9700 HDR InfiniBand or SN5600 Ethernet spine/leaf switches. The storage fabric uses SN4600C (or similar) 100GbE switches in a simpler topology: each DGX node connects its storage CX7 ports to dedicated storage leaf switches, which connect to the storage appliances. There is no reason to run the full rail-optimised fat-tree topology here -- the storage traffic pattern is not all-to-all.
The three physical fabrics on a DGX cluster:
- Compute fabric: GPU-to-GPU RDMA (RoCEv2 or InfiniBand). Eight CX7 HCA ports per DGX. Full bisection bandwidth. Rail-optimised topology.
- Storage fabric: NVMe-oF checkpoint and dataset I/O. Two dual-port CX7 NIC ports per DGX. 2:1 oversubscription acceptable. Simpler leaf topology.
- OOB/management fabric: BMC, SSH, UFM, DCGM telemetry. Separate 1GbE network. Covered in Ch18.