Chapter 0 gave you the storage fabric in three sentences. This chapter gave you the engineering underneath those sentences.
You understand why the storage fabric is physically separate -- not as a best practice, but because mixing storage and compute traffic on the same switches creates PFC interactions and buffer contention that degrades both. The traffic pattern mismatch between sustained sequential storage transfers and bursty synchronised AllReduce traffic is fundamental, not incidental.
You understand GPUDirect Storage: the five-hop pre-GDS path collapses to three hops by having the storage ConnectX-7 DMA engine write directly from GPU HBM to the NVMe-oF target, eliminating CPU memory copies. At 640 GB of GPU HBM per DGX node, eliminating the CPU copies is not a minor optimisation -- it is the difference between loading data in 3 seconds and loading data in 1.4 seconds.
You understand NVMe-oF: NVMe command queues routed over RDMA instead of PCIe, with the host NVMe-oF initiator (using the storage CX7 ports) acting as the initiator and the storage appliance as the target. RDMA transport handles the data movement without CPU copies; GDS extends this so the data flows directly between GPU HBM and storage without touching system RAM.
You understand why dedicated storage ports are separate from compute NICs -- not because ConnectX-7 cannot support NVMe-oF (it can), but because the 8 compute NICs are fully occupied with AllReduce RDMA and sharing them with storage traffic would degrade both. The 2 dedicated storage CX7 cards provide 800 Gbps of storage bandwidth completely independent of the compute fabric.
You understand parallel file systems: data striped across many storage nodes so that many DGX nodes reading simultaneously do not bottleneck on a single storage server. BeeGFS for operational flexibility and open-source control; WEKA for managed performance and AI-optimised I/O.
You understand checkpoint strategy: the MTBF-driven optimisation that determines the right checkpoint frequency, the real cost of a full Adam checkpoint (P x 8 bytes, not P x 2), and why gradient accumulation complicates checkpoint correctness.
And you understand what storage fabric failure looks like from the network side -- different tools, different devices, different diagnostic path -- so that the next time training stalls without any compute fabric errors, you know where to look first.