Before GPUDirect Storage existed, loading a training batch into GPU memory required five distinct operations involving the host CPU.

The pre-GDS path:

The application calls a read from the NVMe-oF storage appliance. The request goes to the kernel. The kernel asks the NIC to fetch the data. The data arrives at the NIC and is DMA'd into a kernel-space buffer in system RAM. The CPU copies it from kernel-space RAM to a user-space application buffer. The application then calls cudaMemcpy to move the data from the CPU-side application buffer into GPU HBM across PCIe.

Count the hops: storage -> NIC -> system RAM (kernel buffer) -> system RAM (user buffer) -> PCIe -> GPU HBM. Five hops. Two CPU memory copies. Every byte of training data touches CPU RAM twice before the GPU sees it.

On a DGX H100 with 80 GB of HBM per GPU and 8 GPUs, the node needs to fill 640 GB of HBM at the start of training. Pre-GDS, that requires moving 640 GB through CPU RAM twice -- 1.28 TB of CPU memory bandwidth consumed for data that the CPU never uses. The 2 Intel Xeon CPUs in a DGX H100 have approximately 400 GB/s of combined memory bandwidth. At full theoretical throughput, pre-loading 640 GB of training data would take at minimum 3.2 seconds -- and that assumes the CPUs are doing nothing else.

The GDS path:

With GPUDirect Storage, a direct DMA channel is established between the NVMe-oF storage target and GPU HBM, bypassing system RAM entirely. The GPU registers a memory buffer and posts a read request. The host NVMe-oF initiator stack (running on the host CPU, using the dual-port ConnectX-7 cards in Slot1/Slot2 as its network interface) translates this into an NVMe-oF command over the storage fabric. The storage appliance transmits the data. The storage ConnectX-7's DMA engine writes it directly into GPU HBM across PCIe -- without the data ever appearing in system RAM, without a CPU memory copy, without a cudaMemcpy.

The path collapses to: storage appliance -> storage CX7 DMA -> PCIe -> GPU HBM. Three hops instead of five. The CPU manages the NVMe-oF command queuing but does not touch the data itself.

Why this matters for checkpoint write speed:

The same mechanism runs in reverse for checkpoint writes. The GPU has computed and accumulated gradients in HBM. The checkpoint must be written to durable storage. With GDS, the storage CX7's DMA engine reads from GPU HBM directly and streams the data to the NVMe-oF target. The 140 GB checkpoint flows from GPU memory to storage with the CPU managing commands but never copying data.

The 800 Gb/s figure from Chapter 0 -- 2x dual-port ConnectX-7 storage cards at 400G each -- is the raw theoretical bandwidth of this path. Real-world checkpoint throughput is typically 60-80% of theoretical, constrained by PCIe bandwidth, storage appliance write throughput, and the parallel file system's write striping efficiency.