GPUDirect Storage (GDS) eliminates the CPU and system DRAM from the checkpoint write path. The data moves directly: GPU HBM -> PCIe -> CX7 NIC -> storage fabric. No cudaMemcpy. No pinned DRAM bounce buffer. The CPU still manages the control plane -- posting WQEs, handling CQEs, running the NVMe-oF initiator state machine -- but it is not in the data path.

How GDS works at the hardware level

The cuFile library (libcufile.so) coordinates between the GPU driver and the NIC driver. At session open time, cuFile:

1. Calls cuMemCreate to allocate the source HBM buffer (or uses an existing tensor allocation).
2. Calls ibv_reg_mr via the RDMA verbs layer to register the HBM buffer directly as an RDMA MR on the CX7 NIC. The NIC firmware learns the IOVA (I/O Virtual Address) mapping for the HBM pages via the PCIe peer-to-peer DMA window opened by the GPU driver.
3. The registered RKEY is placed in the SGL of the NVMe SQE, pointing directly at HBM.
4. When the target issues RDMA Read against that RKEY, the CX7 fetches data from HBM over PCIe without touching system DRAM.

The practical benefit is PCIe bandwidth is used once (HBM -> NIC) instead of twice (HBM -> DRAM -> NIC). At PCIe 5.0 x16 (~64 GB/s), this cuts the PCIe load roughly in half for large sequential checkpoint writes.

GDS requirements and limitations

• Requires nvidia-fs kernel module (the GPUDirect Storage driver).
• The storage appliance must support GDS on the target side (WEKA, DDN, NetApp, Pure FlashBlade with NVMe-oF RDMA all support it).
• GDS does NOT work over NVMe/TCP (TCP binding cannot use peer-to-peer RDMA).
• GDS is most beneficial for large sequential writes (checkpoints, large dataset reads). For small random I/O, the CPU-mediated path often performs comparably because the PCIe round-trip latency dominates.
• Verify the nvidia-fs module is loaded: lsmod | grep nvidia_fs.
• Verify the storage connection is established via RDMA: nvme list-subsys and check transport type.

GDS diagnostic check

# Confirm GPUDirect Storage driver is present
lsmod | grep nvidia_fs

# Run GDS built-in validation
gds_check

# Verify NVMe-oF connection is RDMA, not TCP
nvme list-subsys
# Look for "trtype=rdma" in the output

# Check for any GDS error in kernel log
dmesg | grep -i "gds\|cufile\|nvidia-fs" | tail -20