• The DGX H100/H200 storage path uses the host CPU (Intel Xeon) as the NVMe-oF initiator. The ConnectX-7 NICs in Slot1 and Slot2 carry the storage traffic. No BlueField-3, no ARM offload.
• An NVMe-oF write is a two-phase RDMA exchange: an RDMA Send delivers the 64-byte SQE capsule to the target; the target then issues an RDMA Read against the initiator MR, and the initiator CX7 returns one or more RDMA Read Response frames using the RKEY from the SGL.
• The storage frame over RoCEv2 has the same Ethernet/IP/UDP/BTH outer structure as a compute frame. The BTH OpCode and payload differ: Send capsule for NVMe commands, RDMA Read Response for data.
• The storage fabric runs at 2:1 oversubscription, does not use PFC, and sets ECN thresholds higher than the compute fabric to handle bursty checkpoint writes.
• GDS (GPUDirect Storage) bypasses system DRAM entirely. The CX7 DMA-reads data directly from GPU HBM via a PCIe peer-to-peer window opened by the nvidia-fs driver.
• Storage fabric faults show up in nvme error-log, RDMA QP error counters (rdma stat), and switch buffer/ECN counters -- not in UFM or ibstat.

Continue to Chapter 18 -- OOB and Management Network ->