The host CPU initiator posts a work request to the CX7 NIC. The WQE (Work Queue Entry) describes the first operation: an RDMA Send to deliver the SQE capsule. The data payload moves separately — the target issues an RDMA Read against the initiator's registered MR, and the initiator CX7 responds with RDMA Read Response frames (handled entirely by NIC firmware). The CX7 firmware processes the WQE from its send queue, the same pipeline described in Ch16 Act 3 -- but this time the source of the data is not GPU RDMA memory registered by NCCL. It is either system DRAM (host CPU path) or HBM (GDS path).

Host CPU path (baseline)

1. PyTorch/framework copies checkpoint tensor from GPU HBM to pinned system DRAM via cudaMemcpy or similar. This crosses PCIe. On H100, PCIe 5.0 x16 gives approximately 64 GB/s peak per slot.
2. The CPU NVMe-oF initiator (nvme-rdma kernel module) has already registered the pinned DRAM buffer as an RDMA MR (Memory Region) with the CX7 NIC at connection setup time.
3. The initiator posts an RDMA Send WQE pointing at the SQE capsule (64 bytes in DRAM).
4. The CX7 DMA-reads the SQE capsule from DRAM into its transmit buffer.
5. The CX7 builds the full frame and puts it on the wire toward the storage switch.
6. Target receives the SQE, issues an RDMA Read WQE against the initiator's registered MR.
7. CX7 on the initiator side receives the incoming RDMA Read request, DMA-reads the data payload from the pinned DRAM MR, and sends it back as RDMA Read Response frames.
8. When the transfer completes, the CX7 posts a completion event to the initiator's CQ.
9. The initiator sends the CQE capsule back to the target, completing the transaction.

The bottleneck in this path is the PCIe hop between GPU and DRAM (step 1). A 4 MiB checkpoint block crosses PCIe twice: once from HBM to DRAM, once from DRAM to CX7. GPUDirect Storage eliminates the DRAM staging entirely.

Memory registration and RKEY lifetime

NVMe-oF RDMA connections use persistent memory regions registered at queue pair setup time -- not per-I/O like some NCCL flows. The nvme-rdma driver registers the DMA buffer pool once when the connection is established. The RKEY published in the SGL is valid for the lifetime of the connection. If the connection drops (link failure, appliance reboot), the MR is invalidated and the connection must be re-established. This is why storage fabric link stability matters more than instantaneous bandwidth: a flap invalidates in-flight I/O and forces reconnection.