NVMe-oF is NVMe over Fabrics. The core insight is that the NVMe command queue model -- submission queue entries (SQE) and completion queue entries (CQE) -- is preserved exactly but the queue pair is no longer a PCIe register on a local SSD. It is an RDMA queue pair to a remote storage target over a network fabric.

The transport binding that matters here is NVMe-oF over RoCEv2 (also called NVMe/RoCE or NVMeoF-RDMA). NVMe-oF also supports a TCP transport (NVMe/TCP) which does not require RDMA-capable NICs but has higher latency. In DGX deployments with ConnectX-7 NICs, the RDMA transport is always preferred.

The NVMe Submission Queue Entry

An SQE is 64 bytes. It contains the NVMe command opcode (0x01 = Write, 0x02 = Read), the Namespace Identifier (NSID), the logical block address (LBA), the number of logical blocks, and the SGL (Scatter/Gather List) descriptor that points to the data buffer. In the NVMe-oF RDMA binding, the data transfer itself is a separate RDMA operation. The SQE is a capsule -- a wrapper that triggers the data movement.

The complete NVMe-oF RDMA exchange for a write:

1. Initiator sends a SEND capsule (the SQE) to the target over RDMA.
2. Target issues an RDMA Read Work Request against the initiator's registered memory region (the RKEY published in the SQE SGL). The initiator CX7 responds autonomously with RDMA Read Response frames carrying the checkpoint payload. No initiator CPU instruction touches the data.
3. Target sends a SEND capsule back (the CQE) to notify completion.

Step 2 uses the RKEY mechanism from Ch16: the initiator registers a memory region, publishes the virtual address and RKEY in the SGL, and the target performs an RDMA Read against that registration. The CX7 NIC enforces memory region boundaries just as it does for compute traffic.

Capsule sizes

The SQE capsule: 64-byte NVMe command + optional inline data (for small I/O). The CQE capsule: 16-byte NVMe completion. The data payload for a checkpoint write is carried via RDMA Read (target-pull): the target issues an RDMA Read Work Request against the initiator's pre-registered Memory Region, and the initiator's CX7 responds with RDMA Read Response frames containing the payload. For a 4 MiB checkpoint write, the SQE capsule is tiny (64 bytes) and the data moves as a series of RDMA Read operations initiated by the target.

NVMe-oF discovery

Before any I/O, the initiator performs NVMe-oF discovery: it connects to a well-known discovery controller at TCP port 8009 on the target and receives a list of subsystems and namespaces. Port 8009 is the IANA-assigned NVMe-oF discovery port and is always reached over TCP -- this is mandated by the NVMe-oF specification regardless of whether the data transport will be RDMA or TCP. The discovery response contains the target NQN (NVMe Qualified Name), the transport address (the RDMA portal IP), and the transport type (rdma). The initiator then establishes RDMA queue pairs to each target subsystem it wants to use.