ProDeploy — Learn the fabric that keeps AI clusters alive

NVMe-oF is the protocol that makes the GDS path possible. To understand why it matters, you need to understand what NVMe is and what "over Fabrics" adds.

NVMe -- the protocol underneath

NVMe (Non-Volatile Memory Express) is a command queue protocol designed for SSDs attached directly to a CPU via PCIe. NVMe replaced SATA and SAS for high-performance storage because it was built from the ground up for flash -- not adapted from spinning disk protocols. An NVMe device presents a set of I/O queues. The host posts commands (read, write) to these queues and the SSD executes them asynchronously.

The key property for GPU workloads: NVMe commands are posted by the initiator (the host side) and executed by the target (the SSD side) without round-trip acknowledgement for each byte. The host posts a write command and the SSD handles the DMA transfer. This is structurally identical to the RDMA queue model -- which is exactly why GDS can bridge them.

NVMe-oF -- the same protocol over a network

NVMe-oF takes the NVMe command queue protocol and routes it over a network fabric instead of a PCIe bus. The host NVMe-oF initiator presents a virtual NVMe device to the GPU. From the GPU's perspective, this looks like a local PCIe NVMe SSD. Underneath, every NVMe command the GPU issues travels over the storage fabric to a remote storage appliance that executes it and returns the result.

NVMe-oF transport options — click to compare

Three ways to carry NVMe commands over a network. RDMA is the AI cluster standard.

Latency~2–10 µs

Low

CPU overheadNear-zero — CX7 NIC handles data-path DMA

Low

NVMe commands transmitted using RDMA verbs over RoCEv2. The host CPU NVMe-oF initiator delivers the NVMe SQE capsule via RDMA Send through the CX7 NIC; the target then pulls the data payload via RDMA Read against the initiator's registered MR. No TCP overhead. The CPU manages command queuing but does not touch data bytes on the data path. With GDS, data flows directly GPU HBM → CX7 → storage with no system RAM copies.

Switch required

Ethernet with ECN (PFC optional)

Deployed in

AI training clusters, HPC storage fabrics

Why RDMA wins: Only NVMe-oF/RDMA enables GPUDirect Storage's zero-copy path. The CX7 NIC's RDMA engine handles all data-path DMA — the target pulls checkpoint data from the initiator MR via RDMA Read, writing GPU HBM contents to the storage appliance without host CPU involvement.

Three transport options exist:

NVMe-oF/TCP -- NVMe commands encapsulated in TCP packets. Runs on any standard Ethernet switch. Highest compatibility, lowest performance. Each NVMe command requires TCP ACKs, kernel socket processing, and CPU involvement at both ends. Appropriate for backup storage, cold data tiers, or environments where the storage fabric is shared with general workloads.

NVMe-oF/RDMA (RoCEv2) -- NVMe commands transmitted using RDMA semantics over RoCEv2. The host NVMe-oF initiator issues NVMe commands as RDMA write operations. No TCP overhead. No CPU involvement on the data path at either end. The combination of RDMA transport and GDS completes the zero-CPU-copy path described in Act 2. This is the standard for AI cluster storage fabrics.

NVMe-oF/FC -- NVMe commands over Fibre Channel. Legacy enterprise storage environments. Not used in AI cluster deployments.

Why RDMA transport matters for the checkpoint write path:

When a checkpoint is written using NVMe-oF/RDMA via GDS, the transaction flow is: GPU HBM -> PCIe -> storage CX7 DMA engine -> RDMA write over storage fabric -> NVMe target's DMA engine -> storage appliance NVMe SSD. The The storage CX7 and the storage appliance both use RDMA queue semantics. No system RAM. No CPU copies on either end. The storage appliance's NVMe-oF target processes the write in hardware.

A single RDMA write for a 140 GB checkpoint is actually many RDMA operations -- the NVMe-oF stack chunks the transfer into RDMA Write operations that fit within the storage appliance's maximum transfer unit. Each chunk is DMA'd, transmitted, and acknowledged independently. The checkpoint completes when all chunks are acknowledged. The GPU sees the operation as a single NVMe write command completion.