The DGX H100 has 8 single-port ConnectX-7 HCAs for compute and 2 dual-port ConnectX-7 cards (Slot1/Slot2) for storage and in-band management. This act explains why that boundary exists where it does -- and why blurring it would break the compute fabric.

The ConnectX-7 is fully occupied

During AllReduce across a 256-GPU BasePOD, each ConnectX-7 maintains active RDMA Queue Pairs to every other GPU it communicates with. In a data-parallel job where all 256 GPUs are exchanging gradients, each NIC may have 255 active QPs, each with an outstanding RDMA write in flight.

At 400 Gbps per NIC and 8 NICs per node, the DGX node is sourcing 3.2 Tbps of compute fabric traffic during the AllReduce burst. Every available RDMA queue, every send buffer, and every completion queue on all 8 ConnectX-7 NICs is consumed by the in-flight AllReduce transfers.

Adding NVMe-oF storage traffic to a ConnectX-7 during AllReduce would require one of the following -- all unacceptable:

Sharing RDMA queue resources with storage. The NVMe-oF initiator would consume QPs and completion queues that AllReduce needs. QP exhaustion causes AllReduce initialisation failures.

Sending storage traffic on a compute fabric switch port. Storage traffic competes with AllReduce traffic for switch buffer. PFC storms become possible. The lossless guarantee is at risk.

Adding a ninth ConnectX-7 per node for storage. Doubles the NIC cost. Changes the chassis layout. Changes the switch port count. The DGX H100 was designed around 8 compute NICs.

The storage CX7's dedicated bandwidth

The two dual-port ConnectX-7 cards in Slot1/Slot2 provide 800 Gbps of dedicated storage bandwidth per node (2 cards x 2 ports x 200G, or 2 cards x 1 port active at 400G each, depending on bonding config). These ports connect to storage fabric Ethernet switches -- physically separate from the compute fabric. The 800 Gbps storage bandwidth is entirely independent of the compute fabric bandwidth. A checkpoint write consuming the full 800 Gbps of storage bandwidth has zero impact on the 8 compute ConnectX-7 NICs transmitting AllReduce traffic simultaneously.

The host CPU and NVMe-oF command processing

On the DGX H100/H200, the NVMe-oF initiator stack runs on the host CPU (2x Intel Xeon), using the storage ConnectX-7 ports as its network interface. The host CPU handles NVMe command queuing and RDMA verb posting for the storage transfers. With GDS, the actual data movement (GPU HBM to storage) is handled by the ConnectX-7 DMA engine and does not pass through CPU memory -- but the CPU does process the NVMe command and completion queues. During a checkpoint write, the host CPU spends a small fraction of its capacity managing storage I/O commands, while the GPUs and compute NICs continue unaffected.

Note: the DGX B200 uses a different model. Its BlueField-3 HCAs (in NIC mode) run the NVMe-oF initiator on their embedded ARM cores, offloading that work entirely from the host CPU. This is an architectural difference between the B200 and the H100/H200 -- not a difference in the data path (which is CX7 DMA -> PCIe -> GPU HBM in both cases).

The partition is clean: compute ConnectX-7 NICs handle AllReduce RDMA, storage ConnectX-7 cards carry NVMe-oF traffic, and the two paths never share switch ports or fabric resources.