ProDeploy — Build the fabric that keeps AI clusters running

You have now traced both fabrics to packet level. Here is where they differ:

Fabric Comparison
Compute Fabric vs Storage Fabric
click any row for detail
Dimension
Compute Fabric
Storage Fabric
NIC hardware
8x single-port CX7 HCA per DGX
2x dual-port CX7 NIC per DGX (Slot1 + Slot2)
Switch hardware
QM9700 (InfiniBand HDR) or SN5600 (RoCEv2)
SN4600C 100GbE or similar
Topology
Rail-optimised fat-tree, full bisection (1:1)
Simple leaf, 2:1 oversubscription acceptable
Transport protocol
RDMA (InfiniBand or RoCEv2)
NVMe-oF RDMA (preferred) or NVMe/TCP
PFC
Required -- lossless for AllReduce
NOT used in most storage fabric designs
Congestion control
DCQCN with tight thresholds (Kmin ~80KB, Kmax ~400KB)
ECN only, larger thresholds (Kmin ~200KB, Kmax ~1MB)
Traffic pattern
All-to-all, synchronised (AllReduce barrier)
Burst writes (checkpoint), sequential reads (dataset load)
Payload
NCCL messages (RDMA Write chunks, 256KB-4MB typical)
NVMe command capsules (64B SQE) + RDMA Read data (target-pull, 4MB block)
GDS capable
Not applicable
Yes (RDMA transport only, requires nvidia-fs module)
Key diagnostic
ibstat, perfquery, DCGM, UFM
nvme error-log, rdma stat, fio, gds_check
The core confusion to avoid
Both fabrics use ConnectX-7 NICs and RoCEv2 framing. This does NOT mean they share a configuration. The PFC requirement, ECN thresholds, and traffic patterns are fundamentally different. A storage fabric misconfigured as a compute fabric (with PFC enabled) will pause-storm on checkpoint bursts. A compute fabric misconfigured as a storage fabric (PFC disabled) will drop packets and destroy AllReduce performance.

Dimension	Compute Fabric	Storage Fabric
Hardware	8x single-port CX7 HCA (per DGX)	2x dual-port CX7 NIC (per DGX)
Switch	QM9700 (IB) or SN5600 (RoCEv2)	SN4600C or similar 100GbE
Topology	Rail-optimised fat-tree, 1:1 oversubscription	Simple leaf, 2:1 oversubscription
Transport	RDMA (IB or RoCEv2)	NVMe-oF RDMA or NVMe/TCP
PFC	Required (lossless for AllReduce)	NOT used in most designs
Congestion control	DCQCN with tight thresholds	ECN only with larger thresholds
Traffic pattern	All-to-all, synchronised (AllReduce)	Burst writes (checkpoint), sequential reads
Payload protocol	NCCL messages / raw RDMA writes	NVMe command capsules over RDMA
Key diagnostic	`ibstat`, `perfquery`, DCGM	`nvme error-log`, `rdma stat`, `fio`
GDS possible?	N/A	Yes (RDMA transport only)

DimensionHardware

Compute Fabric8x single-port CX7 HCA (per DGX)

Storage Fabric2x dual-port CX7 NIC (per DGX)

DimensionSwitch

Compute FabricQM9700 (IB) or SN5600 (RoCEv2)

Storage FabricSN4600C or similar 100GbE

DimensionTopology

Compute FabricRail-optimised fat-tree, 1:1 oversubscription

Storage FabricSimple leaf, 2:1 oversubscription

DimensionTransport

Compute FabricRDMA (IB or RoCEv2)

Storage FabricNVMe-oF RDMA or NVMe/TCP

DimensionPFC

Compute FabricRequired (lossless for AllReduce)

Storage FabricNOT used in most designs

DimensionCongestion control

Compute FabricDCQCN with tight thresholds

Storage FabricECN only with larger thresholds

DimensionTraffic pattern

Compute FabricAll-to-all, synchronised (AllReduce)

Storage FabricBurst writes (checkpoint), sequential reads

DimensionPayload protocol

Compute FabricNCCL messages / raw RDMA writes

Storage FabricNVMe command capsules over RDMA

DimensionKey diagnostic

Compute Fabricibstat, perfquery, DCGM

Storage Fabricnvme error-log, rdma stat, fio

DimensionGDS possible?

Compute FabricN/A

Storage FabricYes (RDMA transport only)

The most dangerous assumption is that because both fabrics use ConnectX-7 NICs and RoCEv2, they should be configured identically. They should not. The PFC requirement, ECN thresholds, and traffic patterns are fundamentally different. A storage fabric misconfigured as a compute fabric will PFC-stall on checkpoint bursts. A compute fabric misconfigured as a storage fabric will drop packets and destroy AllReduce performance.