You have now traced both fabrics to packet level. Here is where they differ:
Fabric Comparison
Compute Fabric vs Storage Fabric
click any row for detail
Dimension
Compute Fabric
Storage Fabric
The core confusion to avoid
Both fabrics use ConnectX-7 NICs and RoCEv2 framing. This does NOT mean they share a configuration. The PFC requirement, ECN thresholds, and traffic patterns are fundamentally different. A storage fabric misconfigured as a compute fabric (with PFC enabled) will pause-storm on checkpoint bursts. A compute fabric misconfigured as a storage fabric (PFC disabled) will drop packets and destroy AllReduce performance.
| Dimension | Compute Fabric | Storage Fabric |
|---|---|---|
| Hardware | 8x single-port CX7 HCA (per DGX) | 2x dual-port CX7 NIC (per DGX) |
| Switch | QM9700 (IB) or SN5600 (RoCEv2) | SN4600C or similar 100GbE |
| Topology | Rail-optimised fat-tree, 1:1 oversubscription | Simple leaf, 2:1 oversubscription |
| Transport | RDMA (IB or RoCEv2) | NVMe-oF RDMA or NVMe/TCP |
| PFC | Required (lossless for AllReduce) | NOT used in most designs |
| Congestion control | DCQCN with tight thresholds | ECN only with larger thresholds |
| Traffic pattern | All-to-all, synchronised (AllReduce) | Burst writes (checkpoint), sequential reads |
| Payload protocol | NCCL messages / raw RDMA writes | NVMe command capsules over RDMA |
| Key diagnostic | ibstat, perfquery, DCGM | nvme error-log, rdma stat, fio |
| GDS possible? | N/A | Yes (RDMA transport only) |
The most dangerous assumption is that because both fabrics use ConnectX-7 NICs and RoCEv2, they should be configured identically. They should not. The PFC requirement, ECN thresholds, and traffic patterns are fundamentally different. A storage fabric misconfigured as a compute fabric will PFC-stall on checkpoint bursts. A compute fabric misconfigured as a storage fabric will drop packets and destroy AllReduce performance.