Every design decision in the compute fabric -- InfiniBand's credit-based flow control, SHARP in-network compute, per-packet adaptive routing, non-blocking fat-tree topology -- exists because of the specific demands of synchronised all-to-all AllReduce traffic at microsecond timescales.

None of those demands apply to the storage fabric. This is the one part of AI cluster networking where standard enterprise Ethernet design principles are completely correct.

Traffic pattern analysis

Storage traffic is large sequential transfers: one writer (a DGX node's storage CX7) streaming bytes to one or more targets (storage appliance nodes), or one reader streaming bytes from one or more sources. The flow has a fixed source and destination for its entire duration -- seconds, not microseconds.

ECMP works correctly for this pattern. Once a flow is hashed to a path, it stays on that path for the duration of the transfer. A 140 GB checkpoint that takes 1.4 seconds generates a single sustained flow from the storage CX7 to storage appliance. Standard Ethernet ECMP distributes these flows across available paths without hot-spotting as long as the number of active flows is large relative to the number of paths.

Why oversubscription is acceptable

The compute fabric targets 1:1 non-blocking topology because any congestion event during AllReduce stalls every GPU at the barrier. The storage fabric can tolerate oversubscription -- typically 2:1 to 4:1 -- because:

Storage traffic is not a synchronisation barrier. A checkpoint write that takes 2 seconds instead of 1.4 seconds due to storage fabric congestion delays the next training step slightly. It does not stall 32,000 GPUs at a barrier. The downstream impact is localised.

Storage appliances have local SSD buffers that absorb write bursts. If the storage fabric is briefly congested during simultaneous checkpoint writes, the storage appliances can buffer incoming traffic in their NVMe write caches and drain to persistent storage after the burst passes.

TCP retransmission handles loss without stalling training. If a packet is dropped on the storage fabric, TCP retransmits it. The storage write takes slightly longer. Training continues. Contrast with compute RDMA where a single dropped packet sends a QP into error state and stalls the AllReduce.

Switch selection for storage fabric

Spectrum Ethernet switches (the same family used for RoCEv2 compute fabrics) are the standard choice for storage fabric Ethernet in NVIDIA DGX deployments. The reason is operational simplicity: the same switch operating system (Cumulus Linux or SONiC), the same configuration tooling, the same monitoring approach.

The storage fabric does not need QM9700 InfiniBand switches. It does not need PFC configuration at the same precision as the compute fabric (though basic ECN configuration is still worth enabling to prevent TCP incast storms from the parallel file system). It does not need adaptive per-packet routing.

A correctly configured fat-tree Ethernet fabric with 2:1 oversubscription and basic ECN is sufficient for all storage traffic patterns in a DGX deployment.

Sizing the storage fabric

Each DGX node contributes 2 x 400G = 800 Gbps of storage bandwidth from its dual-port storage CX7 cards. For a 32-node BasePOD, the storage fabric must support 32 x 800 Gbps = 25.6 Tbps of aggregate storage bandwidth in the worst case (all nodes checkpointing simultaneously).

Storage fabric switches are typically sized at 400G or 800G per port, with a 32-node BasePOD requiring a leaf layer with at least 32 x 2 = 64 downlinks to the storage CX7 ports, plus uplinks to spine switches or directly to the storage appliance tier. At 2:1 oversubscription, 32 uplinks of 400G provide 12.8 Tbps of aggregate storage fabric bandwidth -- sufficient for typical storage traffic patterns where simultaneous checkpoint writes are the worst-case event.