The storage leaf switch in a BasePOD is typically a Mellanox Spectrum-1 or Spectrum-2 based switch in the SN4600C family -- 64 ports at 100GbE. It is not the same switch as the compute fabric spine (QM9700 for IB, SN5600 for RoCEv2). The design is deliberately simpler because the storage traffic pattern does not require full bisection bandwidth.

Oversubscription in the storage fabric

A compute fat-tree is designed for 1:1 oversubscription -- every GPU-to-GPU flow gets full bandwidth at every hop. The storage fabric runs at 2:1 oversubscription by design. A DGX node with two dual-port CX7 storage NICs has 4 x 100GbE = 400 Gbps of storage NIC bandwidth. The storage switch uplink to the appliance layer may aggregate to 200 Gbps per DGX node. This is acceptable because checkpoint writes are bursty -- they do not sustain 400 Gbps continuously. The average utilisation across a training run is far lower.

PFC in the storage fabric

This is the most important design difference from the compute fabric: most storage fabric designs do NOT run PFC. The reasons:

1. NVMe-oF reconnection is tolerant of occasional packet loss. The initiator will retry.
2. PFC storm risk on a storage fabric can cascade to the storage appliances, which are more sensitive to congestion than GPU NICs.
3. NVMe/TCP bindings (which some deployments use) explicitly do not use RDMA and therefore cannot use PFC at all.
4. The storage traffic is not latency-bound the same way GPU-to-GPU AllReduce is.

When the storage fabric uses NVMe-oF RDMA (as in DGX with ConnectX-7), you can optionally run RoCEv2 with ECN-only congestion control: DCQCN marks packets at the switch when the buffer crosses a threshold, and the NIC reduces its injection rate. But no PFC. The backpressure signal is DSCP-marked ECN, not pause frames.

ECN thresholds on the storage switch

Because storage traffic is bursty (large sequential writes) rather than fine-grained (many small GPU RDMA messages), the ECN marking thresholds are set higher than on the compute fabric:

Compute fabric ECN thresholds (from Ch5): Kmin ~80KB, Kmax ~400KB. Storage fabric ECN thresholds (typical): Kmin ~200KB, Kmax ~1MB.

The higher thresholds allow the switch buffer to absorb burst traffic from multiple DGX nodes flushing checkpoints simultaneously without triggering DCQCN rate reduction prematurely. The exact values depend on your storage appliance latency targets and checkpoint write size. Always verify: show interface ethernet swp1 queues and show buffers on the storage switch.