Chapter 0 Act 9 states that storage traffic is kept separate from compute traffic "so storage traffic cannot interfere with training communication." That sentence is correct. This act explains the mechanism behind it.
Why storage and compute fabrics must be physically separated
Toggle to see the failure cascade on a mixed fabric vs clean isolation on a separated fabric
During an AllReduce on a non-blocking compute fabric, all 256 GPUs in a BasePOD transmit simultaneously at full line rate. Each of the 8 leaf switches is carrying approximately 16 x 400 Gbps = 6.4 Tbps of inbound AllReduce traffic, simultaneously forwarding it to 16 other GPU nodes. The compute fabric is running at or near saturation for the duration of every AllReduce barrier.
What happens if storage traffic enters this fabric at the wrong moment:
AllReduce traffic is bursty -- it spikes during synchronisation, then drops to near-zero during the forward and backward passes. Storage traffic is sustained -- a checkpoint write is a continuous 140 GB stream that takes the full 1.4 seconds from first byte to last. If these two traffic types share a switch, the storage write arrives at the switch's egress buffer while AllReduce traffic is also arriving.
The switch buffer fills. One of three things happens, all of them bad:
The switch drops packets. For storage TCP traffic, TCP detects the loss and retransmits -- the write takes longer. For compute RDMA traffic, a dropped packet sends a QP into error state and stalls the AllReduce.
The switch sends PFC PAUSE frames. The NIC upstream from the congested port pauses all traffic on that priority class -- including the AllReduce traffic that has nothing to do with the storage write. Every GPU on that rail stalls, waiting for a pause that was caused by a storage read happening in another job entirely.
The switch applies DCQCN rate limiting. NCCL sees degraded busbw on the affected rails. The ML engineer sends a Slack message asking why training throughput dropped by 30%. You spend three hours running nccl-tests before realising the compute fabric is fine -- there was a large storage checkpoint running on a shared switch during the benchmark.
Physical separation prevents all three scenarios. The compute fabric switches carry only compute traffic. The storage fabric switches carry only storage traffic. The two fabrics can only interact at the DGX node itself, where the storage CX7 ports and compute CX7 NICs are on separate PCIe devices with separate queues, separate physical ports, and separate network paths.
The traffic pattern mismatch is the root cause:
Compute traffic: bursty, synchronised, all-to-all, microseconds to complete per AllReduce wave. Requires lossless fabric, PFC, ECN, and non-blocking topology.
Storage traffic: sustained, sequential, point-to-multi-point (many compute nodes reading from a shared pool), seconds to complete per checkpoint. Tolerates packet loss (retransmit), tolerates modest latency (a millisecond here does not compound), and works on oversubscribed Ethernet topology.
Putting these two on the same fabric requires the fabric to be simultaneously optimised for both. That is not achievable. The correct engineering decision -- confirmed by every major AI infrastructure deployment -- is to separate them physically.