Storage fabric failures present differently from compute fabric failures. The compute fabric has ibstat, rdma link show, show interface counters, ethtool -S, and NCCL logs all pointing at the same set of NICs and switches. Storage fabric failures are diagnosed through a completely different set of tools pointing at a completely different set of hardware.

The symptom pattern

A storage fabric degradation during training produces a distinctive failure signature that differs from compute fabric problems:

NCCL logs are clean. AllReduce busbw is at expected levels. No QP timeouts. No timeout errors. The compute fabric appears healthy.

Training throughput is degraded. Loss curves are stalling. The ML engineer reports the training run is taking longer than expected.

Checkpoint writes are timing out or taking significantly longer than the expected 1.4 seconds. The framework logs show checkpoint save times of 10-20 seconds instead of 1-2 seconds.

CPU and GPU utilisation on the DGX nodes shows alternating idle periods aligned with checkpoint timing. The GPUs are computing normally, then stalling for extended periods during checkpoint writes.

The diagnostic path

Start at the storage NIC ports, not the compute ConnectX-7s.

# On the DGX node -- check storage CX7 interface state
# Slot1/Slot2 dual-port CX7 appear as mlx5_7/mlx5_8 and mlx5_1/mlx5_2 (RDMA)
# or as enp170s0f0/f1 and enp41s0f0/f1 (Ethernet) -- check ip link show
ip link show
# Look for storage interfaces -- typically Slot1/Slot2 bonded as a single interface
# for in-band management and storage (check your deployment's bonding config)

# Check storage interface counters
ethtool -S enp170s0f1  # Slot1 port2 -- bonded storage/mgmt port
# Key counters: rx_dropped, tx_dropped, rx_errors
# Non-zero drops indicate storage fabric congestion

On the storage fabric Ethernet switch (separate switch, not the compute leaf switches):

# Spectrum switch -- Cumulus Linux
show interface counters
# Look for: output drops on uplink ports to storage appliances
# A growing output_drops counter on the storage-facing ports indicates
# the storage appliance cannot absorb writes at the rate the storage NICs are sending

# Check for ECN marking (if configured)
show interface counters detailed
# rx_ecn_marked growing -> switch is signalling congestion on storage fabric

On the storage appliance or parallel file system:

For BeeGFS: beegfs-check-servers and beegfs-net show storage server connectivity and throughput per storage target. A storage target with significantly lower throughput than its peers indicates a degraded drive, a saturated uplink, or a failing storage node.

For WEKA: the WEKA management console provides per-node throughput histograms. A storage node at 100% utilisation while others are at 40% indicates uneven striping -- often caused by a hot dataset segment landing disproportionately on one storage node.

The most common storage fabric failures

A storage appliance NVMe drive degrading: one storage node's throughput drops while others remain healthy. BeeGFS redistributes data from failed targets but only for new writes -- existing data on the degraded node is slow to read. WEKA handles this more transparently through its distributed block protocol.

A storage CX7 link failure: the storage interface presents as down to the storage fabric switch. Checkpoint writes from that node fail. The training framework typically catches this as a checkpoint timeout and retries. Check ip link show on the DGX host for the Slot1/Slot2 interface state, and check the storage switch port state. Note that ibstat shows compute CX7 state (mlx5_0-7), not the storage CX7 ports.

Storage fabric switch buffer overflow: all nodes checkpointing simultaneously saturates the switch uplinks to the storage appliance tier. Output drops appear on the storage switch. TCP retransmission handles recovery, but checkpoint times extend. The fix is either stagger checkpoint timing across nodes or increase storage switch uplink capacity.