ProDeploy — Learn the fabric that keeps AI clusters alive

A storage fabric fault looks different from a compute fabric fault. There is no UFM covering it. There are no IB port state counters. The signals are NVMe error logs, RDMA transport errors on the storage QPs, and TCP retransmit counters if you are running NVMe/TCP.

Storage Fabric Diagnostics
Commands by Layer
$ nvme list-subsys
Lists all NVMe-oF subsystems and their transport type (rdma or tcp)
>
$ nvme list
Lists all NVMe namespaces with their device path (/dev/nvme0n1 etc.)
>
$ nvme error-log /dev/nvme0
Shows the error log for the NVMe controller -- command errors and status codes
>
$ nvme smart-log /dev/nvme0n1
SMART health log -- distinguishes drive-side failures from fabric failures
>
$ lsmod | grep nvme_rdma
Confirms the NVMe-oF RDMA initiator kernel module is loaded
>
Storage fault triage order
1
nvme list-subsys
RDMA or TCP?
->
2
nvme error-log
any failed I/O?
->
3
rdma stat show
QP in error?
->
4
switch PFC check
pause frames?
->
5
ethtool -S NIC
physical errors?
->
6
fio latency
E2E baseline

NVMe-oF initiator status

# List all NVMe subsystems and their transport
nvme list-subsys

# List all namespaces with their device path and controller
nvme list

# Check error log for the first namespace device
nvme error-log /dev/nvme0

# Show smart/health log (catches drive-side errors vs fabric errors)
nvme smart-log /dev/nvme0n1

The error log returns a table of error entries, each with an error count, status field (lower 8 bits are the NVMe status code), and a command ID. Status code 0x00 = success. Non-zero means the command was aborted, the media failed, or a transport error occurred.

RDMA transport diagnostics on the storage NIC

# Show RDMA QP statistics for the storage NIC (enp170s0f0 = Slot1 port0)
rdma stat show dev mlx5_8

# Show RDMA link state
rdma link show

# Detailed QP and port error counters on RoCEv2 / Ethernet CX7
rdma stat show mlx5_8/1
# or via ethtool for per-priority and pause counters:
ethtool -S enp170s0f0 | grep -E "tx_|rx_|err|drop|pause|pfc"

Look for out_of_buffer, out_of_sequence, local_length_error, retry_exceeded in the QP error counters. retry_exceeded means the CX7 sent a packet, got no acknowledgement, retried the maximum number of times, and gave up. The QP is now in ERROR state. The initiator will attempt reconnection, but in-flight I/O is lost.

Storage switch counters

On the SN4600C storage leaf switch running Cumulus Linux / NVUE:

# Interface counters for the DGX-facing port
nv show interface swp1 counters

# Error counters specifically
nv show interface swp1 counters errors

# Buffer occupancy -- check for persistent congestion
nv show qos interface swp1 buffer

# PFC pause state on the storage-fabric switch port
nv show qos interface swp1 pfc

# ECN marking statistics (separate command)
nv show qos interface swp1 ecn

If tx_pause is non-zero on the storage switch, that is a problem -- the design should not have PFC enabled. Non-zero tx_pause indicates a misconfigured switch profile that has PFC enabled on the storage traffic class. Disable it: PFC on a storage fabric will cause the appliance to receive pause frames and back up, stalling all storage I/O to that switch segment.

End-to-end latency check

# NVMe-oF round-trip latency with fio (1 I/O depth, 4K block, read)
fio --name=latency_check --filename=/dev/nvme0n1 --rw=randread \
    --bs=4k --iodepth=1 --numjobs=1 --time_based --runtime=30 \
    --lat_percentiles=1 --output-format=normal

# GDS-specific latency (requires cuFile)
gds_perf --read --sequential --file /path/to/testfile --size 1G

Target NVMe-oF RDMA latency for 4K random read: under 100 microseconds. If you are seeing 1-2 ms, the connection is likely falling back to TCP (check nvme list-subsys), the RNIC is in error state, or there is a routing failure forcing traffic through a non-optimal path.