ProDeploy — Learn the fabric that keeps AI clusters alive

ibdiagnet is the most powerful InfiniBand diagnostic tool. It is not a switch CLI command -- it runs on any host with access to the fabric (typically a management host or a DGX node) and performs a complete fabric audit.

What ibdiagnet does

# Run from a DGX host or management server with IB connectivity
$ ibdiagnet --extended_speeds all --pc --ls --vlq

ibdiagnet performs these checks in sequence:

Fabric discovery -- Queries every switch and NIC to build the complete topology
LID consistency -- Verifies every port has a unique LID and no duplicates exist
Path record validation -- Checks that all path records are populated and correct
Routing table consistency -- Verifies switch forwarding tables match the SM's routing
Port state check -- Reports any port not in Active state
Error counter sweep -- Collects error counters from every port in the fabric
Performance counters -- Measures bandwidth utilisation on every link
VL credit check -- Verifies Virtual Lane credit configuration

ibdiagnet output — explore each section

-I- ------------------- Errors -------------------
-E- High Symbol Error Rate:
    Switch:    QM9700-2 port 7
    Connected: DGX-node-12 mlx5_7 (Rail 7)
    SymbolErrors:   14,827
    Threshold:      100
    Delta (1hr):    8,293  ← ACTIVELY GROWING
    
    RECOMMENDATION: Replace transceiver or cable.
    This port will likely progress to link flapping.
    
-W- Non-zero XmtDiscards:
    Switch:    QM9700-5 port 12
    Connected: DGX-node-03 mlx5_4 (Rail 4)
    XmtDiscards: 47
    
    RECOMMENDATION: Check routing on paths through
    this port. Possible routing loop.

What this means: Two problems identified. First: QM9700-2/port7 is actively accumulating symbol errors at 8,293/hour — this port is failing and will progress to link flapping within hours or days. Second: QM9700-5/port12 has non-zero transmit discards, which in InfiniBand's credit-based system should never occur — this indicates a routing or credit flow problem.

Action: Immediate: Schedule maintenance to replace transceiver on QM9700-2/port7. Run show interface ib 1/7 on that switch to confirm. For XmtDiscards: run ibdiagnet with --vlq flag to check VL credit configuration, and review UFM event log for SM routing events.

Running ibdiagnet and reading its output

$ ibdiagnet --ls --pc --sm

-I- Loading IBDIAGNET plugins ...
-I- Discovering the subnet ...
-I- Loaded fabric: 32 switches, 256 nodes, 512 IB ports
-I- ----------------- SM Configuraion ------------------
-I- Master SM: 0x506b4b0300000001 (UFM) - MASTER
-I- Standby SM: 0x506b4b0300000003 (OpenSM) - STANDBY
-I- SUBNET MANAGER: OK

-I- ------------ Fabric Discovery Summary ---------------
-I- Nodes: 256 (32 DGX nodes * 8 NICs each = 256 HCAs)
-I- Switches: 32 (8 leaf + 4 spine * ... )
-I- DISCOVERY: OK

-I- --------------- Link Speed Check --------------------
-W- Link Speed Degraded: Switch QM9700-1 port 3 <-> DGX-node-07 mlx5_2
    Expected: NDR (4x)    Actual: HDR (4x)
    This link is running at HDR speed instead of NDR.
    Check cable compatibility: requires NDR-certified cable.

-I- ------------------- Errors -------------------------
-E- High Symbol Error Rate: Switch QM9700-2 port 7
    SymbolErrors: 14,827 (threshold: 100)
    Last reset: 72 hours ago
    Delta in last hour: 8,293
    RECOMMENDATION: Replace cable or transceiver

-W- Non-zero XmtDiscards: Switch QM9700-5 port 12
    XmtDiscards: 47
    RECOMMENDATION: Check for routing loop on this path

-I- -------------- Performance Summary ------------------
-I- Average link utilisation: 34.2%
-I- Peak link utilisation: 87.3% (Switch QM9700-3 port 1)
-I- Bottleneck port: QM9700-3/1 (87.3% utilisation)
-I- Bottleneck connected to: DGX-node-14 Rail-0

-I- -------------- Summary ------------------------------
-I- PASS: Discovery, SM, LID uniqueness, routing consistency
-I- WARN: 1 link speed degraded, 1 non-zero XmtDiscards
-I- ERROR: 1 high symbol error rate port

-I- Total run time: 4 minutes 37 seconds

Reading the ibdiagnet output:

Link Speed Degraded: QM9700-1 port 3 <-> DGX-node-07 mlx5_2 -- This link is running at HDR instead of NDR. The link negotiated down because one side cannot run NDR. Most likely cause: a non-NDR-rated cable. NDR requires 50 Gb/s per lane electrical specification -- older HDR cables (25 Gb/s per lane) will cause negotiation to HDR. This halves the bandwidth for that GPU rail.

High Symbol Error Rate: Switch QM9700-2 port 7 -- 14,827 symbol errors with 8,293 accumulating in the last hour. This port is actively degrading. The physical layer is failing. Replace the transceiver or cable on this link before it causes link cycling and training failures.

Non-zero XmtDiscards: QM9700-5 port 12 -- 47 transmit discards. Small number, but in InfiniBand this should be zero. Check for a routing anomaly on paths that traverse this port.

Bottleneck port: QM9700-3/1 (87.3%) -- One port is running at 87.3% utilisation while average is 34.2%. This indicates traffic is not distributed evenly across the fabric -- some paths are heavily loaded while others are underutilised. In a fat-tree with FTREE routing this should not happen. Check if the SM has a complete topology view and is routing correctly. This can also indicate a topology fault -- if several links have failed, the SM routes around them and concentrates traffic on surviving paths.

When to run ibdiagnet

At cluster bring-up: before first training job, run ibdiagnet to establish a clean baseline
After any physical work: after adding nodes, replacing cables, or upgrading firmware
When performance degrades without obvious cause: ibdiagnet often finds problems that individual switch CLIs miss
Weekly in production: automated scheduled runs catch slow-developing problems (gradual transceiver degradation)