Now you have all the tools. Here is how they work together in a real investigation.
InfiniBand diagnostic workflow — click each phase
Before touching any CLI, check UFM's event log for the past hour.
Why: UFM gives you the timeline. You need to know what happened, and when, before you start digging. A link flap that happened 2 hours ago is a different problem to one happening right now.
The investigation sequence -- IB fabric
Step 1: Check the alert context (UFM first)
Before touching any CLI, check UFM's event log. What happened, and when? Was there a port flap? An SM restart? A routing change? The event log tells you where to look.
ufm_cluster events list --hours 1 --severity WARNING,ERROR
Step 2: Locate the affected component (UFM topology)
Use UFM topology view to find the physical location of the reported component. Which rack, which switch, which port, which cable, which DGX node.
Step 3: Check port state on the affected switch (ONYX)
SSH to the identified switch. Check the specific port:
switch # show interface ib 1/7
switch # show ib counters
Are SymbolErrors growing? Is the link stable or flapping?
Step 4: Check the DGX side
SSH to the DGX node on the other end of the problematic cable:
ibstat # Is mlx5_N in Active state?
rdma link show # Does this rail show state ACTIVE?
Step 5: Cross-reference SM view
switch # show ib sm
Is the SM aware of this node? Does it show the port as active in its discovered topology?
Step 6: Full sweep if the above does not explain it
ibdiagnet --ls --pc --sm
Let ibdiagnet sweep the entire fabric. Check the errors section. Look for unexpected speed negotiation, error counter violations, or routing inconsistencies.
Step 7: Corrective action
- Physical problem (symbol errors, speed degradation): replace cable, clean connectors, replace transceiver
- SM problem (routing inconsistency, VL15 drops): restart UFM during a maintenance window, let it do a clean sweep
- Credit flow problem (XmtDiscards): check for routing loops, engage NVIDIA support if persistent