You now have all the commands. The next step is knowing which order to run them in. The workflow is not random -- it follows the OSI model from physical layer upward.

Phase 1 -- Physical layer (Links)

ibstat                     -> Is every NIC State: Active?
rdma link show             -> Are all Rails showing state ACTIVE?
show topology              -> Does NIC state match switch port state?

If any link is down at this layer, everything else is irrelevant until you fix the physical problem. Never skip this phase.

Phase 2 -- Traffic layer (Counters)

show interface counters    -> Are there drops? Are there pauses? Buffer utilisation?
ethtool -S eth0            -> rx_pfc_pause_frames growing? tx_dropped non-zero?

Read counters before reading configuration. Counters tell you the symptom. Configuration tells you the cause. You need the symptom first to know what cause you are looking for.

Phase 3 -- Configuration layer

show dcb pfc               -> Is PFC enabled on the right priority?
show dcb ets               -> Is ECN active? Is DCQCN running?
show roce                  -> Does RoCEv2 show a clean summary?

Only read configuration after you know what symptom you are diagnosing. If counters show drops with no pauses, you know PFC is the issue -- go directly to show dcb pfc. If counters show rapidly growing pauses but no drops, PFC is working but congestion is severe -- look at show dcb ets to check ECN.

Phase 4 -- Switch-side investigation

show topology              -> Which rail has the anomaly?
show switch port rail<N>   -> What does the switch say about that port?

Only go to the switch side if the DGX-side commands do not fully explain the problem.

Phase 5 -- Full fabric diagnostics

If the above phases do not identify the problem, escalate to:

ibdiagnet                  -> Full fabric sweep: checks all IB ports, routing tables,
                             LID assignments, path records, and error counters.
                             Run from any node with fabric access.
                             Takes 5-15 minutes for a large fabric.
                             Output is a comprehensive diagnostic report.

ibdiagnet is the nuclear option -- it tells you everything about the fabric but takes time and generates enormous output. Use it when you have ruled out simpler causes.

When to call NVIDIA support:

• ibdiagnet reports routing table inconsistencies
• Multiple rails on multiple nodes are showing the same error simultaneously
• UFM shows SM restarts in the event log
• The problem reappears immediately after you fix it