Different symptoms lead you to different commands. Here is the map.
Symptom → command sequence — click each scenario
Scenario: Training throughput dropped to 87.5%
1. ibstat -> look for any NIC not in State: Active
2. show topology -> confirm which rail is affected
3. show switch port rail<N> -> check switch-side state
-> Most likely: link-flap errdisable. Fix physical problem, re-enable port.
Scenario: Training throughput degraded across all GPUs (not one)
1. show interface counters -> check for drops and pauses
2. ethtool -S eth0 -> check rx_pfc_pause_frames growth rate
3. show dcb pfc -> confirm PFC is enabled and on correct priority
-> Most likely: pause storm or PFC misconfiguration.
Scenario: Training throughput degraded, no drops in counters
1. ethtool -S eth0 -> check rx_pfc_pause_frames (may be growing fast)
2. show dcb pfc -> check watchdog status
-> Most likely: pause storm with no drops (PFC is working, preventing drops,
but holding the NIC in pause state -- effective as a deadlock)
Scenario: AllReduce latency high but no drops or pauses
1. show dcb ets -> check ECN status
2. show roce -> check DCQCN status
3. show interface counters -> check buffer utilisation %
-> Most likely: ECN not configured. Congestion builds silently until
PFC kicks in or drops occur. Enable ECN for proactive rate control.
Scenario: Node not appearing in UFM after boot
1. ibstat on that node -> check State and Base lid
2. systemctl status nvidia-fabricmanager -> FM daemon running?
3. show ib sm on switch -> is SM running and is it seeing this node?
-> Most likely: FM daemon not started, or UFM not running when node booted.