Skip to content

Chapter 3: The CLI -- Reading the Fabric · Part 8 of 10

Act 6 -- Command reference by scenario

Different symptoms lead you to different commands. Here is the map.

Symptom → command sequence — click each scenario

Scenario: Training throughput dropped to 87.5%

1. ibstat -> look for any NIC not in State: Active
2. show topology -> confirm which rail is affected
3. show switch port rail<N> -> check switch-side state
-> Most likely: link-flap errdisable. Fix physical problem, re-enable port.

Scenario: Training throughput degraded across all GPUs (not one)

1. show interface counters -> check for drops and pauses
2. ethtool -S eth0 -> check rx_pfc_pause_frames growth rate
3. show dcb pfc -> confirm PFC is enabled and on correct priority
-> Most likely: pause storm or PFC misconfiguration.

Scenario: Training throughput degraded, no drops in counters

1. ethtool -S eth0 -> check rx_pfc_pause_frames (may be growing fast)
2. show dcb pfc -> check watchdog status
-> Most likely: pause storm with no drops (PFC is working, preventing drops,
  but holding the NIC in pause state -- effective as a deadlock)

Scenario: AllReduce latency high but no drops or pauses

1. show dcb ets -> check ECN status
2. show roce -> check DCQCN status
3. show interface counters -> check buffer utilisation %
-> Most likely: ECN not configured. Congestion builds silently until
  PFC kicks in or drops occur. Enable ECN for proactive rate control.

Scenario: Node not appearing in UFM after boot

1. ibstat on that node -> check State and Base lid
2. systemctl status nvidia-fabricmanager -> FM daemon running?
3. show ib sm on switch -> is SM running and is it seeing this node?
-> Most likely: FM daemon not started, or UFM not running when node booted.