Skip to content

Chapter 3: The CLI -- Reading the Fabric · Part 8 of 10

Act 6 -- Command reference by scenario

Building the model…

Where the commands in this section run, marked on the cluster. Switch the overlay to see the rest of the fabric.

Different symptoms lead you to different commands. Here is the map.

Symptom → command sequence — click each scenario

Scenario: Training throughput dropped to 87.5%

1. ibstat -> look for any NIC not in State: Active
2. show topology -> confirm which rail is affected
3. show switch port rail<N> -> check switch-side state
-> Most likely: link-flap errdisable. Fix physical problem, re-enable port.

Scenario: Training throughput degraded across all GPUs (not one)

1. show interface counters -> check for drops and pauses
2. ethtool -S eth0 -> check rx_pfc_pause_frames growth rate
3. show dcb pfc -> confirm PFC is enabled and on correct priority
-> Most likely: pause storm or PFC misconfiguration.

Scenario: Training throughput degraded, no drops in counters

1. ethtool -S eth0 -> check rx_pfc_pause_frames (may be growing fast)
2. show dcb pfc -> check watchdog status
-> Most likely: pause storm with no drops (PFC is working, preventing drops,
  but holding the NIC in pause state -- effective as a deadlock)

Scenario: AllReduce latency high but no drops or pauses

1. show dcb ets -> check ECN status
2. show roce -> check DCQCN status
3. show interface counters -> check buffer utilisation %
-> Most likely: ECN not configured. Congestion builds silently until
  PFC kicks in or drops occur. Enable ECN for proactive rate control.

Scenario: Node not appearing in UFM after boot

1. ibstat on that node -> check State and Base lid
2. systemctl status nvidia-fabricmanager -> FM daemon running?
3. show ib sm on switch -> is SM running and is it seeing this node?
-> Most likely: FM daemon not started, or UFM not running when node booted.

Read aloudPage 8