This is the single most disorienting thing for network engineers coming from enterprise backgrounds. In a Cisco environment, you SSH into a switch and run commands on that switch. The switch is the authoritative source of information.

In an AI cluster, the information is distributed across multiple devices running different operating systems. Before you can run a command, you need to know which device to run it on.

Here is the complete map:

On the DGX host (DGX OS -- Ubuntu) These commands run on the compute node itself, in a standard Linux shell after SSHing to the DGX management IP:

ibstat                     # Overview of all IB/RDMA NIC ports
rdma link show             # Per-device RDMA link state
ibdiagnet                  # Full fabric diagnostic sweep (run from one node)
ethtool -S eth0            # NIC hardware statistics including pause counters
dcb pfc show dev eth0      # PFC state (using the 'dcb' iproute2 tool)
mlnx_qos -i eth0           # QoS configuration and counters

On the Spectrum-X switch (Cumulus Linux) These commands run after SSHing to the switch management IP:

show dcb pfc               # PFC state per interface
show dcb ets               # ETS bandwidth allocation and ECN config
show interface counters    # Per-port packet and byte counters
show roce                  # RoCEv2 configuration summary
net show bridge macs       # MAC table (for Ethernet forwarding)

On the ONYX switch (InfiniBand) These commands run after SSHing or connecting via console to the switch:

show interfaces ib status  # InfiniBand port state (Active/Down/Polling)
show ib counters           # Per-port error counters
show ib sm                 # Subnet manager status and master SM location
show interface ib 1/1      # Detailed state of a specific IB port

In the ProDeploy simulator The simulator gives each device its own terminal session. Click the DGX node in the topology to open a green-prompt DGX host session. Click a leaf switch to open a blue-prompt Cumulus Linux session. Each session has its own command set -- you cannot run show dcb pfc in the DGX terminal or ibstat in the switch terminal, just as you cannot on real hardware.

The commands in this chapter are grouped by which device they run on. The device context label before each command block tells you which terminal to open in the simulator when you reach the labs.