show interfaces ib status -- the overview

show interfaces ib status gives you the state of all InfiniBand ports on the switch at once. This is your starting point.

switch # show interfaces ib status

IB Interface  Admin  Oper   Width  Speed  PortGUID
------------- -----  ------  -----  -----  --------
IB1/1         Up     Active  4x     NDR    0x506b4b0300a1b200
IB1/2         Up     Active  4x     NDR    0x506b4b0300a1b201
IB1/3         Up     Down    --     --     0x000000000000000
IB1/4         Up     Polling --     --     0x000000000000000
IB1/5         Up     Active  4x     NDR    0x506b4b0300a1b204
...

Reading the columns:

Admin: Up -- An administrator has not manually disabled this port. If you see Admin: Down, someone ran shutdown on that interface. Check if it was intentional.

Oper: Active -- The operational state. This is what matters. States you will see:

• Active -- Port is fully operational. InfiniBand state machine has completed. SM has assigned a LID and programmed the routing table for this port. Traffic flows.
• Down -- No physical signal. Cable unplugged, broken, or the device on the other end is powered off.
• Polling -- Physical signal detected but InfiniBand init has not completed. The port is sending InfiniBand polling signals. Usually seen when a device is booting or when the SM has not yet discovered this port.
• Init -- Physical link established, port entered IB state machine Init state. SM has discovered it but not yet armed it. Brief transient state.
• Armed -- SM has armed the port. One more transition to Active. Should be very brief.

Width: 4x -- The lane count. NDR InfiniBand uses 4 lanes per port. If you see 1x or 2x, lanes have failed -- check the cable or transceiver. Width directly affects bandwidth: 4x NDR = 400 Gb/s, 2x NDR = 200 Gb/s, 1x NDR = 100 Gb/s.

Speed: NDR -- The link speed tier. NDR (Next Data Rate) = 400 Gb/s at 4x. You might also see HDR (200 Gb/s), EDR (100 Gb/s), or FDR (56 Gb/s) if the link has negotiated down. Negotiation down to a lower speed usually means one side cannot run NDR -- check cable compatibility and NIC firmware.

PortGUID: 0x506b4b... -- The hardware-burned GUID of the device on the other end of this cable. A GUID of all zeros means no device is connected or the device is in a non-IB state.

show interface ib 1/1 -- the detail

When you need more than the overview provides, inspect a specific port:

switch # show interface ib 1/1

IB1/1 is up, line protocol is up (connected)
  Hardware is InfiniBand, address is 506b:4b03:00a1:b200
  Interface index: 1
  MTU 4096 bytes
  Current Speed: NDR, Width: 4x
  Supported Speeds: NDR, HDR, EDR
  Auto-negotiate: enabled
  Link layer: InfiniBand
  Local Identifier (LID): 12
  Remote Identifier (Remote LID): 24
  SM LID: 1
  Capability mask: 0x2514afc8
  Description: "dgx-node-01-rail-0"
  Input rate: 142,847 Mb/s (35.7% utilisation)
  Output rate: 141,293 Mb/s (35.3% utilisation)
  5 minute input rate: 139,482 Mb/s
  5 minute output rate: 138,917 Mb/s

LID: 12 -- This port's Local Identifier as assigned by the SM. LIDs are the routing addresses in InfiniBand -- they are analogous to IP addresses but assigned by the SM, not configured statically.

Remote LID: 24 -- The LID of the device on the other end of this cable. You can use this to find the port's counterpart in UFM.

SM LID: 1 -- The LID of the Subnet Manager. If this shows 0, no SM is active -- the fabric cannot route traffic.

Description: "dgx-node-01-rail-0" -- Interface descriptions are gold for troubleshooting. If your team has labelled ports with the connected device and rail number, a problem report immediately tells you which DGX node and which GPU rail is affected.

Input rate: 142,847 Mb/s (35.7%) -- Current bandwidth utilisation as a percentage of the 400 Gb/s port capacity. 35% utilisation on a training cluster means AllReduce is in progress. Near 100% sustained means a bottleneck.