So far all commands have been run from the DGX host. Now we look at the switch side. This matters because the DGX NIC and the switch port can disagree about link state.
show topology -- the rail map
show topology is a ProDeploy command that represents the view you would get from UFM or from querying the fabric topology. It shows the rail-to-NIC mapping -- which NIC on which rail connects to which switch.
show topology — the asymmetry revealed
NIC state is UP — the NIC thinks the link is fine. But the switch port is ERROR-DISABLED. This asymmetry is the key insight: rdma link show on the DGX would show this rail as ACTIVE — you would never find this problem without also checking the switch side.
Next step: show switch port rail3 to see why the switch disabled this port.
$ show topology
DGX H100 -- Rail topology map
------------------------------------------------------------
Rail NIC GUID NIC state Switch port
------------------------------------------------------------
0 mlx5_0 0x506b4b0300a1b200 UP UP
1 mlx5_1 0x506b4b0300a1b201 UP UP
2 mlx5_2 0x506b4b0300a1b202 UP UP
3 mlx5_3 0x506b4b0300a1b203 UP ERROR-DISABLED <- fault
4 mlx5_4 0x506b4b0300a1b204 UP UP
5 mlx5_5 0x506b4b0300a1b205 UP UP
6 mlx5_6 0x506b4b0300a1b206 UP UP
7 mlx5_7 0x506b4b0300a1b207 UP UP
------------------------------------------------------------
8 rails total | 7/8 active
This output reveals something that rdma link show might not: the NIC side shows UP but the switch port shows ERROR-DISABLED. The NIC thinks the link is fine -- it is sending signals into the cable. But the switch has disabled its port due to repeated link flaps.
This asymmetry is one of the trickiest things to diagnose without a topology-aware tool. rdma link show on the DGX would show state ACTIVE for mlx5_3. Only when you look at the switch side do you discover the problem.
show switch port rail3 -- the switch-side detail
When the topology map shows an anomaly on a specific rail, the next step is to inspect that switch port in detail.
$ show switch port rail3
QM9700 Leaf Switch -- Rail 3 port status
------------------------------------------------------------
Interface: Rail3/port1
NIC connected: mlx5_3 (0x506b4b0300a1b203)
Admin state: Up (admin enabled)
Oper state: Err-disabled
Err reason: link-flap
Last flap: 00:07:43 ago
------------------------------------------------------------
Port disabled by errdisable recovery. Detected 6 link
flaps in 30 seconds. Possible causes: faulty cable,
transceiver issue, or physical layer problem.
Re-enable: no shutdown (after resolving physical fault)
Err-disabled with reason link-flap tells you exactly what happened: the switch detected the link going up and down repeatedly (6 times in 30 seconds) and disabled the port to protect the fabric from instability. The administrative state is still Up -- a human has not deliberately shut this port. The switch disabled it automatically.
What causes link flaps:
- Faulty optical transceiver (the most common cause at scale -- optics fail frequently)
- Damaged fiber cable (bent beyond minimum bend radius, crushed, connector contamination)
- Loose QSFP connector not fully seated
- Firmware bug in NIC or switch causing the link state machine to oscillate
What to do:
- Reseat the cable at both ends (NIC side and switch side)
- Replace the optical transceiver with a known-good unit
- If the problem persists, swap the cable
- After fixing the physical problem:
no shutdownon the switch port to re-enable it