The NVLink Switch System requires different diagnostic tools than the IB/RoCEv2 fabric covered in Chapters 3-4.
Diagnostics: IB vs NVLink Switch
Different fabrics, different tools. Select a fault scenario to compare the diagnostic approach on each fabric.
InfiniBand / RoCEv2
Tool
ibstat + UFM
Command
ibstat # shows State: Down ibdiagnet --ber_threshold 1e-12
Expected output
Port 1: State: Down Physical state: Polling UFM alarm: PortDown on 0x506b...
Action
UFM Subnet Manager detects down port, re-routes around it automatically (adaptive routing). Replace cable during maintenance window.
NVLink Switch System
Tool
nvidia-smi nvlink
Command
nvidia-smi nvlink --status -i 0
Expected output
GPU 00000000:03:00.0 Link 0: Active Link 5: Inactive ← degraded Link 6: Active
Action
Inactive link = reduced bandwidth. NO automatic rerouting. Replace OSFP cable immediately — the link will not recover on its own.
Key difference
IB: SM re-routes around failures automatically. NVLink Switch: no SM, no rerouting. A failed lane is a permanent bandwidth loss until the cable is replaced.
What changes vs IB
In an IB fabric, the Subnet Manager (OpenSM or UFM) manages routing, performs fabric discovery, and owns port state. A link fault shows up as a port state change that SM detects and routes around. ibstat and ibdiagnet are your primary tools.
In a NVLink Switch System:
- There is no Subnet Manager -- the NVSwitch hardware manages routing internally
- There is no adaptive re-routing -- NVSwitch uses fixed routing; a failed link is a failed link, not a rerouted path
- Fault isolation is at the driver level:
nvidia-smi nvlink --statusreports per-link state from the driver's perspective - The management interface is the NVSwitch management daemon, not UFM
DCGM NVLink Switch counters
DCGM (covered in Chapter 11) exposes NVLink Switch metrics in addition to GPU NVLink metrics:
# Check NVLink Switch health from DCGM
dcgmi dmon -e 1001,1002,1003 # NVLink switch bandwidth, errors, status
# Per-link NVLink status from nvidia-smi
nvidia-smi nvlink --status -i 0 # GPU 0, all NVLink lanes
# NVLink error counters
nvidia-smi nvlink --errorcounters -i 0
Key counter groups:
| Counter | Meaning | Alert condition |
|---|---|---|
NVLINK_BANDWIDTH_C0_TX / RX | Per-lane transmit/receive bandwidth | Drop below expected during training |
NVLINK_REPLAY_ERROR_COUNT | NVLink replay errors (like IB symbol errors) | Any non-zero value |
NVLINK_CRC_FLIT_ERROR_COUNT | CRC errors at flit level | Any non-zero value -- precursor to link failure |
NVLINK_CRC_DATA_ERROR_COUNT | Data CRC errors | Any non-zero -- more severe than flit errors |
When a NVLink Switch link fails
A failed NVLink lane does not trigger re-routing the way an IB link failure triggers SM re-computation. The NVSwitch hardware has no equivalent of OpenSM failover. A failed lane means reduced bandwidth, period.
The diagnostic sequence for a suspected NVLink fault:
# Step 1: Check per-GPU NVLink lane status
nvidia-smi nvlink --status
# Step 2: Check error counters for non-zero values
nvidia-smi nvlink --errorcounters
# Step 3: Correlate with DCGM -- look for NVLINK_BANDWIDTH drop
dcgmi dmon -e 1001
# Step 4: Check nvidia-bug-report for NVSwitch driver messages
nvidia-bug-report.sh
grep -i "nvswitch\|nvlink" /var/log/messages | tail -50
If nvidia-smi nvlink --status shows a lane as Inactive and error counters are non-zero, the lane is degraded. Replace the OSFP cable connecting the affected DGX port to the NVLink Switch module. The physical diagnosis is identical to IB: marginal DAC cable -> inspect both ends, replace with a known-good cable.