ProDeploy — Build the fabric that keeps AI clusters running

The NVLink Switch System requires different diagnostic tools than the IB/RoCEv2 fabric covered in Chapters 3-4.

Diagnostics: IB vs NVLink Switch

Different fabrics, different tools. Select a fault scenario to compare the diagnostic approach on each fabric.

InfiniBand / RoCEv2

Tool

ibstat + UFM

Command

ibstat  # shows State: Down
ibdiagnet --ber_threshold 1e-12

Expected output

Port 1:
  State: Down
  Physical state: Polling
UFM alarm: PortDown on 0x506b...

Action

UFM Subnet Manager detects down port, re-routes around it automatically (adaptive routing). Replace cable during maintenance window.

NVLink Switch System

Tool

nvidia-smi nvlink

Command

nvidia-smi nvlink --status -i 0

Expected output

GPU 00000000:03:00.0
  Link 0: Active
  Link 5: Inactive  ← degraded
  Link 6: Active

Action

Inactive link = reduced bandwidth. NO automatic rerouting. Replace OSFP cable immediately — the link will not recover on its own.

Key difference

IB: SM re-routes around failures automatically. NVLink Switch: no SM, no rerouting. A failed lane is a permanent bandwidth loss until the cable is replaced.

What changes vs IB

In an IB fabric, the Subnet Manager (OpenSM or UFM) manages routing, performs fabric discovery, and owns port state. A link fault shows up as a port state change that SM detects and routes around. ibstat and ibdiagnet are your primary tools.

In a NVLink Switch System:

There is no Subnet Manager -- the NVSwitch hardware manages routing internally
There is no adaptive re-routing -- NVSwitch uses fixed routing; a failed link is a failed link, not a rerouted path
Fault isolation is at the driver level: nvidia-smi nvlink --status reports per-link state from the driver's perspective
The management interface is the NVSwitch management daemon, not UFM

DCGM NVLink Switch counters

DCGM (covered in Chapter 11) exposes NVLink Switch metrics in addition to GPU NVLink metrics:

# Check NVLink Switch health from DCGM
dcgmi dmon -e 1001,1002,1003  # NVLink switch bandwidth, errors, status

# Per-link NVLink status from nvidia-smi
nvidia-smi nvlink --status -i 0  # GPU 0, all NVLink lanes

# NVLink error counters
nvidia-smi nvlink --errorcounters -i 0

Key counter groups:

Counter	Meaning	Alert condition
`NVLINK_BANDWIDTH_C0_TX` / `RX`	Per-lane transmit/receive bandwidth	Drop below expected during training
`NVLINK_REPLAY_ERROR_COUNT`	NVLink replay errors (like IB symbol errors)	Any non-zero value
`NVLINK_CRC_FLIT_ERROR_COUNT`	CRC errors at flit level	Any non-zero value -- precursor to link failure
`NVLINK_CRC_DATA_ERROR_COUNT`	Data CRC errors	Any non-zero -- more severe than flit errors

CounterNVLINK_BANDWIDTH_C0_TX / RX

MeaningPer-lane transmit/receive bandwidth

Alert conditionDrop below expected during training

CounterNVLINK_REPLAY_ERROR_COUNT

MeaningNVLink replay errors (like IB symbol errors)

Alert conditionAny non-zero value

CounterNVLINK_CRC_FLIT_ERROR_COUNT

MeaningCRC errors at flit level

Alert conditionAny non-zero value -- precursor to link failure

CounterNVLINK_CRC_DATA_ERROR_COUNT

MeaningData CRC errors

Alert conditionAny non-zero -- more severe than flit errors

Measuring the fabric instead of inferring it

Everything above reads state. None of it tells you whether the fabric is actually moving data at the rate it should. On the scale-out side you would reach for ib_write_bw. The scale-up equivalent is nvbandwidth, and it is the tool that turns "NVLink looks fine" into a number.

# Run inside a CUDA container so the toolkit is present
sudo docker run --gpus all --rm -it nvcr.io/nvidia/cuda:12.5.0-devel-ubuntu22.04

git clone https://github.com/NVIDIA/nvbandwidth.git
cd nvbandwidth && ./debian_install.sh

# Test 20: bidirectional device-to-device copy, SM-initiated read
./nvbandwidth -t 20

The output is an all-pairs matrix -- every GPU against every other GPU. On a healthy 8-GPU node each pair reports roughly 670-673 GB/s bidirectional, against an expected figure around 650 GB/s.

Read the matrix for uniformity, not for the peak. The absolute number varies with generation and clock state and is easy to argue about. What is not arguable is the shape:

Every cell within a few GB/s of every other cell -- the fabric is healthy.
One row and its matching column low, everything else normal -- that GPU's NVLink ports, or the switch ports they land on, are degraded. The row/column pair localises it to a single endpoint.
A scattered handful of low cells with no row/column pattern -- suspect the measurement, not the fabric. Re-run with clocks pinned (Chapter 8, Act 4) before believing it.

This is the test to run before and after any NVLink cable replacement, and the one to run when a training job reports poor scaling on a node whose nvidia-smi nvlink --status is entirely clean. State and throughput are different questions.

When a NVLink Switch link fails

A failed NVLink lane does not trigger re-routing the way an IB link failure triggers SM re-computation. The NVSwitch hardware has no equivalent of OpenSM failover. A failed lane means reduced bandwidth, period.

The diagnostic sequence for a suspected NVLink fault:

# Step 1: Check per-GPU NVLink lane status
nvidia-smi nvlink --status

# Step 2: Check error counters for non-zero values
nvidia-smi nvlink --errorcounters

# Step 3: Correlate with DCGM -- look for NVLINK_BANDWIDTH drop
dcgmi dmon -e 1001

# Step 4: Check nvidia-bug-report for NVSwitch driver messages
nvidia-bug-report.sh
grep -i "nvswitch\|nvlink" /var/log/messages | tail -50

If nvidia-smi nvlink --status shows a lane as Inactive and error counters are non-zero, the lane is degraded. Replace the OSFP cable connecting the affected DGX port to the NVLink Switch module. The physical diagnosis is identical to IB: marginal DAC cable -> inspect both ends, replace with a known-good cable.