ProDeploy — Learn the fabric that keeps AI clusters alive

← Labs ← Chapter 11: Monitoring & TelemetryTriage a silent fabric degradation

Desktop recommendedChecking session

Triage a silent fabric degradation

advanced

Elapsed

00:00

Scenario

An ML engineer messages you: 'Training is about 12% slower than yesterday. No errors in the NCCL logs. GPU utilisation looks normal in the dashboard.' At a quick glance: - All 8 rails report Active in ibstat - No ports are Down or Disabled in the UFM active-port view - PFC is enabled, ECN is enabled - No RDMA retransmit counters are incrementing Nothing is obviously broken, but training is measurably slower. Your task: 1. Use UFM's REST API to find which active port is flagged for high BER 2. Identify the degrading port pair on both ends of the link 3. Confirm the GPU-side symptom from DCGM and the NIC counters 4. Correlate that host view with the leaf-side FEC/PFC counters 5. Determine the root-cause layer (physical / transport / application) 6. Use the Physical Infra panel to request the connector reseat

Physical Infra

Connector reseat runbook

Use this out-of-band maintenance action after UFM/DCGM/NVUE point to a marginal physical connector. Keeping it here avoids pretending it is a stock UFM shell command.

Conditions

ufmPortsChecked
offendingPortIdentified
dcgmCorrelated
timestampsCorrelated
rootCauseLayerIdentified
remediationIssued

Need Help?

Open the step-by-step command guide for this lab.

Topology

Filled green circle = Active

Unfilled red circle = Down

Solid line = 400G link

Dashed line = degraded

Click a device node to open its terminal session

Preparing lab environment...