A single physical event -- a DAC cable with a marginal connector on leaf-3 port 22 -- produces a cascade of observable events across four separate monitoring layers. Understanding the cascade is how you attribute root cause in under 10 minutes instead of 2 hours.
Cross-Layer Correlation Timeline
A single marginal DAC cable triggers a cascade across four monitoring layers. Click any event to see its detail. The layer that anomalies first is the root cause layer.
IBSymbolErrorRising alert fires
Prometheus rule: rate(symbol_errors[10m]) > 0.17 for 5m → alert fires. PagerDuty notification sent. Engineer investigates during business hours. Link is still Active — no job impact yet.
Here is the cascade in order, with the timestamp offset from the initial event:
T+0s -- Physical layer: The marginal connector introduces signal integrity noise. The DAC's DSP starts seeing elevated pre-FEC bit errors. No counter has incremented yet.
T+15s -- IB layer: SymbolErrors begins incrementing on both ends of the link (DGX-07 HCA port 1, and leaf-3 port 22). UFM detects this as a counter delta on the next poll cycle. No alarm has fired yet.
T+3m -- UFM alert: IBSymbolErrorRising alert fires. You get a PagerDuty notification. The link is still Active.
T+7m -- NCCL layer: The IB link's increasing error rate forces retransmissions at the transport layer. AllReduce busbw measured by nccl-tests drops from 146 GB/s to 89 GB/s. NCCL logs show increased timeout_count. The job is still running but slowing.
T+12m -- GPU layer: DCGM shows DCGM_FI_DEV_GPU_UTIL on DGX-07 dropping from 97% to 71%. The GPU is spending more time waiting at the AllReduce barrier. GPUUtilDrop alert fires.
T+18m -- Training logs: The training framework (PyTorch DDP / Megatron) logs a NCCL timeout or Watchdog abort. The job may checkpoint and restart, or may fail entirely.
T+20m -- ML engineer's Slack message: "Training seems slow / crashed."
With a calibrated monitoring stack, you are already investigating at T+3m. Without it, you start at T+20m.
Correlating timestamps across streams
The critical skill is timestamp alignment across four systems that may have different clock sources:
| Layer | Data source | Timestamp source |
|---|---|---|
| IB fabric | UFM event API | UFM server clock |
| GPU / NVLink | DCGM -> Prometheus | DGX host clock (NTP) |
| NCCL | Log files on DGX | DGX host clock (NTP) |
| Training | Framework logs | DGX host clock (NTP) |
All DGX hosts and UFM server must be NTP-synchronised to the same time source (preferably PTP-disciplined for sub-millisecond accuracy). A clock skew of 5 seconds between the UFM server and a DGX host will make a causal chain look like two independent events.
Practical correlation workflow:
- In Grafana: set a time window starting 5 minutes before the reported symptom
- Stack the panels: GPU util (DCGM), NVLink BW (DCGM), UFM alarm state, NCCL busbw (if logged to Prometheus via a custom exporter)
- Look for the earliest anomaly -- that is your root cause layer
- If UFM SymbolErrors leads GPU util drop by 7+ minutes, the fault is physical layer (cable/optics)
- If GPU util drops first and UFM shows no errors, the fault is in the compute path (NCCL misconfiguration, NVLink degradation, or CPU-side bottleneck)