A single physical event -- a DAC cable with a marginal connector on leaf-3 port 22 -- produces a cascade of observable events across four separate monitoring layers. Understanding the cascade is how you attribute root cause in under 10 minutes instead of 2 hours.
Cross-Layer Correlation Timeline
A single marginal DAC cable triggers a cascade across four monitoring layers. Click any event to see its detail. The layer that anomalies first is the root cause layer.
IBSymbolErrorRising alert fires
Prometheus rule: rate(symbol_errors[10m]) > 0.17 for 5m → alert fires. PagerDuty notification sent. Engineer investigates during business hours. Link is still Active — no job impact yet.
Here is the cascade in order, with the timestamp offset from the initial event:
T+0s -- Physical layer: The marginal connector introduces signal integrity noise. The DAC's DSP starts seeing elevated pre-FEC bit errors. No counter has incremented yet.
T+15s -- IB layer: SymbolErrors begins incrementing on both ends of the link (DGX-07 HCA port 1, and leaf-3 port 22). UFM detects this as a counter delta on the next poll cycle. No alarm has fired yet.
T+3m -- UFM alert: IBSymbolErrorRising alert fires. You get a PagerDuty notification. The link is still Active.
T+7m -- NCCL layer: The IB link's increasing error rate forces retransmissions at the transport layer. AllReduce busbw measured by nccl-tests drops from 146 GB/s to 89 GB/s. NCCL logs show increased timeout_count. The job is still running but slowing.
T+12m -- GPU layer: DCGM shows DCGM_FI_DEV_GPU_UTIL on DGX-07 dropping from 97% to 71%. The GPU is spending more time waiting at the AllReduce barrier. GPUUtilDrop alert fires.
T+18m -- Training logs: The training framework (PyTorch DDP / Megatron) logs a NCCL timeout or Watchdog abort. The job may checkpoint and restart, or may fail entirely.
T+20m -- ML engineer's Slack message: "Training seems slow / crashed."
With a calibrated monitoring stack, you are already investigating at T+3m. Without it, you start at T+20m.
Correlating timestamps across streams
The critical skill is timestamp alignment across four systems that may have different clock sources:
| Layer | Data source | Timestamp source |
|---|---|---|
| IB fabric | UFM event API | UFM server clock |
| GPU / NVLink | DCGM -> Prometheus | DGX host clock (NTP) |
| NCCL | Log files on DGX | DGX host clock (NTP) |
| Training | Framework logs | DGX host clock (NTP) |
All DGX hosts and UFM server must be NTP-synchronised to the same time source (preferably PTP-disciplined for sub-millisecond accuracy). A clock skew of 5 seconds between the UFM server and a DGX host will make a causal chain look like two independent events.
Practical correlation workflow:
- In Grafana: set a time window starting 5 minutes before the reported symptom
- Stack the panels: GPU util (DCGM), NVLink BW (DCGM), UFM alarm state, NCCL busbw (if logged to Prometheus via a custom exporter)
- Look for the earliest anomaly -- that is your root cause layer
- If UFM SymbolErrors leads GPU util drop by 7+ minutes, the fault is physical layer (cable/optics)
- If GPU util drops first and UFM shows no errors, the fault is in the compute path (NCCL misconfiguration, NVLink degradation, or CPU-side bottleneck)
From a job ID to a switch port
The workflow above assumes you already know which nodes to look at. Most incidents do not arrive that way. They arrive as "job 48213 was slow last night" -- a job identifier, a rough time, and nothing else.
The bridge from that to a set of switch ports is the scheduler, and it is a step worth writing down because it is the one people improvise badly.
While a job is running, the environment carries the mapping:
| Variable | What it gives you |
|---|---|
SLURM_JOB_ID | The identifier everyone else will quote at you |
SLURM_JOB_NODELIST | The node list -- this is the one you need |
SLURM_JOB_NUM_NODES | How many nodes, for sanity-checking scale |
SLURM_PROCID | The MPI rank, so a rank number in an NCCL log becomes a host |
SLURM_STEP_NUM_TASKS | Total ranks, i.e. expected participants in a collective |
SLURM_RESTART_COUNT | How many times this job has already died and retried |
SLURM_JOB_IDSLURM_JOB_NODELISTSLURM_JOB_NUM_NODESSLURM_PROCIDSLURM_STEP_NUM_TASKSSLURM_RESTART_COUNTSLURM_PROCID is the quiet one. NCCL reports failures by rank -- "rank 47
timed out waiting on rank 112" -- and rank numbers mean nothing to a fabric
engineer until they are hosts. This is the translation.
SLURM_RESTART_COUNT is worth a dashboard panel of its own. A job that has
restarted eleven times is telling you about the fabric even if nobody has
filed a ticket.
After the fact, reconstruct it from the accounting database:
# Which nodes did this job actually run on, and how did it end?
sacct -j 48213 --format=JobID,Start,End,State,NodeList,ExitCode --long
# Everything that failed in the window, across all users
sacct --allusers \
--starttime=2026-08-30T22:00 --endtime=2026-08-31T04:00 \
--state=FAILED,TIMEOUT,NODE_FAIL \
--format=JobID,Start,End,State,NodeList
# Nodes the scheduler has taken out of service, and why
sinfo -R
sinfo -R deserves attention: the scheduler drains nodes for reasons, those
reasons are free text somebody typed, and a cluster with four nodes drained
for "IB link flapping" is a fabric report that never reached the fabric team.
The full chain, then, is:
job ID -> sacct ... --format=NodeList -> node names -> the port map
from Chapter 18's provisioning Act -> switch and port -> the counters in
Act 2 and the sweeps in Act 7.
Two practical notes. Slurm exposes the same data over REST through
slurmrestd, which is what to automate against rather than parsing sacct
output -- and it pairs naturally with the UFM REST API in Act 2, so a
correlation job can pull both sides itself. And benchmark numbers taken from
a shared allocation are not measurements; if the run did not have
--exclusive, treat its busbw as an anecdote.