A DGX cluster running a 70B parameter training job generates roughly 4 GB/s of telemetry: GPU power samples, NVLink utilisation counters, fabric port error rates, buffer occupancy histograms, application timing logs. Almost none of it is visible to you unless you have built the pipeline to collect and store it.
Without that pipeline, your monitoring model looks like this:
- Training job slow -> ML engineer pings you
- You run
ibstaton the affected DGX -> find a degraded HCA - You check the switch -> find a flapping port
- You fix the port
- Job recovers -- or was already checkpointed and restarted from scratch
The problem: step 1 to step 5 takes 20-45 minutes. On a 256-GPU cluster at $40K/hr, that is $13-30K per incident, assuming the job can even recover cleanly. A job that ran for 18 hours without checkpointing loses everything.
The observability model looks like this instead:
rx_symbol_errcounter on leaf-2 port 14 trends upward over 6 hours- Alert fires at threshold -- you investigate during business hours
- You replace the DAC cable before the link degrades to err-disabled
- Training job never notices
The gap is trend data vs point-in-time snapshots. The CLI commands you learned in Ch3 and Ch4 are snapshots. Observability is the continuous collection of those same counters, stored as time-series, so you can see direction of travel not just current state.
UFM REST API
Click an endpoint to see its response shape, key fields, and recommended polling strategy.
All currently active alarms in the fabric. Empty list = no known issues. Non-empty list tells you what UFM has already detected — always check this before starting a manual investigation.
Key response fields
E.g. PortDown, BERAlarm, SMFailover — categorises the fault
Critical | Major | Minor | Warning
GUID of the affected port, switch, or host
Human-readable fault description
ISO 8601 — when the alarm was raised (NTP-sync essential)
Sample response
[
{
"alarm_type": "BERAlarm",
"severity": "Warning",
"affected_object": "0x506b4b0300a1c2d4",
"description": "Bit error rate threshold exceeded on port",
"timestamp": "2025-11-14T03:47:22Z",
"alarm_id": 10042
}
]curl example
curl -sk -u admin:pass \ https://ufm-host/ufmRest/resources/alarms
The three pillars of AI fabric observability are:
- UFM REST API -- fabric topology, port counters, and events at scale
- DCGM -- GPU and NVLink metrics that correlate with fabric failures
- Prometheus + Grafana -- storage, alerting, and visualisation
They are not replacements for the CLI. They are the continuous layer that tells you when to reach for the CLI.