You can now operate an AI fabric at the observability level, not just the diagnostic level. Specifically:
UFM REST API: You know how to poll /alarms, /ports, and /events programmatically, and at what frequency each endpoint should be called. You understand that UFM REST covers the IB fabric layer only -- Ethernet and GPU layers need separate tooling.
DCGM: You know which metrics (NVLINK_BANDWIDTH_TOTAL, NVLINK_REPLAY_ERROR_COUNT, ECC_DBE_VOL_TOTAL, GPU_UTIL) act as early indicators of fabric health issues, and how DCGM plugs into Prometheus via dcgm-exporter.
Prometheus + Grafana: You know how to configure scrape jobs, write alert rules with for clauses to avoid transient noise, and why training_job_active as a synthetic metric is essential for avoiding false positives during maintenance windows.
Alert calibration: You know that PFC pause frames are expected under load, that symbol errors are only meaningful as a rate-of-change rather than an absolute count, and that alert-to-action ratio is your metric for threshold quality.
Cross-layer correlation: You know the four-layer cascade from a physical event to a training job failure, and that the layer that anomalies first in the timeline is the root cause layer. You know that NTP synchronisation across all components is a precondition for any correlation to be trustworthy.
ibdiagnet sweeps: You know how to run scheduled health baselines and why nightly sweeps catch routing correctness and BER trends that continuous counter polling misses.