Prometheus is a pull-based time-series database. Every 60 seconds (configurable) it scrapes your exporters -- dcgm-exporter on each DGX host, ufm-exporter if you build one, node-exporter for host metrics -- and stores the samples. Grafana queries Prometheus and renders dashboards.

This section covers what to configure, not how to install -- assume a working Prometheus + Grafana stack.

The scrape config

# prometheus.yml -- minimal AI fabric scrape config
scrape_configs:
  - job_name: 'dcgm'
    static_configs:
      - targets:
          - dgx-01:9400
          - dgx-02:9400
          # ... all DGX hosts
    scrape_interval: 60s
    scrape_timeout: 10s

  - job_name: 'node'
    static_configs:
      - targets:
          - dgx-01:9100
          - dgx-02:9100
    scrape_interval: 30s

  - job_name: 'ufm_custom'
    # If you build a UFM REST -> Prometheus bridge
    static_configs:
      - targets: ['ufm-exporter:9500']
    scrape_interval: 60s

Alert rules

Prometheus alert rules live in a separate YAML file loaded by the Prometheus config. Each rule defines a PromQL expression and a threshold. When the expression evaluates true for the for duration, the alert fires and is routed to Alertmanager.

# alerts/fabric.yml
groups:
  - name: fabric_health
    rules:

      # Link flap: port changes from Active within the hour
      - alert: IBPortFlapping
        expr: |
          increase(ufm_port_logical_state_changes_total[1h]) > 3
        for: 0m
        labels:
          severity: warning
        annotations:
          summary: "IB port {{ $labels.guid }} flapping"

      # Symbol error rate rising -- precursor to link failure
      - alert: IBSymbolErrorRising
        expr: |
          rate(ufm_port_symbol_errors_total[10m]) > 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Symbol errors rising on {{ $labels.guid }}"

      # GPU utilisation drop during expected training window
      - alert: GPUUtilDrop
        expr: |
          DCGM_FI_DEV_GPU_UTIL < 80
          and on(instance) training_job_active == 1
        for: 5m
        labels:
          severity: critical
        annotations:
          summary: "GPU {{ $labels.gpu }} underutilised during training"

      # NVLink bandwidth drop
      - alert: NVLinkBandwidthLow
        expr: |
          DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL < 400
          and on(instance) training_job_active == 1
        for: 3m
        labels:
          severity: warning
        annotations:
          summary: "NVLink BW {{ $value }} GB/s on {{ $labels.instance }}"

      # Double-bit ECC -- immediate action
      - alert: GPUDoubleBitECC
        expr: increase(DCGM_FI_DEV_ECC_DBE_VOL_TOTAL[5m]) > 0
        for: 0m
        labels:
          severity: critical
        annotations:
          summary: "Uncorrectable ECC error on {{ $labels.instance }} GPU {{ $labels.gpu }}"

The training_job_active metric in those rules is a synthetic gauge you push from your job scheduler -- 1 when a training job is running, 0 when idle. Without it, GPU utilisation alerts fire constantly during maintenance windows and idle periods.