DCGM (NVIDIA Data Center GPU Manager) is a daemon that runs on every DGX host and exposes GPU metrics over a local socket. A Prometheus exporter (dcgm-exporter) scrapes DCGM and makes those metrics available to Prometheus.

DCGM is the bridge between the GPU and the network fabric. Many network problems manifest first as GPU metric anomalies -- a degraded HCA shows up as reduced NVLink bandwidth before the IB error counter crosses threshold.

The metric groups that matter for fabric health

NVLink bandwidth and error counters

• DCGM_FI_DEV_NVLINK_BANDWIDTH_TOTAL -- aggregate NVLink bandwidth (GB/s). A healthy H100 in active training should be at 600-900 GB/s. Sustained drop below 400 GB/s indicates a degraded NVLink lane or NVSwitch issue.
• DCGM_FI_DEV_NVLINK_REPLAY_ERROR_COUNT_* (per link) -- NVLink replay errors. Non-zero is a problem. These increment when NVLink must retransmit a flit due to corruption -- analogous to TCP retransmits but at intra-node speeds.
• DCGM_FI_DEV_NVLINK_CRC_FLIT_ERROR_* -- CRC errors at flit level. Pre-cursor to link degradation.

ECC errors

• DCGM_FI_DEV_ECC_SBE_VOL_TOTAL -- single-bit ECC errors (correctable). A few per day is normal. Rapid accumulation signals HBM degradation.
• DCGM_FI_DEV_ECC_DBE_VOL_TOTAL -- double-bit ECC errors (uncorrectable). Any non-zero value requires immediate investigation. The GPU will typically page-retire the affected memory row.

GPU utilisation and power

• DCGM_FI_DEV_GPU_UTIL -- GPU compute utilisation %. During training this should be 95-100% between AllReduce barriers. Sustained drops to 60-70% indicate AllReduce stall -- the network is the bottleneck.
• DCGM_FI_DEV_POWER_USAGE -- power draw (W). Correlated with compute load. Sudden drops in power often mean the GPU has throttled due to thermal or ECC issues.
• DCGM_FI_DEV_GPU_TEMP -- GPU die temperature (C). H100 throttles at 83C, shuts down at 87C. Sustained operation above 80C warrants inspection of cooling.

PCIe counters (relevant when diagnosing DPU <-> CPU <-> NVMe traffic)

• DCGM_FI_DEV_PCIE_REPLAY_COUNTER -- PCIe replays. Non-zero suggests a PCIe link quality issue between the GPU and the CPU/NVMe path.

Running DCGM and dcgm-exporter

# On the DGX host -- check DCGM daemon status
systemctl status nvidia-dcgm

# Query DCGM directly via dcgmi CLI
dcgmi dmon -e 155,203,210,211,252,1001  # power, temp, ecc, nvlink bw

# dcgm-exporter exposes Prometheus metrics on :9400
curl http://localhost:9400/metrics | grep DCGM_FI_DEV_GPU_UTIL

The dcgm-exporter Helm chart is the standard deployment for Kubernetes-based DGX clusters. For bare-metal deployments, run it as a systemd service on each DGX host.

MIG and multi-tenant monitoring

In clusters where H100 GPUs are partitioned with MIG (Multi-Instance GPU) for multi-tenant inference workloads, DCGM exposes per-instance metrics with a gpu_i_id label. The metric names are identical -- you filter by instance rather than by device. Note that NVLink metrics are not available per-MIG-instance; NVLink operates at the physical GPU level and is only relevant in training configurations where MIG is not typically used.

Confidential Computing note: In clusters using H100 Confidential Computing (TEE mode), GPU metrics that would reveal information about encrypted workloads are restricted. DCGM can report power, temperature, and PCIe counters, but per-SM compute counters and some ECC details may be unavailable depending on the attestation policy. This is expected behaviour -- the TEE prevents the host from inspecting encrypted gradient buffers. Plan your observability stack accordingly if you operate mixed confidential and standard GPU partitions.