UFM (Unified Fabric Manager) is the management plane for InfiniBand fabrics. You have seen it as a web GUI and as the source of topology visualisations. But as a network engineer, you want to know: how does UFM actually communicate with switches, and over which network?

UFM runs on a dedicated server -- typically a standalone 2U server with its own BMC and OOB management connection. It needs two network connections:

1. OOB management connection: UFM's management Ethernet port connects to the OOB management switch. UFM uses this to SSH into every switch's mgmt0 interface for configuration pushes, log collection, and management-plane health checks.

2. InfiniBand connection: UFM has an InfiniBand HCA connected to the fabric it manages. This is how UFM discovers the fabric topology -- it uses the IB Subnet Manager protocol, which runs directly over InfiniBand (not Ethernet). The SM sends SubnGet MADs (Management Datagrams) to each switch port and node to discover LIDs, GUIDs, and link properties.

The key insight for network engineers: UFM uses two completely different communication paths for two different functions:

• Fabric topology and LID assignment: InfiniBand MADs over the IB fabric itself.
• Switch configuration, firmware updates, logging: SSH/REST over the OOB management network.

This means if the OOB network is down, UFM can still see the fabric topology (via IB MADs) but cannot push configuration changes to switches. If the IB fabric is down, UFM cannot discover topology but can still SSH into switches via OOB for diagnostics.

UFM's switch management model

When UFM needs to configure an ONYX switch, it SSHes into the switch's mgmt0 interface and issues ONYX CLI commands programmatically, or calls the ONYX REST API. UFM maintains a list of all switch management IP addresses (the IPs you set on mgmt0). These must be reachable from UFM's OOB management interface.

# From UFM server, verifying OOB reachability to a spine switch
ip vrf exec mgmt ping 10.0.2.10  # spine switch mgmt0 IP

# Check UFM's list of managed switches
# (In UFM web GUI: Fabric > Switches -- each row shows last management contact time)

# UFM event log (on UFM server)
tail -f /var/log/ufm/event.log | grep "switch\|mgmt\|SSH"

Telemetry: DCGM and NVSM

DCGM (Data Center GPU Manager) and NVSM (NVIDIA System Management) run as agents on each DGX node. They expose metrics -- GPU utilisation, temperature, PCIe error counts, NVLink bandwidth, power draw, fan speed -- via a local REST API and via NVIDIA's management stack.

The monitoring collection path is in-band: the Prometheus/DCGM exporter on the DGX host listens on a TCP port (typically 9400), and a Prometheus scraper reaches it via the in-band management interface on the DGX's CX7 Slot1/Slot2. This is not the OOB network -- it is the DGX host's regular Ethernet interface. The OOB network is not involved in DCGM telemetry.

# On a DGX node: check DCGM exporter is running and serving metrics
systemctl status dcgm-exporter
curl http://localhost:9400/metrics | grep -E "dcgm_gpu_util|dcgm_mem_temp"

# NVSM checks
nvsm show health       # overall system health
nvsm show gpus         # per-GPU state
nvsm show nvlinks      # NVLink error counters
nvsm show storage      # NVMe drive health