ProDeploy — Learn the fabric that keeps AI clusters alive

The Subnet Manager is the most important software component in an InfiniBand fabric, and the least visible until it fails.

show ib sm -- what is running your fabric

switch # show ib sm

Subnet Manager:
  Master SM:
    Node GUID:    0x506b4b0300000001
    Port GUID:    0x506b4b0300000002
    LID:          1
    Priority:     14
    SM type:      UFM
    SM version:   6.8.0
    SM state:     Master

  Standby SM:
    Node GUID:    0x506b4b0300000003
    LID:          2
    Priority:     12
    SM type:      OpenSM
    SM state:     Standby

Subnet prefix:    0xfe80000000000000
Subnet size:      256 nodes, 512 ports
Last sweep:       00:01:23 ago
Sweep interval:   5 seconds

show ib sm — subnet manager states

Subnet Manager:
  Master SM:
    Node GUID:  0x506b4b0300000001
    LID:        1
    Priority:   14
    SM type:    UFM
    SM state:   Master
  
  Standby SM:
    Node GUID:  0x506b4b0300000003
    LID:        2
    Priority:   12
    SM type:    OpenSM
    SM state:   Standby

Last sweep:     00:01:23 ago
Subnet size:    256 nodes

Diagnosis: Healthy configuration. UFM is master, OpenSM standby. Failover available within 30–60 seconds if UFM fails.

Action: No action. This is the correct production state.

Reading the output:

Master SM: UFM -- SM state: Master -- UFM is running the fabric and is the active SM. This is the healthy state.

Standby SM: OpenSM -- SM state: Standby -- A backup SM is running on another host. If the master SM fails, the standby takes over within 30-60 seconds. The transition causes a brief fabric disruption as the new SM does a full fabric sweep and reprograms routing tables.

Priority: 14 / 12 -- SM priority determines which SM becomes master in an election. Higher priority wins. If UFM and OpenSM are both running, UFM (priority 14) is master. If UFM fails, OpenSM (priority 12) takes over.

Last sweep: 00:01:23 ago -- The SM periodically sweeps the fabric to detect topology changes. If this shows a very long time (hours), the SM is not performing maintenance sweeps -- it may be stuck or overloaded.

Subnet size: 256 nodes, 512 ports -- This tells you what the SM has discovered. If the node count is less than expected, some nodes are not registered with the SM.

What happens when the SM restarts

When the SM (UFM) restarts, the following sequence occurs:

SM enters discovery mode -- queries every port with management datagrams (MADs)
SM builds complete topology map (takes 30 seconds to several minutes depending on fabric size)
SM assigns LIDs to all newly discovered or modified ports
SM computes routing tables for all path pairs
SM programs forwarding tables on every switch
Fabric becomes fully operational again

During this sequence, InfiniBand traffic still flows on previously programmed paths. The fabric does not go dark during an SM restart -- it uses cached routing tables. However, any new connections that require path records will fail until the SM completes its sweep.

The SM storm:

A "Subnet Manager storm" happens when two or more SM instances are running simultaneously and both believe they are the master. They send conflicting MADs, trying to overwrite each other's LID assignments and routing tables. Symptoms: LIDs changing rapidly, training jobs getting random connection failures, show ib sm showing multiple Masters.

Cause: UFM failover where the old master did not properly relinquish control before the standby took over, or someone accidentally started OpenSM on a DGX node.

Fix: Identify which SM is correct (usually UFM). Stop all other SM instances. Wait for the surviving SM to complete a clean sweep.

# Check for rogue SM instances on DGX nodes
# On each DGX host:
ps aux | grep -E "opensm|ufm"
systemctl status opensm  # should be inactive on compute nodes

Routing algorithm selection

The SM's choice of routing algorithm significantly affects training performance. The algorithms differ in how they distribute traffic across available paths.

SM routing algorithms — what each does

Fat Tree routing

Routes traffic along the canonical up-down paths in a fat-tree topology. Up-paths go from edge to spine, down-paths go from spine to edge. Traffic is distributed across equal-cost paths at each level.

Pros

+Optimal for perfectly symmetrical fat-tree topologies
+Maximises bisection bandwidth
+Selected automatically by UFM when fat-tree is detected
+AllReduce traffic stays within single rails

Cons

−Requires a perfectly balanced fat-tree — missing links or asymmetric connections degrade performance
−Not suitable for irregular topologies

When to use: Use for any DGX BasePOD or SuperPOD deployment. This is the correct default.

FTREE (Fat Tree): Designed specifically for fat-tree topologies like DGX SuperPOD. Routes traffic along the shortest path in the fat-tree, maximising bisection bandwidth. This is the correct algorithm for a properly built SuperPOD. UFM selects it automatically when it detects a fat-tree topology.

MINHOP: Minimises hop count for each path. Good for general topologies. Does not specifically optimise for fat-tree bisection bandwidth.

SSSP (Shortest Shortest Path): Distributes traffic across equal-cost paths. Better load balancing than MINHOP for irregular topologies or partially failed fabrics.

DFSSSP: Deterministic variant of SSSP. Produces consistent routing tables across SM restarts (same inputs always produce same output). Important for reproducible performance benchmarking.

Changing the routing algorithm requires a full SM sweep to reprogram all switches. Do this during a maintenance window, not during active training.