This is the most important conceptual difference between InfiniBand and Ethernet management, and it changes how you operate the fabric.
Management philosophy — centralised vs distributed
Queries every switch and node using InfiniBand MAD (Management Datagrams)
Discovers complete topology — every port, every cable, every GUID
Assigns LIDs (16-bit addresses) to every port in the fabric
Runs routing algorithm (FTREE, SSSP, or MINHOP) to compute optimal paths
Programs forwarding tables on every switch
Forwards packets based on UFM-programmed tables — passively
Monitors for topology changes — re-routes within seconds when a link fails
InfiniBand: centralised control via the Subnet Manager
InfiniBand's architecture requires a Subnet Manager (SM). The SM is a software process that runs on one node in the fabric (typically on the UFM server, or as a backup on a switch). It has a complete view of the entire fabric topology and is responsible for:
- Discovering all switches and nodes by querying the fabric
- Assigning every port a LID (Local Identifier) -- a 16-bit address unique within the subnet
- Computing optimal paths between all LID pairs (using algorithms like FTREE or SSSP)
- Programming every switch's forwarding table based on computed paths
- Detecting topology changes (link failures, new nodes) and re-routing automatically
When you add a new DGX node to an InfiniBand fabric, you do not configure the switches -- you let UFM discover the new node, assign it LIDs, and update the routing tables. The switches are passive recipients of UFM's decisions. This is fundamentally different from Ethernet where each switch builds its own routing table via distributed protocols.
Practical implication: On InfiniBand, the CLI on individual ONYX switches is primarily for monitoring and diagnostics. You use show commands to check port state, link speed, and error counters. You rarely use configure terminal to set InfiniBand parameters -- UFM handles that. The CLI you will use most often on an ONYX switch is show ib sm (subnet manager status), show interface ib (port states), and show ib counters (performance).
Ethernet (Spectrum-X): distributed configuration
Spectrum-X Ethernet switches are configured individually -- either via the Cumulus or SONiC CLI, or via a fabric controller (NVIDIA Air, or a third-party NMS). There is no centralised equivalent of the InfiniBand subnet manager for Ethernet. RoCEv2 configuration (PFC, ECN, DCQCN) is applied per-switch and per-port. This gives more flexibility but requires more consistent configuration management -- a PFC misconfiguration on one switch affects the entire fabric.