In Chapter 3 you learned to read a Spectrum-X switch running Cumulus Linux. You typed show dcb pfc and show interface counters and learned what each field means. That CLI felt familiar -- it has the same verb patterns as enterprise networking, similar output formats, similar diagnostic logic.
InfiniBand is different. Not slightly different. Architecturally different in ways that change how you think about the entire fabric.
In Ethernet, every switch independently builds its own routing table through BGP or OSPF. You SSH into a switch and configure it directly. When something breaks, you trace the path hop by hop, checking each device's forwarding table.
In InfiniBand, the Subnet Manager computes the entire fabric's routing table centrally and programs every switch. The switches are passive -- they do exactly what the SM tells them. When you SSH into an ONYX switch, you can see what the SM programmed, but you cannot change InfiniBand routing parameters without going through UFM. The ONYX CLI is primarily a monitoring and diagnostics interface, not a configuration interface for routing.
This changes everything about how you investigate a problem.
InfiniBand vs Ethernet — the architectural difference
Distributed. Each switch runs BGP or OSPF independently. They exchange routes and each builds its own forwarding table.
Centralised. The Subnet Manager computes routing for the entire fabric and programs every switch. Switches are passive recipients of SM decisions.
What this means in practice:
When an InfiniBand link fails, the SM automatically reroutes around it within seconds. You may not even see a training job failure -- the SM handles it. But if the SM itself has a problem, if routing tables are inconsistent, or if LID assignment has gone wrong, the entire fabric may behave unpredictably in ways that no individual switch's CLI will explain. You need UFM and ibdiagnet to see the full picture.
This chapter teaches you the ONYX CLI for monitoring and diagnostics, the Subnet Manager commands for fabric health, ibdiagnet for full fabric sweeps, and UFM correlation for connecting CLI findings to the management plane.