Skip to content

Chapter 1: Operating Systems, Management Platforms, and First Power-On · Part 6 of 9

First power-on -- what actually happens

Jump to section

1. Before you touch anything2. The software stack -- what runs where3. The management philosophy -- centralised vs distributed4. Is it all CLI? Or all orchestrated?5. The platforms you need to know -- and why NVIDIA dominates6. First power-on -- what actually happens7. First access -- what you expect to see8. How the connectivity looks -- the full picture9. What you now know

You have racked the hardware, run the cables, and you are ready to power on. The order matters.

First power-on sequence — click each step

Total bring-up time from cold: approximately 35–40 minutes

Step 1: Power on the out-of-band management network first

The 1GbE management switches and BMC ports need to be up before anything else powers on. You need to be able to reach each device's BMC for power control and console access even if the device has not booted yet.

Step 2: Power on the spine switches

The InfiniBand spine switches (Q3400) come up next. ONYX boots in about 3-5 minutes. During this time it loads its configuration, initialises ASICs, and brings up interfaces. The InfiniBand Subnet Manager may start on the spine if configured as the SM location (common in smaller deployments).

Step 3: Power on the leaf switches

The QM9700 leaf switches boot next (2-3 minutes). They connect upward to the spine and outward to the DGX nodes. At this point the fabric structure is physically present but no nodes are connected.

Step 4: Start UFM

If UFM is running on a dedicated management server (recommended for production), start it now. UFM begins its discovery process -- it will discover the switches and build the initial topology map. No LIDs are assigned yet because no nodes are connected.

Official NVIDIA DGX SuperPOD H100 management rack configuration diagram.

Figure: Official NVIDIA DGX SuperPOD H100 management rack example. It makes the bring-up order more concrete: management switches, UFM appliances, and supporting servers are all separate infrastructure that need to be available before the compute nodes come online.

Step 5: Power on DGX nodes

DGX nodes take 8-12 minutes to POST and boot. During boot: the BIOS initialises hardware, DGX OS starts, NVIDIA drivers load, and the nvidia-fabricmanager daemon starts. When it starts, it initialises the NVSwitch fabric inside the node and then registers the node's ConnectX-7 NICs with UFM. UFM sees the new endpoints, assigns LIDs, and updates routing tables across all switches.

Critical: why order matters

If DGX nodes come up before the switches and UFM are ready, the nvidia-fabricmanager daemon cannot find the fabric. It enters a retry loop. Once switches and UFM are ready, it retries and eventually succeeds -- but the initial training job may fail if it started before fabric manager completed initialisation. Always bring up the fabric before the compute.

67% complete

Previous

The platforms you need to know -- and why NVIDIA dominates

Next

First access -- what you expect to see