Skip to content

Chapter 1: Operating Systems, Management Platforms, and First Power-On · Part 9 of 9

Taking stock

Jump to section

1. Before you touch anything2. The software stack -- what runs where3. The management philosophy -- centralised vs distributed4. Is it all CLI? Or all orchestrated?5. The platforms you need to know -- and why NVIDIA dominates6. First power-on -- what actually happens7. First access -- what you expect to see8. How the connectivity looks -- the full picture9. Taking stock
Building the model…

The same cluster with all four networks drawn: compute rails, frontend, storage and out-of-band, each with its own switches and its own failure domain.

You can now answer the questions a network engineer asks on day one:

  • DGX nodes run DGX OS (Ubuntu-based). Quantum switches run ONYX. Spectrum switches run Cumulus or SONiC. (DGX B200 nodes additionally include BlueField-3 adapters running their own firmware, but H100/H200 nodes use standard ConnectX-7 NICs throughout.)
  • InfiniBand is centrally managed by UFM -- you do not configure individual switches for routing. Ethernet is distributed -- each switch is configured individually.
  • Day-to-day operations use both GUI (UFM, BCM) and CLI (ONYX SSH, DGX SSH). Troubleshooting is mostly CLI.
  • Power-on order: out-of-band management -> spine switches -> leaf switches -> UFM -> DGX nodes.
  • First access: ONYX console cable at 115200 baud, iDRAC web GUI for DGX, UFM web GUI for fabric view.
  • First commands: nvidia-smi on DGX to verify GPUs, ibstat to verify NIC state, UFM discovery to verify fabric topology.

The next chapter introduces the CLI formally -- the commands you will use every day to monitor and troubleshoot this infrastructure.

Continue to Chapter 2 ->

100% complete

Previous

How the connectivity looks -- the full picture

Chapter complete

Back to curriculum

Read aloudPage 9