Skip to content

Chapter 1: Operating Systems, Management Platforms, and First Power-On · Part 5 of 9

The platforms you need to know -- and why NVIDIA dominates

Jump to section

1. Before you touch anything2. The software stack -- what runs where3. The management philosophy -- centralised vs distributed4. Is it all CLI? Or all orchestrated?5. The platforms you need to know -- and why NVIDIA dominates6. First power-on -- what actually happens7. First access -- what you expect to see8. How the connectivity looks -- the full picture9. What you now know

A minimal working AI cluster requires these specific software components. Some are mandatory, some are optional alternatives.

Platform landscape — what you actually need

Mandatory for any DGX InfiniBand cluster:

  • DGX OS (or vanilla Ubuntu with NVIDIA drivers -- but then you lose nvsm integration)
  • ONYX on all Quantum switches (no alternative -- it is the only OS for these switches)
  • UFM on a management server (technically optional but practically required at any scale -- without it you run opensm manually, which is painful)
  • Fabric Manager (nvidia-fabricmanager) daemon on each DGX host (required for NVLink initialisation)

Mandatory for Spectrum-X RoCEv2 clusters:

  • Cumulus Linux or SONiC on Spectrum switches (your choice)
  • MLNX-OFED driver on each DGX host (for ConnectX-7 RDMA support)

Optional but standard:

  • Base Command Manager (BCM) for provisioning and job management
  • DCGM (Data Center GPU Manager) for GPU health monitoring
  • NCCL (NVIDIA Collective Communications Library) -- ships with DGX OS, required for distributed training

Why NVIDIA specifically?

Vertical integration. If you deploy DGX nodes, NVIDIA owns the entire GPU-to-fabric interface. ConnectX-7 NICs are designed for DGX's NVLink topology. Quantum switches are designed for the rail-optimised DGX wiring pattern. UFM understands DGX's 8-NIC architecture and assigns LIDs accordingly. No other vendor integrates at this depth -- mixing GPU vendors, NIC vendors, and switch vendors means you own the integration problems yourself.

CUDA ecosystem. Nearly all AI training frameworks (PyTorch, JAX, TensorFlow) are CUDA-native. The collective communications stack (nccl), the direct-memory DMA path (gpudirect), and the GPU diagnostic toolchain (nvidia-smi, dcgmi) are all built around CUDA semantics. Switching GPU vendors means re-qualifying the entire software stack.

Reference architecture. DGX SuperPOD is a validated, documented, widely-deployed design. When something breaks, there is a known-good configuration to compare against. Thousands of engineers have debugged the same topology, and the patterns are documented. This is worth more than it sounds when you are troubleshooting at 2 AM.

This does not mean NVIDIA is always the right choice. Chapter 0 covers the alternatives. But it explains why this curriculum focuses on NVIDIA's stack.

56% complete

Previous

Is it all CLI? Or all orchestrated?

Next

First power-on -- what actually happens