Skip to content

Chapter 1: Operating Systems, Management Platforms, and First Power-On · Part 8 of 9

How the connectivity looks -- the full picture

Jump to section

1. Before you touch anything2. The software stack -- what runs where3. The management philosophy -- centralised vs distributed4. Is it all CLI? Or all orchestrated?5. The platforms you need to know -- and why NVIDIA dominates6. First power-on -- what actually happens7. First access -- what you expect to see8. How the connectivity looks -- the full picture9. What you now know

Now that you understand the software, you can map it onto the physical connections you saw in Chapter 0.

The four networks — click each to explore

Path

DGX mlx5_0→7 (ConnectX-7) → QM9700 leaf switch (Rail 0-7) → Q3400 spine switch → other leaf switches → other DGX nodes

Software

MLNX-OFED drivers, NCCL for training communication, UFM for routing, Fabric Manager for NVLink/IB initialisation

Purpose

GPU-to-GPU AllReduce during AI training. Zero tolerance for packet loss. Sub-2 microsecond latency required. No management traffic here.

Physical cable

OSFP to QSFP56 active optical cable (AOC) or OSFP DAC for short runs. Generates significant heat. Cost often exceeds switch cost.

Example traffic

NCCL AllReduce gradient tensors (140+ GB per step for 70B model), RDMA Write operations, GPUDirect traffic

Three management planes, completely separated:

Out-of-band management (1GbE)

  • Path: Management workstation -> 1GbE switch -> iDRAC port on each DGX -> BMC CPU
  • Also: Management workstation -> 1GbE switch -> ONYX switch management port
  • Also: UFM server -> 1GbE switch -> ONYX switch management port (UFM needs this to talk to switches)
  • Software: iDRAC web, IPMI, SSH to ONYX management interface
  • Purpose: Power control, console access, OS provisioning, firmware updates. Works even when compute network is down.

In-band management (10GbE/25GbE)

  • Path: DGX eth0 (management NIC, not ConnectX-7) -> 10GbE switch -> UFM server -> BCM server
  • Software: SSH to DGX OS, BCM provisioning, job scheduling
  • Purpose: OS-level management, job submission, health monitoring

Compute fabric (400G InfiniBand)

  • Path: DGX mlx5_0 -> Rail 0 leaf switch -> spine switches -> Rail N leaf switch -> mlx5_N on other DGX
  • Software: MLNX-OFED, nccl, nvidia-fabricmanager, UFM manages routing
  • Purpose: GPU-to-GPU AllReduce during training. No management traffic here.

Storage fabric (400G Ethernet)

  • Path: DGX Slot1/Slot2 dual-port ConnectX-7 ports -> Spectrum Ethernet switch -> NVMe-oF storage arrays
  • Software: NVMe-oF, GPUDirect Storage
  • Purpose: Reading training datasets, writing checkpoints
89% complete

Previous

First access -- what you expect to see

Next

What you now know