Skip to content

Chapter 18: OOB and Management Network · Part 2 of 11

Act 1 -- The Three-Fabric Model Completed

You have traced the compute fabric (Ch16) and the storage fabric (Ch17). The OOB management network is the third fabric. Unlike the first two, it is not designed for data throughput. It is designed for control-plane reliability: it must work when everything else does not.

Three-Fabric Model
DGX Cluster -- Complete Fabric Picture
click a fabric for full detail
Compute Fabric
SWITCH
QM9700 (IB) / SN5600 (RoCEv2)
SPEED / PORT
200 / 400 Gbps per port
PFC
Required -- lossless
Storage Fabric
SWITCH
SN4600C (100GbE)
SPEED / PORT
100 Gbps per port
PFC
NOT used -- ECN only
OOB Management
SWITCH
SN2201 (1GbE, 48-port)
SPEED / PORT
1 Gbps per port
PFC
N/A -- standard Ethernet
DGX H100 -- Which NIC goes to which fabric
mlx5_0-7 (CX7 HCA x8)
-> compute
enp170s0f0/1 Slot1 CX7
-> storage
enp41s0f0/1 Slot2 CX7
-> storage
BMC RJ45 (iDRAC)
-> oob

The three fabrics carry fundamentally different traffic:

Compute fabric -- GPU-to-GPU RDMA, AllReduce synchronisation, pipeline parallel messages. 100/200/400 Gbps per port. Lossless with PFC. Rail-optimised topology. Lives on the QM9700 or SN5600 switches.

Storage fabric -- NVMe-oF checkpoint writes and dataset reads. 100 Gbps per port. No PFC. 2:1 oversubscription acceptable. Lives on the SN4600C switches.

OOB management fabric -- BMC communication, SSH to switch management ports, UFM polling, firmware uploads, power control, virtual console. 1 Gbps per port. Standard L2 network. Completely physically separate. Lives on commodity 1GbE switches (Mellanox SN2201 or similar). Note: DCGM and NVSM telemetry collection is in-band -- the Prometheus/DCGM exporter on each DGX node is scraped over the in-band CX7 Slot1/Slot2 management interface, not over the OOB switch. This is covered in Act 7.

As a network engineer, the OOB network is where your classical skills still apply directly. It is a flat L2 domain with a simple L3 gateway. No ECMP. No RDMA. No PFC. But it carries traffic that will determine whether you can recover from any failure in the other two fabrics. Getting the OOB network wrong -- wrong IP ranges, overlapping with production, inadequate switch redundancy -- is how a bad day becomes a catastrophic day.