The OOB management network is physically separate. It runs on its own switches -- typically the Mellanox SN2201, a 1RU switch with 48 x 1GbE RJ45 downlinks and 4 x 100GbE QSFP28 uplinks (25GbE operation is available via breakout cable or adapters, but 100GbE is the native uplink interface). The SN2201 runs Cumulus Linux, same as the storage leaf switches, which means the same NVUE command set applies. One 48-port SN2201 can terminate the BMC ports for 16 DGX nodes (each DGX has one BMC RJ45), all the switch management ports (each ONYX/Cumulus switch has one dedicated management eth0 port), and the UFM server's management interface.
A BasePOD OOB network supports roughly:
- 8 DGX nodes x 1 BMC port = 8 OOB connections
- 8 DGX nodes x 1 in-band management port (Slot1 CX7, bonded) = used on compute fabric, not OOB
- 4 leaf switches x 1 management port (eth0) = 4 OOB connections
- 2 spine switches x 1 management port (eth0) = 2 OOB connections
- 1 UFM server x 1 management port = 1 OOB connection
- 1 storage switch x 1 management port (eth0) = 1 OOB connection Total: 16-20 ports consumed from a 48-port OOB switch for a single BasePOD.
IP addressing on the OOB network
The OOB network uses a completely separate RFC 1918 range from the compute and storage fabrics. A common allocation pattern:
| Device type | Example range | Notes |
|---|---|---|
| DGX BMC ports | 10.0.1.0/24 | One IP per DGX, statically assigned |
| Switch management | 10.0.2.0/24 | eth0 on all leaf/spine switches |
| UFM server | 10.0.3.1 | Static, critical |
| Storage switch mgmt | 10.0.2.64/26 | Sub-block within switch range |
| Management workstation | 10.0.0.1/24 | Gateway for OOB network |
Do NOT use the same /24 for OOB as you use for compute or storage subnets. The most common mistake is allocating 10.20.x.x for everything and ending up with overlapping routes when you add more pods. OOB traffic should never route through the compute or storage fabric switches. It should always stay on its own physical switches, reaching the outside world via the management workstation's uplink to your corporate network.
VLANs on the OOB network
Most OOB switches run untagged traffic on a single VLAN for simplicity. In multi-tenant or larger installations, you may VLAN-separate the BMC traffic from the switch management traffic for security. This prevents a compromised DGX host from reaching switch management interfaces directly. Both use the same physical switch but different VLAN tags:
# Cumulus Linux / NVUE on the SN2201 OOB switch
# Create VLAN for BMC traffic
nv set bridge domain br_default vlan 10
# DGX BMC ports in VLAN 10
nv set interface swp1-8 bridge domain br_default access 10
# Switch management ports in VLAN 20
nv set interface swp9-16 bridge domain br_default access 20
nv config apply