Skip to content

Chapter 19: IP Addressing and Planning · Part 7 of 11

Act 6 -- Management Address Planning in Full Detail

This is the addressing that Ch18 assumed existed. Now you plan it.

Management Address Planning
Three Separate Planes -- Three Separate Ranges
Range
10.0.1.0/24 (BasePOD 1)
NIC
AST2600 BMC 1GbE RJ45 (rear panel)
Switch
SN2201 OOB switch -- physically separate
Routable from other fabrics
NOT routable from compute or storage fabric
Use this plane when
Host OS is down. Hard power cycle. BIOS/firmware update. Physical access substitute.
Device
IP Address
Notes
DGX-01 BMC
10.0.1.1
iDRAC -- always on (standby power)
DGX-02 BMC
10.0.1.2
leaf-01 mgmt0
10.0.2.1
Switch management -- VRF mgmt
spine-01 mgmt0
10.0.2.16
UFM server
10.0.3.1
UFM OOB NIC
Typical command using this plane
$ ipmitool -I lanplus -H 10.0.1.1 -U admin -P pass chassis status

Three separate management planes, three separate address ranges:

OOB plane: BMC and switch management

This lives in 10.0.0.0/16. Physically on the OOB switch (SN2201). Never routable from the compute fabric.

10.0.1.0/24   --  DGX BMC addresses (BasePOD 1)
  10.0.1.1     DGX-01 BMC (iDRAC)
  10.0.1.2     DGX-02 BMC
  ...
  10.0.1.8     DGX-08 BMC
  (.9-.127 reserved for node expansion to 127 nodes)

10.0.2.0/24   --  Switch management (BasePOD 1)
  10.0.2.1     leaf-01 mgmt0
  10.0.2.2     leaf-02 mgmt0
  10.0.2.3     leaf-03 mgmt0
  10.0.2.4     leaf-04 mgmt0
  10.0.2.16    spine-01 mgmt0
  10.0.2.17    spine-02 mgmt0
  10.0.2.32    storage-sw-01 mgmt0
  10.0.2.33    oob-sw-01 mgmt0  (the OOB switch has its own mgmt interface)
  10.0.2.48    ufm-server-01

10.0.3.0/24   --  Monitoring and management servers
  10.0.3.1     prometheus-01
  10.0.3.2     grafana-01
  10.0.3.3     ansible-control

Critical routing rule: there must be NO route from the compute fabric (10.10.x.x) to the OOB plane (10.0.x.x). The OOB switch is not connected to any compute or storage switch. The only access to OOB addresses is via the management workstation or jump server.

In-band management plane

DGX nodes need a management interface reachable from tools like Prometheus, Ansible, and DCGM scraper. This uses one port from each CX7 Slot1 and Slot2 adapter (bonded for redundancy), connected to the compute leaf switch -- not the storage switch -- with an IP in the compute fabric routable space, in a dedicated /24 separate from the GPU compute /32 addresses:

10.10.4.0/24   --  DGX in-band management (BasePOD 1)
  10.10.4.1    DGX-01 mgmt bond (enp170s0f0 + enp41s0f0)
  10.10.4.2    DGX-02 mgmt bond
  ...
  10.10.4.8    DGX-08 mgmt bond

These are routed via the compute fabric BGP: each compute leaf advertises the /24 block for its connected DGX management bond addresses. Prometheus reaches them via normal BGP-learned routes without touching the storage fabric. They are NOT in the OOB 10.0.x.x range.

The distinction matters: if a DGX compute NIC goes down, in-band management via the storage CX7 NIC is still reachable. If the OOB switch goes down, in-band management is still reachable. You need the OOB BMC only when the host OS itself is down.

Storage plane

10.20.1.0/24   --  DGX storage NIC addresses (BasePOD 1)
  10.20.1.1    DGX-01 Slot1 port0 (enp170s0f0)
  10.20.1.2    DGX-01 Slot1 port1 (enp170s0f1)
  10.20.1.3    DGX-01 Slot2 port0 (enp41s0f0)
  10.20.1.4    DGX-01 Slot2 port1 (enp41s0f1)
  10.20.1.5    DGX-02 Slot1 port0
  ...
  10.20.1.32   DGX-08 Slot2 port1

10.20.2.0/24   --  Storage appliance NVMe-oF target portals
  10.20.2.1    WEKA-01 portal 1
  10.20.2.2    WEKA-01 portal 2
  10.20.2.3    WEKA-02 portal 1
  ...