This is the addressing that Ch18 assumed existed. Now you plan it.
Three separate management planes, three separate address ranges:
OOB plane: BMC and switch management
This lives in 10.0.0.0/16. Physically on the OOB switch (SN2201). Never routable from
the compute fabric.
10.0.1.0/24 -- DGX BMC addresses (BasePOD 1)
10.0.1.1 DGX-01 BMC (iDRAC)
10.0.1.2 DGX-02 BMC
...
10.0.1.8 DGX-08 BMC
(.9-.127 reserved for node expansion to 127 nodes)
10.0.2.0/24 -- Switch management (BasePOD 1)
10.0.2.1 leaf-01 mgmt0
10.0.2.2 leaf-02 mgmt0
10.0.2.3 leaf-03 mgmt0
10.0.2.4 leaf-04 mgmt0
10.0.2.16 spine-01 mgmt0
10.0.2.17 spine-02 mgmt0
10.0.2.32 storage-sw-01 mgmt0
10.0.2.33 oob-sw-01 mgmt0 (the OOB switch has its own mgmt interface)
10.0.2.48 ufm-server-01
10.0.3.0/24 -- Monitoring and management servers
10.0.3.1 prometheus-01
10.0.3.2 grafana-01
10.0.3.3 ansible-control
Critical routing rule: there must be NO route from the compute fabric (10.10.x.x) to
the OOB plane (10.0.x.x). The OOB switch is not connected to any compute or storage
switch. The only access to OOB addresses is via the management workstation or jump server.
In-band management plane
DGX nodes need a management interface reachable from tools like Prometheus, Ansible, and DCGM scraper. This uses one port from each CX7 Slot1 and Slot2 adapter (bonded for redundancy), connected to the compute leaf switch -- not the storage switch -- with an IP in the compute fabric routable space, in a dedicated /24 separate from the GPU compute /32 addresses:
10.10.4.0/24 -- DGX in-band management (BasePOD 1)
10.10.4.1 DGX-01 mgmt bond (enp170s0f0 + enp41s0f0)
10.10.4.2 DGX-02 mgmt bond
...
10.10.4.8 DGX-08 mgmt bond
These are routed via the compute fabric BGP: each compute leaf advertises the /24 block for its connected DGX management bond addresses. Prometheus reaches them via normal BGP-learned routes without touching the storage fabric. They are NOT in the OOB 10.0.x.x range.
The distinction matters: if a DGX compute NIC goes down, in-band management via the storage CX7 NIC is still reachable. If the OOB switch goes down, in-band management is still reachable. You need the OOB BMC only when the host OS itself is down.
Storage plane
10.20.1.0/24 -- DGX storage NIC addresses (BasePOD 1)
10.20.1.1 DGX-01 Slot1 port0 (enp170s0f0)
10.20.1.2 DGX-01 Slot1 port1 (enp170s0f1)
10.20.1.3 DGX-01 Slot2 port0 (enp41s0f0)
10.20.1.4 DGX-01 Slot2 port1 (enp41s0f1)
10.20.1.5 DGX-02 Slot1 port0
...
10.20.1.32 DGX-08 Slot2 port1
10.20.2.0/24 -- Storage appliance NVMe-oF target portals
10.20.2.1 WEKA-01 portal 1
10.20.2.2 WEKA-01 portal 2
10.20.2.3 WEKA-02 portal 1
...