Skip to content

Chapter 19: IP Addressing and Planning · Part 7 of 11

Act 6 -- Management Address Planning in Full Detail

Building the model…

The same cluster with all four networks drawn: compute rails, frontend, storage and out-of-band, each with its own switches and its own failure domain.

This is the addressing that Ch18 assumed existed. Now you plan it.

Management Address Planning
Three Separate Planes -- Three Separate Ranges
Range
10.0.1.0/24 (BasePOD 1)
NIC
AST2600 BMC 1GbE RJ45 (rear panel)
Switch
SN2201 OOB switch -- physically separate
Routable from other fabrics
NOT routable from compute or storage fabric
Use this plane when
Host OS is down. Hard power cycle. BIOS/firmware update. Physical access substitute.
Device
IP Address
Notes
DGX-01 BMC
10.0.1.1
iDRAC -- always on (standby power)
DGX-02 BMC
10.0.1.2
leaf-01 mgmt0
10.0.2.1
Switch management -- VRF mgmt
spine-01 mgmt0
10.0.2.16
UFM server
10.0.3.1
UFM OOB NIC
Typical command using this plane
$ ipmitool -I lanplus -H 10.0.1.1 -U admin -P pass chassis status

Three separate management planes, three separate address ranges:

What actually needs a management address

Before allocating, know what you are allocating for. A SuperPOD's management plane is not just BMCs and switches -- there is a set of cluster services that must exist, several of which are odd-numbered because they are quorum members, and one pair that needs a shared virtual address.

ServiceCountWhat it doesAddress note
Head node2Cluster manager daemon and database, scheduler control, provisioningHA pair -- needs a VIP on top of the two host addresses
Login node2User entry pointNot an HA pair. Two independent hosts behind DNS round-robin -- two addresses, no VIP
Kubernetes control plane3Cluster orchestration servicesQuorum -- allocate three, contiguously
NMX-M3Manages and provisions the NVSwitch traysQuorum. This is the NVLink fabric's management authority (Chapter 12)
UFM2InfiniBand fabric management and Subnet ManagerRedundant pair -- see below
Storage controllersper arrayStorage fabric endpointsCovered under the storage plane below
ServiceHead node
Count2
What it doesCluster manager daemon and database, scheduler control, provisioning
Address noteHA pair -- needs a VIP on top of the two host addresses
ServiceLogin node
Count2
What it doesUser entry point
Address noteNot an HA pair. Two independent hosts behind DNS round-robin -- two addresses, no VIP
ServiceKubernetes control plane
Count3
What it doesCluster orchestration services
Address noteQuorum -- allocate three, contiguously
ServiceNMX-M
Count3
What it doesManages and provisions the NVSwitch trays
Address noteQuorum. This is the NVLink fabric's management authority (Chapter 12)
ServiceUFM
Count2
What it doesInfiniBand fabric management and Subnet Manager
Address noteRedundant pair -- see below
ServiceStorage controllers
Countper array
What it doesStorage fabric endpoints
Address noteCovered under the storage plane below

Two of these routinely surprise people.

Login nodes are not highly available. They are load-shared, not failed-over. Two addresses, published through DNS, and a user whose session was on the one that died is simply disconnected. Do not allocate a VIP and do not promise HA behaviour you have not built.

UFM is a redundant pair, not a single host. Chapters 4 and 18 talk about "the UFM server" in the singular; production designs run two, and the standby takes over Subnet Manager duties on failure. That changes both the address plan and the failover expectation -- the SM identity moves, and anything that pinned itself to one UFM address has just lost its fabric management.

NMX-M is the one nobody plans for. It provisions the NVSwitch trays. If your cluster has an NVLink switch fabric, it has a management authority for that fabric, it needs three addresses, and it will not appear on any diagram that was drawn before the scale-up fabric existed.

The prefix that is already taken on every DPU host

One reservation that has nothing to do with your design and everything to do with the hardware:

192.168.100.0/30  --  RESERVED. Do not allocate.

Every BlueField DPU server has a private point-to-point link between the x86 host and the DPU's Arm cores, tunnelled over PCIe (tmfifo_net0, Chapter 18 Act 8). The DPU side ships at 192.168.100.2/30 and the host takes 192.168.100.1/30.

It is a per-host island -- the same four addresses repeat on every node and never appear on a wire, so it does not conflict with itself. It conflicts with you, if 192.168.100.0/24 is also allocated to something real. The symptom is a host that can reach a service from most nodes but not from the ones with DPUs, which is a genuinely unpleasant thing to debug.

Add it to the reserved list alongside the link-local and documentation prefixes, and check it against whatever your cluster manager uses by default for its own provisioning network before you commit the plan.

OOB plane: BMC and switch management

This lives in 10.0.0.0/16. Physically on the OOB switch (SN2201). Never routable from the compute fabric.

10.0.1.0/24   --  DGX BMC addresses (BasePOD 1)
  10.0.1.1     DGX-01 BMC (iDRAC)
  10.0.1.2     DGX-02 BMC
  ...
  10.0.1.8     DGX-08 BMC
  (.9-.127 reserved for node expansion to 127 nodes)

10.0.2.0/24   --  Switch management (BasePOD 1)
  10.0.2.1     leaf-01 mgmt0
  10.0.2.2     leaf-02 mgmt0
  10.0.2.3     leaf-03 mgmt0
  10.0.2.4     leaf-04 mgmt0
  10.0.2.16    spine-01 mgmt0
  10.0.2.17    spine-02 mgmt0
  10.0.2.32    storage-sw-01 mgmt0
  10.0.2.33    oob-sw-01 mgmt0  (the OOB switch has its own mgmt interface)
  10.0.2.48    ufm-server-01

10.0.3.0/24   --  Monitoring and management servers
  10.0.3.1     prometheus-01
  10.0.3.2     grafana-01
  10.0.3.3     ansible-control

Critical routing rule: there must be NO route from the compute fabric (10.10.x.x) to the OOB plane (10.0.x.x). The OOB switch is not connected to any compute or storage switch. The only access to OOB addresses is via the management workstation or jump server.

In-band management plane

DGX nodes need a management interface reachable from tools like Prometheus, Ansible, and DCGM scraper. This uses one port from each CX7 Slot1 and Slot2 adapter (bonded for redundancy), connected to the compute leaf switch -- not the storage switch -- with an IP in the compute fabric routable space, in a dedicated /24 separate from the GPU compute /32 addresses:

10.10.4.0/24   --  DGX in-band management (BasePOD 1)
  10.10.4.1    DGX-01 mgmt bond (enp170s0f0 + enp41s0f0)
  10.10.4.2    DGX-02 mgmt bond
  ...
  10.10.4.8    DGX-08 mgmt bond

These are routed via the compute fabric BGP: each compute leaf advertises the /24 block for its connected DGX management bond addresses. Prometheus reaches them via normal BGP-learned routes without touching the storage fabric. They are NOT in the OOB 10.0.x.x range.

The distinction matters: if a DGX compute NIC goes down, in-band management via the storage CX7 NIC is still reachable. If the OOB switch goes down, in-band management is still reachable. You need the OOB BMC only when the host OS itself is down.

Storage plane

10.20.1.0/24   --  DGX storage NIC addresses (BasePOD 1)
  10.20.1.1    DGX-01 Slot1 port0 (enp170s0f0)
  10.20.1.2    DGX-01 Slot1 port1 (enp170s0f1)
  10.20.1.3    DGX-01 Slot2 port0 (enp41s0f0)
  10.20.1.4    DGX-01 Slot2 port1 (enp41s0f1)
  10.20.1.5    DGX-02 Slot1 port0
  ...
  10.20.1.32   DGX-08 Slot2 port1

10.20.2.0/24   --  Storage appliance NVMe-oF target portals
  10.20.2.1    WEKA-01 portal 1
  10.20.2.2    WEKA-01 portal 2
  10.20.2.3    WEKA-02 portal 1
  ...
Read aloudPage 7