This is the addressing that Ch18 assumed existed. Now you plan it.
Three separate management planes, three separate address ranges:
What actually needs a management address
Before allocating, know what you are allocating for. A SuperPOD's management plane is not just BMCs and switches -- there is a set of cluster services that must exist, several of which are odd-numbered because they are quorum members, and one pair that needs a shared virtual address.
| Service | Count | What it does | Address note |
|---|---|---|---|
| Head node | 2 | Cluster manager daemon and database, scheduler control, provisioning | HA pair -- needs a VIP on top of the two host addresses |
| Login node | 2 | User entry point | Not an HA pair. Two independent hosts behind DNS round-robin -- two addresses, no VIP |
| Kubernetes control plane | 3 | Cluster orchestration services | Quorum -- allocate three, contiguously |
| NMX-M | 3 | Manages and provisions the NVSwitch trays | Quorum. This is the NVLink fabric's management authority (Chapter 12) |
| UFM | 2 | InfiniBand fabric management and Subnet Manager | Redundant pair -- see below |
| Storage controllers | per array | Storage fabric endpoints | Covered under the storage plane below |
Two of these routinely surprise people.
Login nodes are not highly available. They are load-shared, not failed-over. Two addresses, published through DNS, and a user whose session was on the one that died is simply disconnected. Do not allocate a VIP and do not promise HA behaviour you have not built.
UFM is a redundant pair, not a single host. Chapters 4 and 18 talk about "the UFM server" in the singular; production designs run two, and the standby takes over Subnet Manager duties on failure. That changes both the address plan and the failover expectation -- the SM identity moves, and anything that pinned itself to one UFM address has just lost its fabric management.
NMX-M is the one nobody plans for. It provisions the NVSwitch trays. If your cluster has an NVLink switch fabric, it has a management authority for that fabric, it needs three addresses, and it will not appear on any diagram that was drawn before the scale-up fabric existed.
The prefix that is already taken on every DPU host
One reservation that has nothing to do with your design and everything to do with the hardware:
192.168.100.0/30 -- RESERVED. Do not allocate.
Every BlueField DPU server has a private point-to-point link between the x86
host and the DPU's Arm cores, tunnelled over PCIe (tmfifo_net0, Chapter 18
Act 8). The DPU side ships at 192.168.100.2/30 and the host takes
192.168.100.1/30.
It is a per-host island -- the same four addresses repeat on every node and
never appear on a wire, so it does not conflict with itself. It conflicts
with you, if 192.168.100.0/24 is also allocated to something real. The
symptom is a host that can reach a service from most nodes but not from the
ones with DPUs, which is a genuinely unpleasant thing to debug.
Add it to the reserved list alongside the link-local and documentation prefixes, and check it against whatever your cluster manager uses by default for its own provisioning network before you commit the plan.
OOB plane: BMC and switch management
This lives in 10.0.0.0/16. Physically on the OOB switch (SN2201). Never routable from
the compute fabric.
10.0.1.0/24 -- DGX BMC addresses (BasePOD 1)
10.0.1.1 DGX-01 BMC (iDRAC)
10.0.1.2 DGX-02 BMC
...
10.0.1.8 DGX-08 BMC
(.9-.127 reserved for node expansion to 127 nodes)
10.0.2.0/24 -- Switch management (BasePOD 1)
10.0.2.1 leaf-01 mgmt0
10.0.2.2 leaf-02 mgmt0
10.0.2.3 leaf-03 mgmt0
10.0.2.4 leaf-04 mgmt0
10.0.2.16 spine-01 mgmt0
10.0.2.17 spine-02 mgmt0
10.0.2.32 storage-sw-01 mgmt0
10.0.2.33 oob-sw-01 mgmt0 (the OOB switch has its own mgmt interface)
10.0.2.48 ufm-server-01
10.0.3.0/24 -- Monitoring and management servers
10.0.3.1 prometheus-01
10.0.3.2 grafana-01
10.0.3.3 ansible-control
Critical routing rule: there must be NO route from the compute fabric (10.10.x.x) to
the OOB plane (10.0.x.x). The OOB switch is not connected to any compute or storage
switch. The only access to OOB addresses is via the management workstation or jump server.
In-band management plane
DGX nodes need a management interface reachable from tools like Prometheus, Ansible, and DCGM scraper. This uses one port from each CX7 Slot1 and Slot2 adapter (bonded for redundancy), connected to the compute leaf switch -- not the storage switch -- with an IP in the compute fabric routable space, in a dedicated /24 separate from the GPU compute /32 addresses:
10.10.4.0/24 -- DGX in-band management (BasePOD 1)
10.10.4.1 DGX-01 mgmt bond (enp170s0f0 + enp41s0f0)
10.10.4.2 DGX-02 mgmt bond
...
10.10.4.8 DGX-08 mgmt bond
These are routed via the compute fabric BGP: each compute leaf advertises the /24 block for its connected DGX management bond addresses. Prometheus reaches them via normal BGP-learned routes without touching the storage fabric. They are NOT in the OOB 10.0.x.x range.
The distinction matters: if a DGX compute NIC goes down, in-band management via the storage CX7 NIC is still reachable. If the OOB switch goes down, in-band management is still reachable. You need the OOB BMC only when the host OS itself is down.
Storage plane
10.20.1.0/24 -- DGX storage NIC addresses (BasePOD 1)
10.20.1.1 DGX-01 Slot1 port0 (enp170s0f0)
10.20.1.2 DGX-01 Slot1 port1 (enp170s0f1)
10.20.1.3 DGX-01 Slot2 port0 (enp41s0f0)
10.20.1.4 DGX-01 Slot2 port1 (enp41s0f1)
10.20.1.5 DGX-02 Slot1 port0
...
10.20.1.32 DGX-08 Slot2 port1
10.20.2.0/24 -- Storage appliance NVMe-oF target portals
10.20.2.1 WEKA-01 portal 1
10.20.2.2 WEKA-01 portal 2
10.20.2.3 WEKA-02 portal 1
...