Before touching a single octet, you need to understand what actually needs an address in a modern AI fabric. The list is longer than it looks.
In a BGP-unnumbered fabric (Ch15 Act 2), the inter-switch links do not get IPv4 addresses. BGP peers over IPv6 link-local. So the address families that DO need planned space are:
Loopback addresses -- Every switch and every server that participates in BGP routing needs a loopback. The loopback is the switch's identity. It is the address UFM uses for SNMP polling, the address Prometheus uses for telemetry scraping, the BGP router-ID, and the VTEP IP for VXLAN encapsulation. If a loopback changes, everything that references that switch breaks simultaneously.
Server host prefixes -- Each DGX server needs an IP on its in-band management or compute interface. In a flat L3 fabric, each server is typically a /32 host route advertised by its directly connected leaf into BGP. In an EVPN-VXLAN design, each server gets an IP within its tenant VRF subnet.
Management addresses -- Three separate planes, all needing distinct ranges:
- BMC/IPMI (OOB): one IP per DGX, one per switch management port
- In-band management: one IP per DGX host management interface (CX7 Slot1/Slot2)
- UFM/monitoring server: one IP
Storage addresses -- Each DGX has two dual-port CX7 storage NICs (4 addresses total per DGX). The storage appliances each have NVMe-oF target portal IPs.
VXLAN VTEP addresses -- In an EVPN design, each leaf switch needs a VTEP IP. This is typically the same as the loopback or a dedicated loopback2 address.
VLAN IDs and VXLAN VNIs -- Not IP addresses but allocated alongside them. VLANs are 12-bit (1-4094). VNIs are 24-bit (1-16777215). They need a mapping scheme.
The scale of a BasePOD and SuperPOD makes discipline mandatory:
| Scope | Switches | DGX nodes | Server prefixes | Loopbacks |
|---|---|---|---|---|
| BasePOD | ~10 | 8-32 | 32-128 | ~50 |
| SuperPOD (1 BasePOD) | ~10 | 32 | 128 | ~50 |
| SuperPOD (20 BasePODs) | ~200 | 640 | 2560 | ~900 |
| SuperPOD (140 DGX H100) | ~50 | 140 | 560 | ~200 |
A full 140-node H100 SuperPOD is a common deployment unit. You need addresses for 140 x 8 = 1120 GPU network interfaces, 140 BMCs, 140 storage NIC pairs, ~50 switches, and associated management services. Get this wrong once and fixing it in production means coordinated downtime across everything.