Out-of-band capacity is a counting exercise. Six 48-port management switches give 288 copper access ports and 24 × 100 GbE uplinks. Twelve of those uplinks carry dual attachment — one link from each switch to each border leaf — leaving twelve spare. Against roughly 115 endpoints (72 node BMCs, 16 switch management ports, edge and gateway management, controller and subnet-manager interfaces, PDUs, console servers and environmental sensors), that is generous, and the generosity is the point: six switches distribute failure domains so that losing one costs a twelfth of your management reach rather than all of it.
Three rules keep the management network trustworthy. Put it in a dedicated VRF with no route leaking to tenant space. Do not casually bridge dual uplinks without loop protection — use a routed dual-uplink design with distinct access subnets, or an explicitly designed MLAG. And keep an independent console path: if the management switches uplink through the border leaves, then a border failure removes both the service and the tool you would use to diagnose it, so a terminal server on a separate access route is not optional.
Addressing, laid out once and then automated. The automation platform's IPAM is the source of truth; this table is the shape it should hold.
| Block | Use |
|---|---|
10.255.0.0/24 | Loopbacks, VTEP addresses, router IDs |
10.255.8.0/21 | Point-to-point links between fabric devices and to the edge |
10.10.0.0/16 | Out-of-band: BMCs, switch management, controllers, subnet managers |
10.20.0.0/16 | Provisioning and PXE virtual network |
100.64.0.0/13 | Tenant VPC space, a /21 per tenant |
203.0.113.0/24 | Documentation prefix, shown as a placeholder. In a real build this is the assigned public IPv4 block, subdivided into infrastructure and VIP space, tenant elastic-IP and NAT pools, and load-balancer addresses |
2001:db8::/32 | Documentation prefix. A real build carries an assigned aggregate — a /40, for example, yields 256 customer /48s |
10.255.0.0/2410.255.8.0/2110.10.0.0/1610.20.0.0/16100.64.0.0/13/21 per tenant203.0.113.0/242001:db8::/32/40, for example, yields 256 customer /48sTwo disciplines matter more than the exact numbers. Never advertise documentation space — 203.0.113.0/24 and 2001:db8::/32 exist for diagrams and must never reach a provider. And treat route origin authorisation as part of the design: the public aggregate needs a valid ROA matching the origin ASN and its maximum length, aligned with the routing registry objects and the provider's filters, verified before turn-up rather than discovered during it.
Tenancy is one object across two fabrics. A tenant is a VPC on the Ethernet side and a partition on the InfiniBand side, created together. Virtual networks stretch across the frontend pair via EVPN; internet access flows through the gateway nodes' NAT and elastic-IP mapping; and the compute fabric's partition keys confine that tenant's collectives to its own GPUs. Bare-metal tenants get switch-port isolation per virtual network; virtualised and container tenants get the overlay extended onto the host. The important property is that there is one provisioning operation, not two that can drift apart — a tenant whose Ethernet isolation and fabric partition disagree is a security finding waiting to be written up.