Skip to content

Chapter 19: IP Addressing and Planning · Part 9 of 11

Act 8 -- Scaling from BasePOD to SuperPOD

A BasePOD is a single fabric island. A SuperPOD connects multiple BasePODs via a shared spine layer (the "super-spine"). The IP addressing must scale cleanly across this boundary.

SuperPOD Scaling
BGP Route Aggregation at Pod Boundaries
BasePODs:2
Super-Spine10.10.100.1-4/32BGP table: 2 summaries10.10.0.0/23BasePOD-1ASN: 4200000001leaf-01leaf-0210.10.0.0/2410.10.1.0/2410.20.1.0/2410.0.1.0/248 DGX x 8 NICs = 64 /32s10.10.10.0/23BasePOD-2ASN: 4200000002leaf-01leaf-0210.10.10.0/2410.10.11.0/2410.20.11.0/2410.0.11.0/248 DGX x 8 NICs = 64 /32s
Without summarisation
130+ routes
Super-spine carries every /32 from every pod. At 2 pods x 8 DGX x 8 NICs = 128 server /32s plus 20 switch loopbacks. Reconvergence after super-spine reboot takes minutes.
With summarisation
2 routes
Super-spine carries one /23 per pod. 10.10.0.0/23, 10.10.10.0/23. Reconvergence is seconds regardless of cluster size. Works only if pod addressing is aggregatable from day one.
Aggregate route config (on leaf switches, suppresses /32s toward super-spine)
# leaf-01 in BasePOD-1: aggregate compute prefix before advertising to super-spine
nv set vrf default router bgp address-family ipv4-unicast aggregate-route 10.10.0.0/23 summary-only
nv config apply
# summary-only: suppresses the individual /32 host routes from being sent upward
# The super-spine only sees 10.10.0.0/23, not 64 individual /32s

The pod aggregation problem

Each BasePOD has its own BGP ASN structure. When you connect BasePODs, you have two design options for route distribution:

Option A: full mesh iBGP between BasePODs -- All pods share the same AS and exchange routes via iBGP through route reflectors. Simpler but all /32 host routes leak between pods, creating very large BGP tables at the super-spine layer.

Option B: per-pod eBGP with summary advertisements -- Each BasePOD has its own ASN. At the super-spine, each pod advertises a summary aggregate (/16 or /24) rather than individual /32 host routes. The super-spine BGP table stays small. Traffic crosses pods via the super-spine. This is the recommended model.

With Option B, the RFC 1918 top-level partitioning you chose in Act 2 pays off directly: each BasePOD summarises its compute space to 10.10.0.0/24 (pod 1) or 10.10.10.0/24 (pod 2). The super-spine sees two /24 summaries instead of 256 individual /32s.

# On leaf-01 in BasePOD 1: aggregate the pod's compute prefix before advertising to super-spine
nv set vrf default router bgp address-family ipv4-unicast aggregate-route 10.10.0.0/20 summary-only
# summary-only suppresses the more-specific /32s from being advertised to the super-spine
nv config apply

SuperPOD address allocation example (140-node H100)

SuperPOD (BasePOD 1 + BasePOD 2, 140 DGX H100 total):

  Super-spine loopbacks:    10.10.100.0/24
    super-spine-01:  10.10.100.1/32
    super-spine-02:  10.10.100.2/32
    super-spine-03:  10.10.100.3/32
    super-spine-04:  10.10.100.4/32

  BasePOD 1 summary:  10.10.1.0/24  (compute), 10.0.1.0/24 (OOB), 10.20.1.0/24 (storage)
  BasePOD 2 summary:  10.10.11.0/24 (compute), 10.0.11.0/24 (OOB), 10.20.11.0/24 (storage)

The key insight: when you planned the pod-indexed addressing in Act 2 (pod N uses 10.10.(N*10).0/24), you implicitly built the summarisation hierarchy. The third octet encodes the pod, making /20 summaries natural (pod 1-3 = 10.10.0.0/20, pod 4-6 = 10.10.32.0/20).