ProDeploy — Learn the fabric that keeps AI clusters alive

The loopback is the most important address on any switch in the fabric. It is reachable via any path, it never goes down due to a link failure, and it is the BGP router-ID. Get loopback assignments wrong and debugging is painful -- two switches with the same loopback produce BGP table corruption that looks like an intermittent hardware fault.

Loopback Addressing
Pod-Indexed /32 Assignment Scheme
10.10.0.0/24 -- BasePOD 1 switch loopbacks (256 addresses, click a segment)
.1-.15
.16-.31
.32-.47
.48-.63
.64-.79
.80 - .255 RESERVED
Leaf switches -- .1-.15
leaf-01: 10.10.0.1/32
leaf-02: 10.10.0.2/32
leaf-03: 10.10.0.3/32
leaf-04: 10.10.0.4/32
[.5-.15 reserved]
Why /32 not /24 for loopbacks?
A /24 loopback would attract traffic for the entire subnet to that switch. A /32 is exactly one address -- the switch's identity. No broadcast domain. Each switch advertises only its own /32 into BGP. The fabric table has one entry per switch, not one subnet per tier.
Pod-indexing convention
Pod N uses 10.10.(N*10).0/24. Pod 1 = 10.10.0.0/24. Pod 2 = 10.10.10.0/24. This leaves room within each /24 for device-type sub-allocation (leaf .1-.15, spine .16-.31) AND makes the pod the summarisable unit at the SuperPOD layer.

/32 loopbacks -- why host routes

Every switch loopback is a /32. Not a /24, not a /30. A /32 host route. This is a BGP fabric-specific convention that differs from traditional OSPF designs and it confuses engineers coming from enterprise backgrounds.

The reason: in a BGP unnumbered fabric, loopbacks are redistributed into BGP as /32 host routes. Each switch advertises its own loopback /32. The BGP table on each switch then has one /32 per switch in the fabric -- a flat table of identities. There are no subnet masks to maintain, no split subnets to debug, and no case where a /24 loopback subnet accidentally attracts traffic meant for a different switch.

This also means there is no broadcast domain. A /32 has one host. No ARP, no need for subnet-directed broadcast suppression.

Loopback allocation scheme for a BasePOD

BasePOD 1 switch loopbacks: 10.10.0.0/24

  Leaf switches:
    leaf-01  10.10.0.1/32
    leaf-02  10.10.0.2/32
    leaf-03  10.10.0.3/32
    leaf-04  10.10.0.4/32
    (reserve .5-.15 for leaf expansion)

  Spine switches:
    spine-01  10.10.0.16/32
    spine-02  10.10.0.17/32
    (reserve .18-.31 for spine expansion)

  UFM/management server:
    ufm-01    10.10.0.32/32

  Super-spine (if extending to SuperPOD):
    super-spine-01  10.10.0.48/32
    super-spine-02  10.10.0.49/32

The structure within the /24 is deliberate:

.1-.15: leaf switches (4 used + 11 reserved for growth)
.16-.31: spine switches (2 used + 14 reserved)
.32-.47: management/UFM
.48-.63: super-spines
.64-.255: reserved

When BasePOD 2 is added: it uses 10.10.10.0/24 for its loopbacks. Pod 3: 10.10.20.0/24. The third octet increments by 10 per pod, leaving room for the pod-internal device sub-allocation.

VTEP loopbacks for EVPN-VXLAN

In an EVPN-VXLAN deployment, each leaf switch acts as a VTEP (VXLAN Tunnel Endpoint). The VTEP encapsulates tenant traffic into VXLAN packets and sends them to remote VTEPs. The VTEP IP is the source/destination IP in the outer VXLAN IP header.

You have two options:

Option A: use the primary loopback as VTEP -- Simple. The same /32 that identifies the switch in BGP is also the VTEP. Drawback: you cannot use anycast VTEP (a resilience technique where two leaf switches share one VTEP IP for dual-homed servers) without a dedicated VTEP loopback.

Option B: dedicated loopback2 for VTEP -- The switch has two loopbacks: loopback0 for BGP router-ID and management identity, loopback1 for VTEP. In anycast VTEP deployments, two leaf switches that dual-home a server share the same loopback1 address. This allows the server to ARP for one gateway MAC and have traffic delivered via either leaf without reconvergence.

# Cumulus Linux NVUE: configure primary loopback and VTEP loopback
nv set interface lo ip address 10.10.0.1/32
nv set interface lo ip address 10.10.0.1/32  # BGP router-ID
nv set nve vxlan source-ip 10.10.0.1         # VTEP IP (Option A: same as loopback)
# Or Option B: dedicated VTEP address
nv set interface vtep0 ip address 10.10.0.65/32  # anycast VTEP pair shares .65
nv set nve vxlan source-ip 10.10.0.65
nv config apply