ProDeploy — Learn the fabric that keeps AI clusters alive

This is the decision point where most network engineers coming from enterprise backgrounds get confused. Traditional OSPF/IS-IS designs give every P2P link a /30 or /31 subnet -- two addresses, one per end. In a 200-switch AI fabric that means 2400 P2P addresses that must be planned, assigned, documented, and debugged.

BGP unnumbered (Ch15 Act 2, RFC 5549) eliminates this entirely. The link has no IPv4 address. BGP peers using the IPv6 link-local address that every interface auto-generates. The NLRI (the prefixes being advertised) remains IPv4. The mechanics work through IPv6 ND.

P2P Link Addressing
BGP Unnumbered vs Numbered /31
leaf-01
BGP ASN 4200000001
swp33
NO IPv4 address
fe80::a288:c2ff:fe01:203
Physical link -- NO IPv4
BGP over fe80:: link-local
NLRI = IPv4 /32 routes
spine-01
BGP ASN 4200000010
swp1
NO IPv4 address
fe80::b0d5:e8ff:fe01:405
BGP Unnumbered Session Establishment -- click each step
1
leaf-01Sends ICMPv6 Neighbor Solicitation on swp33
2
spine-01Replies with ICMPv6 Neighbor Advertisement (learns fe80:: address of leaf-01)
3
spine-01Sends Router Advertisement with its own fe80:: link-local address
4
leaf-01Learns spine-01's fe80:: address -- uses it as BGP neighbor address
5
leaf-01Opens BGP TCP session to spine-01's fe80:: address (RFC 5549)
6
bothExchange BGP OPEN, KEEPALIVE. BGP session established over link-local.
7
leaf-01Advertises IPv4 /32 NLRI (10.10.1.1/32) with IPv6 next-hop (RFC 5549 extension)
8
spine-01Installs route: 10.10.1.1/32 via fe80::a288... dev swp1 (ECMP-able)

When to use numbered P2P links

BGP unnumbered is the right choice for all inter-switch links in a modern AI fabric. But there are specific scenarios where you still want numbered P2P addresses:

Troubleshooting target: when you suspect a specific inter-switch link is dropping packets, you want to ping the far end of that link directly. With unnumbered links you have no IPv4 address to target -- you must use show interface swp1 counters and rely on per-interface error counts rather than reachability testing.

Legacy switch compatibility: some older switches or third-party switches in the fabric do not support BGP unnumbered. Numbered /31 links are the fallback.

Peering with external networks: the peering session from the fabric to the corporate border router almost always uses a numbered /30 or /31. BGP unnumbered requires both sides to support RFC 5549.

/31 vs /30 for numbered P2P links

Use /31 (RFC 3021). A /30 wastes two addresses (network address and broadcast) on a link that will only ever have two hosts. A /31 gives exactly two host addresses. Modern network equipment handles /31 correctly. Save your address space.

/30 link:  10.10.0.0   network address (unusable)
           10.10.0.1   switch A
           10.10.0.2   switch B
           10.10.0.3   broadcast (unusable)

/31 link:  10.10.0.0   switch A
           10.10.0.1   switch B

If you are building a numbered fabric (unusual but valid), allocate a dedicated /24 for P2P links that is separate from the loopback /24:

P2P link range: 10.10.5.0/24  (BasePOD 1, if using numbered links)
  leaf-01 <-> spine-01:  10.10.5.0/31  (.0 = leaf side, .1 = spine side)
  leaf-01 <-> spine-02:  10.10.5.2/31
  leaf-02 <-> spine-01:  10.10.5.4/31
  ...

IPv6 link-local mechanics (what actually happens with BGP unnumbered)

Every interface automatically generates an IPv6 link-local address in the fe80::/10 range. The address is derived from the interface MAC address using the EUI-64 algorithm: take the 48-bit MAC, insert ff:fe in the middle to get 64 bits, flip the 7th bit.

For a CX7 with MAC a0:88:c2:01:02:03:

Split MAC at midpoint: a0:88:c2 and 01:02:03
Insert ff:fe: a0:88:c2:ff:fe:01:02:03
Flip bit 7 of first byte: a0 = 10100000 -> bit 7 flipped -> 10100010 = a2
Result: fe80::a288:c2ff:fe01:203

This link-local address is reachable only on the directly attached link -- it cannot be routed. BGP uses it purely for the peering session exchange. The prefixes (IPv4 /32 host routes) are what gets advertised and routed.

# Verify IPv6 link-local on a Cumulus switch (auto-generated, no config needed)
ip -6 addr show swp1
# Output: inet6 fe80::a288:c2ff:fe01:203/64 scope link

# Verify BGP unnumbered session using link-local peer address
nv show vrf default router bgp peer swp1
# Peer address will show as fe80::... not an IPv4 address