Skip to content

Chapter 15: IP Routing for AI/ML Fabrics · Part 3 of 10

Act 2 -- eBGP unnumbered: eliminating address overhead

A 200-switch AI fabric has roughly 2,400 inter-switch links (32 spines x 8 leaf connections, 8 leaf x 32 spine connections, times the pod count). In traditional eBGP deployments, every one of those links needs a point-to-point IPv4 subnet -- typically a /30 or /31. That is 2,400 subnets to assign, document, and maintain. When a new spine is added, you touch every leaf it connects to. When a link fails, you debug against a table of addresses you have to look up.

eBGP unnumbered, specified in RFC 5549 ("Advertising IPv4 Network Layer Reachability Information with an IPv6 Next Hop"), eliminates all of this. The insight is elegant: every network interface already has an IPv6 link-local address derived from its MAC address (EUI-64). That address is unique per link without any configuration. If you can establish a BGP session using that link-local address, and if you can use the link-local address as the IPv4 next-hop, you never need to assign a single point-to-point IPv4 address in your fabric.

eBGP unnumbered (RFC 5549)
Step through how a BGP session forms with no IPv4 link addressing.
Leaf1
ASN 65001 fe80::leaf1:1 (derived from MAC)
400G link
Spine1
ASN 65000 fe80::spine1:1 (derived from MAC)
Step 1/5Physical link up
Leaf1 and Spine1 are cabled. No IPv4 addresses configured on the inter-switch link. Both interfaces have auto-derived IPv6 link-local addresses based on MAC address (EUI-64).

The mechanics work in three phases. First, ICMPv6 Neighbor Discovery runs automatically. Leaf1 sends a Neighbor Solicitation to the all-nodes multicast address; Spine1 responds with a Neighbor Advertisement carrying its link-local address (fe80::...) and MAC address. Both sides learn each other's link-local addresses without any IP configuration.

Second, Spine1's routing daemon sends a Router Advertisement that includes ICMPv6 information about its link-local address. Leaf1 now knows the BGP neighbor address to use. The TCP session for BGP is established to and from the link-local addresses directly.

Third, when BGP Update messages are exchanged, the NLRI (the IP prefix being advertised, such as 10.1.1.0/24) is IPv4 -- your server addresses are still IPv4. But the BGP next-hop attribute is the IPv6 link-local address of the advertising switch, not an IPv4 address. This is what RFC 5549 defines: the Extended Next Hop Encoding capability, negotiated in the BGP Open message, allows the next-hop attribute to carry an IPv6 address for an IPv4 prefix.

The practical result is that the administrator only needs to:

  1. Assign each switch a unique BGP ASN (the 32-bit private range 4200000000-4294967294 gives you billions of values, trivially auto-assigned from the rack number)
  2. Enable BGP on all uplink interfaces with neighbor interface-range pointing to link-local addresses
  3. Redistribute connected routes so server prefixes appear in BGP

Everything else -- peering addresses, next-hop computation, route installation -- happens automatically. When a new spine is added, you cable it and enable BGP; the link-local address appears, ND runs, BGP comes up, routes propagate. There is nothing to pre-configure on the leaves.

One nuance: the BGP next-hop rewrite that normally happens when an eBGP router re-advertises a prefix continues to work correctly. When Spine1 receives a server prefix from Leaf1 (with next-hop = Leaf1's link-local address toward Leaf1) and re-advertises it to Leaf2, Spine1 rewrites the next-hop to its own link-local address toward Leaf2. Leaf2 sees the server as reachable via Spine1's local address -- correct, and still requiring no IPv4 addressing on any inter-switch link.