Skip to content

Segment Routing for AI Fabrics · Part 5 of 8

Act 5 — EVPN + SRv6 for Multi-Tenant AI

Building the model…

Three example tenants sharing one cluster. Isolation has to hold in every fabric, not just the compute rails.

Chapter 15 covered EVPN-VXLAN for multi-tenancy. EVPN+SRv6 replaces VXLAN with SRv6 as the underlay transport, and the combination is increasingly the preferred design for new GPU cloud infrastructure deployments.

The EVPN-VXLAN problem at scale

EVPN-VXLAN requires each leaf to:

  1. Maintain a VTEP (VXLAN Tunnel Endpoint) for every remote leaf
  2. Encapsulate every tenant frame in a VXLAN header (8 bytes) inside a UDP/IP header (28 bytes)
  3. Perform encapsulation and decapsulation at every leaf-to-leaf boundary

For a 64-leaf SuperPOD, each leaf maintains 63 VTEP tunnels. Under high-availability scenarios, the VTEP table state must be replicated and verified. The 36-byte VXLAN/UDP/IP encapsulation reduces effective MTU for the inner payload.

EVPN+SRv6: the simplification

In EVPN+SRv6, the tenant route (RT-5, IP prefix route) carries an SRv6 VPN SID as its next-hop instead of a VTEP IP address. The SID encodes both the destination (which leaf to reach) and the tenant context (which VRF to deliver into). No VXLAN headers, no VTEPs.

EVPN-VXLAN route announcement:
  RT-5: 10.100.1.0/24
    Next-hop: VTEP 192.168.1.5       ← The VTEP IP of the remote leaf
    VXLAN VNI: 1000                  ← Carried separately

EVPN+SRv6 route announcement:
  RT-5: 10.100.1.0/24
    Next-hop: 2001:db8:0:leaf05::100 ← This IS the SID (End.DT4 in VRF RED)
                                       No VTEP, no VNI — SID encodes everything

When traffic is forwarded to 10.100.1.0/24 from leaf-01, leaf-01 encapsulates it with an IPv6 outer header destined to 2001:db8:0:leaf05::100. The Spectrum-4 ASIC on leaf-05 sees this destination matches its local End.DT4 SID for VRF RED, decapsulates, and performs an IPv4 lookup in VRF RED. No VXLAN header, no VNI lookup — the SID carries it all.

COMPARISON
EVPN-VXLAN vs EVPN+SRv6: Control and Data Plane
Step through control plane events and data plane encapsulation for both approaches.
EVPN-VXLAN
BGP EVPN RT-5 route with VTEP next-hop
BGP UPDATE:
  Route Type: 5 (IP Prefix)
  Route Distinguisher: 65001:100
  Prefix: 10.100.1.0/24
  Next-hop: 192.168.1.5       ← VTEP IP of remote leaf
  VXLAN VNI: 1000             ← Separate attribute
  ESI: 0 (single-homed)
  
# Each leaf must maintain:
# - VTEP table for 63 remote leaves
# - VNI-to-VRF mapping table
# - MAC/ARP table per VNI
Control plane carries VTEP IP + VNI separately. Receiver must maintain VTEP state and correlate VNI at packet arrival.
EVPN+SRv6
BGP EVPN RT-5 route with SRv6 VPN SID
BGP UPDATE:
  Route Type: 5 (IP Prefix)
  Route Distinguisher: 65001:100
  Prefix: 10.100.1.0/24
  Next-hop: 2001:db8:0:leaf08::100  ← SID encodes BOTH
                                       destination AND VRF
  # No separate VNI attribute needed
  # No VTEP table needed
  # SID = End.DT4 (decap + VRF RED lookup)

# Each leaf only needs:
# - Normal IPv6 route to the SID prefix (via IS-IS)
# - Local SID table (own SIDs only)
The SRv6 VPN SID encodes both where to go (leaf-08) and what to do there (VRF RED lookup). No VTEP state, no VNI correlation.
PropertyEVPN-VXLANEVPN+SRv6
Outer encap overhead50 bytes (Eth+IP+UDP+VXLAN)40 bytes (IPv6 only)
VTEP state per leafN-1 VTEP entriesNone
Tenant ID mechanismVNI (separate attribute)Embedded in SID function
Control plane lookups (egress)VTEP → VNI-VRF → FIB (3)SID → VRF FIB (2)
MTU headroom−50 bytes−40 bytes (or 0 w/ uSID)
SR-TE path steeringNot nativeNative (SID list in header)

EVPN+SRv6 configuration walkthrough

# On leaf-01: configure EVPN+SRv6 for Tenant RED
router bgp 65001
  address-family l2vpn evpn
    advertise-all-vni
    segment-routing srv6
     locator LEAF01                   # Use leaf-01's SRv6 locator
  !
!

# Assign End.DT4 SID to VRF RED:
router bgp 65001
  vrf RED
    address-family ipv4 unicast
      segment-routing srv6
       locator LEAF01
       sid-allocation per-vrf         # One SID per VRF (vs per-prefix)
    !
  !
!

# Verify EVPN routes are carrying SRv6 VPN SIDs:
vtysh -c "show bgp l2vpn evpn route type 5"
# Expected:
# Route Distinguisher: 65001:100
#   *> 10.100.1.0/24 via 2001:db8:0:leaf03::100
#      SRv6 VPN SID: 2001:db8:0:leaf03::100 (End.DT4/VRF RED)
#      Path: 65001 65003

Multi-tenant traffic separation with SRv6 + GBP

When combined with GBP (Group-Based Policy, Chapter 27), SRv6 provides both path isolation and security policy enforcement. The SRv6 path carries traffic to the correct VRF; GBP at the egress leaf enforces whether the source EPG (Endpoint Group) is allowed to communicate with the destination EPG. The two mechanisms are complementary and operate at different layers.


Read aloudPage 5