Skip to content

BGP-EVPN Multi-Tenancy on Spectrum-X · Part 6 of 8

Act 6 - Troubleshooting EVPN Tenant Isolation Failures

Multi-tenant EVPN fabrics fail in a small set of predictable patterns. A practitioner who has memorized these four failure modes and their diagnostic commands can resolve the majority of production incidents before they escalate.

Failure A: Cross-tenant route leak (TenantA routes visible in TenantB's VRF)

This is the most critical failure - it directly enables tenant-to-tenant traffic and represents a security incident in a GPU cloud. The cause is almost always a misconfigured route-target import statement on one leaf.

# Symptom: TenantB can ping TenantA's storage server
# Step 1: Check what routes are in TenantB's VRF
cumulus@leaf-02:~$ net show bgp vrf TENANT2 ipv4 unicast
BGP routing table for VRF TENANT2, address family IPv4 Unicast
BGP table version is 18

   Network          Next Hop        MED LocPrf Weight Path
*> 10.20.1.0/24     0.0.0.0              32768 ?     (local)
*> 10.20.2.0/24     10.0.0.1        0          0 65100 65001 ?
*> 10.10.1.0/24     10.0.0.1        0          0 65100 65001 ?  <-- LEAKED!

# Step 2: Check what RTs are being imported into TENANT2's BGP
cumulus@leaf-02:~$ net show bgp vrf TENANT2 evpn route type 5
...
Route Distinguisher: 65001:100  <-- This is TENANT1's RD appearing in TENANT2!
*> [5]:[0]:[24]:[10.10.1.0]
   Route target: 65000:100
   Imported into: TENANT2 VRF   <-- Root cause: TENANT2 is importing RT 65000:100

# Step 3: Check the import RT config on leaf-02
cumulus@leaf-02:~$ nv show vrf TENANT2 router bgp route-import
Route-Import Configuration for VRF TENANT2:
  from-evpn route-target 65000:200   (correct)
  from-evpn route-target 65000:100   <-- WRONG - should not be here

# Fix: Remove the erroneous RT import
cumulus@leaf-02:~$ nv unset vrf TENANT2 router bgp route-import from-evpn route-target 65000:100
cumulus@leaf-02:~$ nv config apply

# Verify the leaked route is gone
cumulus@leaf-02:~$ net show bgp vrf TENANT2 ipv4 unicast | grep 10.10
(no output - correct)

Failure B: Tenant cannot reach own storage (missing RT import)

The inverse of Failure A - a legitimate RT import is missing, so the storage prefix is not imported into the compute VRF.

# Symptom: TENANT1 compute cannot reach 10.10.3.0/24 (storage subnet)
cumulus@leaf-01:~$ net show bgp vrf TENANT1 ipv4 unicast | grep 10.10.3
(no output)

# Check what RTs are exported by the storage leaf
cumulus@storage-leaf:~$ nv show vrf TENANT1_STORAGE router bgp route-export
from-evpn route-target: 65000:103  <-- Storage uses a dedicated sub-RT

# Compute VRF only imports 65000:100 - not 65000:103
# Fix: Add the storage RT to the compute VRF import
cumulus@leaf-01:~$ nv set vrf TENANT1 router bgp route-import from-evpn route-target 65000:103
cumulus@leaf-01:~$ nv config apply

Failure C: VXLAN flood-and-learn enabled (incorrect for multi-tenant)

Flood-and-learn is the default VXLAN behaviour without EVPN - unknown unicast and BUM (Broadcast, Unknown unicast, Multicast) traffic is flooded to all VTEPs in the fabric. In a multi-tenant fabric this is catastrophic: all tenants receive all BUM traffic from all other tenants, breaking both security and performance isolation.

# Detect if F&L is active
cumulus@leaf-01:~$ nv show vxlan flooding
Flooding mode: head-end-replication    <-- WRONG for EVPN fabric
Remote VTEPs in flood list: 10.0.0.2, 10.0.0.3, 10.0.0.4

# Correct: should be EVPN-controlled
# Fix
cumulus@leaf-01:~$ nv set vxlan flooding enable off
cumulus@leaf-01:~$ nv config apply

# Verify
cumulus@leaf-01:~$ nv show vxlan flooding
Flooding mode: evpn    <-- Correct

Failure D: WJH VXLAN drop reasons

When VXLAN is misconfigured, the Spectrum-4 WJH engine logs specific drop reasons that identify the fault:

cumulus@leaf-01:~$ net show wjh
Severity  Timestamp            Drop Reason              Src IP        Dst IP
--------  -------------------  -----------------------  -----------   -----------
Error     2026-04-12 09:14:23  VXLAN_VNI_NOT_FOUND     10.30.1.5     10.0.0.2
Error     2026-04-12 09:14:31  VXLAN_VTEP_MAC_MISS     10.0.0.3      10.0.0.1
Warning   2026-04-12 09:15:02  VXLAN_VNI_MISMATCH      10.0.0.2      10.0.0.1

# VXLAN_VNI_NOT_FOUND: Ingress VNI not configured on this switch
# VXLAN_VTEP_MAC_MISS: Remote VTEP's MAC not in ARP/ND table (underlay issue)
# VXLAN_VNI_MISMATCH: Encap/decap VNI doesn't match - usually config drift

Each drop reason maps to a specific fix: VNI_NOT_FOUND means the vxlan interface was not created on this leaf; VTEP_MAC_MISS means the underlay BGP session is not advertising the VTEP loopback; VNI_MISMATCH means the same VNI is configured as different VLANs on different leaves - a classic copy-paste configuration error.