Skip to content

BGP-EVPN Multi-Tenancy on Spectrum-X · Part 7 of 8

Act 7 - NVIDIA Air for Multi-Tenant Testing

Before deploying a new multi-tenant configuration to production, every change should be validated in NVIDIA Air - the cloud-hosted network simulation platform that can reproduce a Spectrum-X SuperPOD topology with full Cumulus Linux functionality. Air is covered in depth in Ch31; this act focuses specifically on the multi-tenant EVPN validation workflow.

The validation workflow follows five steps: build, configure, inject-fault, observe, fix-and-verify. Building the Air topology can be done from exported NVUE YAML, which allows you to maintain topology-as-code:

# Export current leaf-01 NVUE config to YAML (on production switch)
cumulus@leaf-01:~$ nv config save
cumulus@leaf-01:~$ cat /etc/nvue.d/startup.yaml > /tmp/leaf-01-config.yaml

# Upload to Air and create simulation
air> topology create --file leaf-01-config.yaml --name "ch27-evpn-test"
Topology created: ch27-evpn-test (4 nodes)
Simulation ID: sim-abc123

# Start simulation
air> simulation start sim-abc123
Status: Running (all nodes up after 45s)

Once the simulation is running, inject the fault you want to test - in this case, the route-target misconfiguration that causes Failure A:

# On the simulated leaf-02, inject the RT misconfiguration
air-leaf-02> nv set vrf TENANT2 router bgp route-import from-evpn route-target 65000:100
air-leaf-02> nv config apply

# Verify the leak manifests
air-dgx-tenantb> ping 10.10.1.5
PING 10.10.1.5: 64 data bytes
64 bytes from 10.10.1.5: icmp_seq=0 ttl=63 time=0.4 ms
# Leak confirmed - TENANT2 host can reach TENANT1 storage

# Run NetQ automated isolation check
netq check evpn
EVPN check result: FAILED
  leaf-02: VRF TENANT2 importing RT 65000:100 - cross-tenant route leak detected
  Affected prefix: 10.10.1.0/24 visible in TENANT2 VRF on 3 leaves

# Fix and re-verify
air-leaf-02> nv unset vrf TENANT2 router bgp route-import from-evpn route-target 65000:100
air-leaf-02> nv config apply
netq check evpn
EVPN check result: PASSED (all tenants isolated)

The NetQ check evpn command (covered in depth in Ch28) runs automated route-target consistency checks across all leaves in the fabric and flags any cross-tenant RT imports. Integrating this check into a CI/CD pipeline - triggered on every NVUE config change - means that isolation violations are caught in simulation before they ever touch production.

For ib_write_bw-based isolation verification between tenants:

# On TenantA's simulated DGX node - start RDMA server
air-dgx-tenanta> ib_write_bw -d mlx5_0 -x 3 --port 18515

# From TenantB's simulated DGX - attempt cross-tenant RDMA (should fail)
air-dgx-tenantb> ib_write_bw -d mlx5_0 -x 3 10.10.1.5 --port 18515
RDMA CM connection failed: ECONNREFUSED
# GBP policy blocking the connection - correct behaviour

The combination of NetQ automated route-target checking and ib_write_bw connectivity testing gives operators confidence in two complementary properties: routing isolation (no routes leak between VRFs) and data-plane isolation (even if a route were to leak, GBP prevents the actual data transfer). Both checks should be mandatory gates in any GitOps pipeline managing Spectrum-X multi-tenant configuration.