ProDeploy — Learn the fabric that keeps AI clusters alive

Spectrum-X provides two complementary isolation mechanisms for multi-tenant AI clouds: VRF-per-tenant for L3 routing isolation, and GBP (Group-Based Policy) for fine-grained microsegmentation within and across VRFs. These are independent layers and both should be deployed together.

VRF-per-tenant: the routing isolation baseline

Each tenant gets a dedicated VRF with its own IP routing table. Tenant A's GPU prefixes (10.100.0.0/24) exist only in VRF_A. Tenant B's prefixes (10.200.0.0/24) exist only in VRF_B. Even if a Tenant B node sends a packet destined for a Tenant A address, the egress leaf's forwarding lookup occurs in VRF_B — where 10.100.0.0/24 has no route. The packet is dropped before it ever reaches Tenant A.

# NVUE: create tenant VRFs on Spectrum-X leaf
nv set vrf VRF_A
nv set vrf VRF_B

# Assign SVIs (Switched Virtual Interfaces) per tenant:
nv set interface vlan100 ip vrf VRF_A
nv set interface vlan100 ip address 10.100.0.254/24

nv set interface vlan200 ip vrf VRF_B
nv set interface vlan200 ip address 10.200.0.254/24

nv config apply

# Verify VRF routing isolation:
nv show vrf VRF_A router rib ipv4 route
nv show vrf VRF_B router rib ipv4 route
# Tenant A routes appear ONLY in VRF_A, never in VRF_B

EVPN RT control: inter-leaf route propagation

For routes to propagate correctly between leaf switches (so Tenant A's GPU on leaf-01 can reach Tenant A's GPU on leaf-03), EVPN Route Targets must be configured to import only same-tenant routes:

# On leaf-01: configure BGP EVPN for VRF_A
router bgp 65001 vrf VRF_A
  address-family l2vpn evpn
    advertise ipv4 unicast
    route-target import 65000:100   # only import routes tagged for Tenant A
    route-target export 65000:100   # tag our routes for Tenant A
  !

router bgp 65001 vrf VRF_B
  address-family l2vpn evpn
    advertise ipv4 unicast
    route-target import 65000:200   # only import routes tagged for Tenant B
    route-target export 65000:200
  !

With mismatched RTs, a Tenant B prefix announcement would carry RT 65000:200. Tenant A's VRF import policy only accepts RT 65000:100. The route is silently discarded at the control plane — it never enters VRF_A's routing table.

GBP microsegmentation: intra-VRF enforcement

VRFs prevent inter-tenant routing. But within a tenant, you may need finer control. GBP (Group-Based Policy) operates at the Spectrum-4 ASIC's TCAM and can enforce allow/deny policies between Endpoint Groups (EPGs) — logical groups of workloads.

A common GPU cloud use case: within VRF_A, Tenant A's GPU nodes should be able to reach the shared storage cluster, but they should not be able to reach each other's management interfaces (preventing lateral movement if one GPU node is compromised).

# NVUE: define GBP EPGs and policy
nv set system global-id group-policy enable on

# Create EPGs
nv set bridge domain br_default group-policy epg GPU-NODES id 10
nv set bridge domain br_default group-policy epg STORAGE id 20
nv set bridge domain br_default group-policy epg MGMT id 30

# Policy: GPU-NODES → STORAGE = allow
nv set bridge domain br_default group-policy contract GPU-TO-STORAGE \
  source-epg GPU-NODES destination-epg STORAGE action permit

# Policy: GPU-NODES → MGMT = deny
nv set bridge domain br_default group-policy contract GPU-TO-MGMT \
  source-epg GPU-NODES destination-epg MGMT action deny

nv config apply

# Verify GBP is programmed in TCAM:
nv show bridge domain br_default group-policy contract
cl-resource-query | grep gbp
# Shows TCAM entries consumed by GBP rules

The enforcement happens in hardware at line rate. There is no performance penalty for GBP policy — the Spectrum-4 ASIC processes the TCAM lookup on every packet forwarded, taking no additional forwarding time compared to standard L3 forwarding.

INTERACTIVE
Spectrum-X Multi-Tenant Isolation
Toggle GBP policy and simulate a cross-tenant access attempt.
TenantAGPU-01TenantAGPU-02TenantBGPU-01TenantBGPU-02Leaf-01(VRF_A)Leaf-02(VRF_B)Spine-01Storage(VRF_A)VRF_AVRF_B
■ Tenant A (VRF_A) · ■ Tenant B (VRF_B) · ■ Storage · No GBP — VRF routing only; within-VRF access uncontrolled

Testing isolation with NVIDIA Air

Before applying GBP policy changes to production, NVIDIA Air (the digital twin simulation platform, covered in detail in Chapter 31) provides a safe testing environment:

# Export current NVUE config:
nv config save --format json > fabric-config.json

# Push to Air simulation (Air CLI):
air topology push --config fabric-config.json --topology ai-fabric-4leaf

# In Air: test cross-tenant traffic
air node exec tenantB-node -- \
  ibv_rc_pingpong -d mlx5_0 -g 3 10.100.0.5
# Expected with GBP: connection refused or no route
# Expected without GBP: connection succeeds (isolation gap)