Skip to content

BGP-EVPN Multi-Tenancy on Spectrum-X · Part 4 of 8

Act 4 - GBP Microsegmentation (Extending Ch23)

Building the model…

Three example tenants sharing one cluster. Isolation has to hold in every fabric, not just the compute rails.

In Chapter 23 we introduced GBP (Group-Based Policy) in the context of AI networking security - EPGs (Endpoint Policy Groups) as a logical grouping mechanism and policy contracts as the allow/deny rules between them. In this act we extend that foundation to the operational configuration of GBP within a multi-tenant Spectrum-X fabric, with focus on the TCAM implementation and the practical limits practitioners need to plan for.

GBP on Spectrum-4 works by assigning a 16-bit scalable group tag (SGT) to each EPG. When a packet arrives at an ingress leaf, the Spectrum-4 ASIC looks up the source IP in a policy table and stamps the packet with the appropriate SGT. At the egress point - either the destination leaf or an inline enforcement switch - the ASIC reads the SGT and looks up the policy contract in the TCAM to determine whether the packet should be forwarded or dropped. This enforcement is hardware-accelerated with no per-packet CPU involvement.

The practical value of this in a GPU cloud is significant. Consider a scenario where TenantA has both GPU training nodes and a storage cluster on the same Spectrum-X fabric, but TenantB should not be able to reach TenantA's storage even if a routing misconfiguration momentarily leaks a prefix. GBP provides a second enforcement layer: even if TenantB receives TenantA's storage route, the TCAM will drop the packet because TenantB's SGT has no permit contract to TenantA's storage EPG.

# Define EPGs (endpoint groups) - leaf-01
cumulus@leaf-01:~$ nv set system security gbp epg TENANT1_COMPUTE sgt 100
cumulus@leaf-01:~$ nv set system security gbp epg TENANT1_STORAGE sgt 101
cumulus@leaf-01:~$ nv set system security gbp epg TENANT2_COMPUTE sgt 200
cumulus@leaf-01:~$ nv set system security gbp epg TENANT2_STORAGE sgt 201

# Bind EPGs to interfaces (or subnets via ACL)
cumulus@leaf-01:~$ nv set interface swp3 security gbp epg TENANT1_COMPUTE
cumulus@leaf-01:~$ nv set interface swp5 security gbp epg TENANT1_STORAGE

# Create policy contracts
# Allow TENANT1_COMPUTE to reach TENANT1_STORAGE
cumulus@leaf-01:~$ nv set system security gbp contract C1_COMPUTE_TO_STORAGE \
  src-epg TENANT1_COMPUTE dst-epg TENANT1_STORAGE action permit

# Allow intra-EPG (compute-to-compute within tenant)
cumulus@leaf-01:~$ nv set system security gbp contract C1_INTRA_COMPUTE \
  src-epg TENANT1_COMPUTE dst-epg TENANT1_COMPUTE action permit

# Deny TENANT2 to TENANT1 storage (explicit deny)
cumulus@leaf-01:~$ nv set system security gbp contract CROSS_TENANT_DENY \
  src-epg TENANT2_COMPUTE dst-epg TENANT1_STORAGE action deny log

cumulus@leaf-01:~$ nv config apply

# Verify TCAM entries consumed
cumulus@leaf-01:~$ nv show system security gbp stats
EPG Policies in TCAM: 6 / 4096 available
Contract entries:      4 / 8192 available
SGT bindings active:   48 hosts

TCAM capacity is a planning constraint. Spectrum-4 supports up to 4,096 EPG policy entries and 8,192 contract entries. In a SuperPOD with 128 leaf switches, each switch enforces its own TCAM policy locally - policies are not shared across switches. For most GPU cloud deployments, a practical upper bound is 32 EPGs and 256 contracts per tenant, which leaves ample headroom even on a 32-tenant fabric.

POLICY MATRIX
GBP Microsegmentation Contracts
SRC / DSTT1 ComputeT1 StorageT2 ComputeT2 Storage
T1 Compute
T1 Storage
T2 Compute
T2 Storage
Permit
8
Deny
8
Unset
0
TCAM model
Line-rate

One operational note: GBP contracts are directional. A permit contract from TENANT1_COMPUTE to TENANT1_STORAGE does not automatically permit return traffic from TENANT1_STORAGE back to TENANT1_COMPUTE. In stateless Ethernet environments, you must explicitly create the reverse contract or use a stateful policy mode. For RoCEv2 training traffic, which is RDMA Write (unidirectional data push from initiator to target), this asymmetry is actually desirable - the storage EPG should not be able to initiate connections to the compute EPG.


Read aloudPage 4