Skip to content

BGP-EVPN Multi-Tenancy on Spectrum-X · Part 4 of 8

Act 4 - GBP Microsegmentation (Extending Ch23)

In Chapter 23 we introduced GBP (Group-Based Policy) in the context of AI networking security - EPGs (Endpoint Policy Groups) as a logical grouping mechanism and policy contracts as the allow/deny rules between them. In this act we extend that foundation to the operational configuration of GBP within a multi-tenant Spectrum-X fabric, with focus on the TCAM implementation and the practical limits practitioners need to plan for.

GBP on Spectrum-4 works by assigning a 16-bit scalable group tag (SGT) to each EPG. When a packet arrives at an ingress leaf, the Spectrum-4 ASIC looks up the source IP in a policy table and stamps the packet with the appropriate SGT. At the egress point - either the destination leaf or an inline enforcement switch - the ASIC reads the SGT and looks up the policy contract in the TCAM to determine whether the packet should be forwarded or dropped. This enforcement is hardware-accelerated with no per-packet CPU involvement.

The practical value of this in a GPU cloud is significant. Consider a scenario where TenantA has both GPU training nodes and a storage cluster on the same Spectrum-X fabric, but TenantB should not be able to reach TenantA's storage even if a routing misconfiguration momentarily leaks a prefix. GBP provides a second enforcement layer: even if TenantB receives TenantA's storage route, the TCAM will drop the packet because TenantB's SGT has no permit contract to TenantA's storage EPG.

# Define EPGs (endpoint groups) - leaf-01
cumulus@leaf-01:~$ nv set system security gbp epg TENANT1_COMPUTE sgt 100
cumulus@leaf-01:~$ nv set system security gbp epg TENANT1_STORAGE sgt 101
cumulus@leaf-01:~$ nv set system security gbp epg TENANT2_COMPUTE sgt 200
cumulus@leaf-01:~$ nv set system security gbp epg TENANT2_STORAGE sgt 201

# Bind EPGs to interfaces (or subnets via ACL)
cumulus@leaf-01:~$ nv set interface swp3 security gbp epg TENANT1_COMPUTE
cumulus@leaf-01:~$ nv set interface swp5 security gbp epg TENANT1_STORAGE

# Create policy contracts
# Allow TENANT1_COMPUTE to reach TENANT1_STORAGE
cumulus@leaf-01:~$ nv set system security gbp contract C1_COMPUTE_TO_STORAGE \
  src-epg TENANT1_COMPUTE dst-epg TENANT1_STORAGE action permit

# Allow intra-EPG (compute-to-compute within tenant)
cumulus@leaf-01:~$ nv set system security gbp contract C1_INTRA_COMPUTE \
  src-epg TENANT1_COMPUTE dst-epg TENANT1_COMPUTE action permit

# Deny TENANT2 to TENANT1 storage (explicit deny)
cumulus@leaf-01:~$ nv set system security gbp contract CROSS_TENANT_DENY \
  src-epg TENANT2_COMPUTE dst-epg TENANT1_STORAGE action deny log

cumulus@leaf-01:~$ nv config apply

# Verify TCAM entries consumed
cumulus@leaf-01:~$ nv show system security gbp stats
EPG Policies in TCAM: 6 / 4096 available
Contract entries:      4 / 8192 available
SGT bindings active:   48 hosts

TCAM capacity is a planning constraint. Spectrum-4 supports up to 4,096 EPG policy entries and 8,192 contract entries. In a SuperPOD with 128 leaf switches, each switch enforces its own TCAM policy locally - policies are not shared across switches. For most GPU cloud deployments, a practical upper bound is 32 EPGs and 256 contracts per tenant, which leaves ample headroom even on a 32-tenant fabric.

POLICY MATRIX
GBP Microsegmentation Contracts
SRC / DSTT1 ComputeT1 StorageT2 ComputeT2 Storage
T1 Compute
T1 Storage
T2 Compute
T2 Storage
Permit
8
Deny
8
Unset
0
TCAM model
Line-rate

One operational note: GBP contracts are directional. A permit contract from TENANT1_COMPUTE to TENANT1_STORAGE does not automatically permit return traffic from TENANT1_STORAGE back to TENANT1_COMPUTE. In stateless Ethernet environments, you must explicitly create the reverse contract or use a stateful policy mode. For RoCEv2 training traffic, which is RDMA Write (unidirectional data push from initiator to target), this asymmetry is actually desirable - the storage EPG should not be able to initiate connections to the compute EPG.