In Chapter 23 we introduced GBP (Group-Based Policy) in the context of AI networking security - EPGs (Endpoint Policy Groups) as a logical grouping mechanism and policy contracts as the allow/deny rules between them. In this act we extend that foundation to the operational configuration of GBP within a multi-tenant Spectrum-X fabric, with focus on the TCAM implementation and the practical limits practitioners need to plan for.
GBP on Spectrum-4 works by assigning a 16-bit scalable group tag (SGT) to each EPG. When a packet arrives at an ingress leaf, the Spectrum-4 ASIC looks up the source IP in a policy table and stamps the packet with the appropriate SGT. At the egress point - either the destination leaf or an inline enforcement switch - the ASIC reads the SGT and looks up the policy contract in the TCAM to determine whether the packet should be forwarded or dropped. This enforcement is hardware-accelerated with no per-packet CPU involvement.
The practical value of this in a GPU cloud is significant. Consider a scenario where TenantA has both GPU training nodes and a storage cluster on the same Spectrum-X fabric, but TenantB should not be able to reach TenantA's storage even if a routing misconfiguration momentarily leaks a prefix. GBP provides a second enforcement layer: even if TenantB receives TenantA's storage route, the TCAM will drop the packet because TenantB's SGT has no permit contract to TenantA's storage EPG.
# Define EPGs (endpoint groups) - leaf-01
cumulus@leaf-01:~$ nv set system security gbp epg TENANT1_COMPUTE sgt 100
cumulus@leaf-01:~$ nv set system security gbp epg TENANT1_STORAGE sgt 101
cumulus@leaf-01:~$ nv set system security gbp epg TENANT2_COMPUTE sgt 200
cumulus@leaf-01:~$ nv set system security gbp epg TENANT2_STORAGE sgt 201
# Bind EPGs to interfaces (or subnets via ACL)
cumulus@leaf-01:~$ nv set interface swp3 security gbp epg TENANT1_COMPUTE
cumulus@leaf-01:~$ nv set interface swp5 security gbp epg TENANT1_STORAGE
# Create policy contracts
# Allow TENANT1_COMPUTE to reach TENANT1_STORAGE
cumulus@leaf-01:~$ nv set system security gbp contract C1_COMPUTE_TO_STORAGE \
src-epg TENANT1_COMPUTE dst-epg TENANT1_STORAGE action permit
# Allow intra-EPG (compute-to-compute within tenant)
cumulus@leaf-01:~$ nv set system security gbp contract C1_INTRA_COMPUTE \
src-epg TENANT1_COMPUTE dst-epg TENANT1_COMPUTE action permit
# Deny TENANT2 to TENANT1 storage (explicit deny)
cumulus@leaf-01:~$ nv set system security gbp contract CROSS_TENANT_DENY \
src-epg TENANT2_COMPUTE dst-epg TENANT1_STORAGE action deny log
cumulus@leaf-01:~$ nv config apply
# Verify TCAM entries consumed
cumulus@leaf-01:~$ nv show system security gbp stats
EPG Policies in TCAM: 6 / 4096 available
Contract entries: 4 / 8192 available
SGT bindings active: 48 hosts
TCAM capacity is a planning constraint. Spectrum-4 supports up to 4,096 EPG policy entries and 8,192 contract entries. In a SuperPOD with 128 leaf switches, each switch enforces its own TCAM policy locally - policies are not shared across switches. For most GPU cloud deployments, a practical upper bound is 32 EPGs and 256 contracts per tenant, which leaves ample headroom even on a 32-tenant fabric.
| SRC / DST | T1 Compute | T1 Storage | T2 Compute | T2 Storage |
|---|---|---|---|---|
| T1 Compute | ||||
| T1 Storage | ||||
| T2 Compute | ||||
| T2 Storage |
One operational note: GBP contracts are directional. A permit contract from TENANT1_COMPUTE to TENANT1_STORAGE does not automatically permit return traffic from TENANT1_STORAGE back to TENANT1_COMPUTE. In stateless Ethernet environments, you must explicitly create the reverse contract or use a stateful policy mode. For RoCEv2 training traffic, which is RDMA Write (unidirectional data push from initiator to target), this asymmetry is actually desirable - the storage EPG should not be able to initiate connections to the compute EPG.