When a data centre offers GPU-as-a-Service -- selling training capacity to multiple organizations who must not see each other's model parameters or gradient updates -- network-level isolation becomes a compliance requirement, not just an operational convenience.
Network-level isolation: EVPN-VXLAN
The standard model for multi-tenant AI fabrics uses EVPN-VXLAN with per-tenant routing instances. Each tenant is assigned a MAC-VRF (MAC Virtual Routing and Forwarding) instance on every leaf switch where that tenant's servers are connected. The MAC-VRF is identified by a VLAN tag on the server-facing interface and a VNI (VXLAN Network Identifier) on the fabric-facing encapsulation.
Route Type 5 (RT5) IP prefixes extend isolation to the IP routing layer. Within a tenant's network, GPU servers communicate via VXLAN tunnels controlled by EVPN BGP signalling. Spine switches act as EVPN route reflectors -- they do not participate in tenant data planes but relay RT5 advertisements between leaves. When GPU1 (Tenant A, connected to Leaf1) sends traffic to GPU9 (also Tenant A, connected to Leaf3), Leaf1 encapsulates the packet in VXLAN with Tenant A's VNI, forwards it across the fabric, and Leaf3 decapsulates and delivers to GPU9. Leaf2, connected only to Tenant B servers, never sees Tenant A's packets -- the VNI acts as a complete data-plane barrier.
The combined MAC-VRF + RT5 model handles multi-rail deployments cleanly. A four-rail AI cluster with 512 GPUs per tenant can allocate GPU0-GPU127 to Rail 1 and GPU128-GPU255 to Rail 2. The EVPN RT5 tunnels between rails carry only the inter-rail traffic, and the spine switches route between rails using the tenant's VRF -- without ever mixing Tenant A's inter-rail packets with Tenant B's.
Server-level isolation: MIG + SR-IOV network implications
Chapter 14 explained MIG partitioning from the GPU perspective. The network implications are specific: when an H100 is partitioned into 7 MIG instances for 7 inference tenants, each instance needs a separate network identity and bandwidth allocation.
SR-IOV (Single Root I/O Virtualization) on the ConnectX-7 provides this. The physical NIC is logically split into one Physical Function (PF) and up to 128 Virtual Functions (VFs). Each VF has its own MAC address, its own RoCEv2 queue pair space, and can be rate-limited independently. A 7-instance MIG partition maps to 7 VFs, each with a configured bandwidth limit of total_link_bandwidth / 7 (adjustable based on SLA).
On the leaf switch, each VF's traffic arrives with a different VLAN tag (the hypervisor or SR-IOV driver tags each VF's traffic before it hits the switch port). The leaf maps each VLAN to a MAC-VRF instance -- 7 VLANs on one server port, 7 MAC-VRF instances, 7 independent EVPN instances. From the fabric's perspective, 7 independent tenants are connected to that single physical port.
The monitoring implication (connected to Chapter 11): UFM sees RoCEv2 counters per VF, not per physical NIC. If one MIG instance's SR-IOV VF is generating excessive pause frames, the counter is vf3_pause_frames, not nic_pause_frames. Alert thresholds calibrated for whole-GPU workloads need to be divided by the MIG partition count and applied per-VF.
GBP microsegmentation: intra-tenant isolation
VXLAN Group-Based Policy (GBP) adds a second isolation layer within a single tenant's VNI. Two GPU servers in the same VNI (same tenant, same IP subnet) can be prevented from communicating directly if their GBP tags differ. The GBP tag is encoded in the VXLAN header alongside the VNI. Policy rules on the leaf switch enforce the tag-based access control at the data plane, before packets leave the ingress port.
The AI fabric use case: within a single tenant running seven simultaneously, each sub-job gets a unique GBP tag. Job A's GPUs can only communicate with other Job A GPUs; the switch drops Job A packets destined for Job B GPUs at the data plane, without requiring firewall appliances. When the jobs complete and the orchestrator reallocates the GPUs, GBP tags are updated.
Radius-based dynamic ACL: the centralised alternative
An alternative to EVPN multi-tenancy is Radius-based dynamic ACL. When a GPU server authenticates (802.1X or MACSEC), the Radius server returns an ACL policy specific to that tenant. The leaf switch applies the ACL to that server's port, filtering inter-tenant traffic based on IP prefix lists.
This approach is operationally simpler -- one Radius server controls the entire fabric's tenant policies -- but scales poorly at high tenant counts. TCAM (Ternary Content-Addressable Memory) on typical data center switches holds 16,000-64,000 ACL entries. With 512 servers, 7 tenants, and per-server per-tenant ACL rules, TCAM exhaustion is a real risk. EVPN-VXLAN's VNI-based isolation has no such TCAM constraint; it uses hardware forwarding tables (L2 MAC and L3 IP) whose scale is an order of magnitude larger.