VXLAN (Virtual Extensible LAN) solves the fundamental problem that Ethernet was never designed to operate at the scale of a multi-tenant data centre. A standard 802.1Q VLAN tag is 12 bits - 4,094 usable VLANs - which is insufficient when you need tens of thousands of isolated tenant segments. VXLAN extends this to a 24-bit VNI field, providing approximately 16 million distinct network identifiers, which is more than adequate even for the largest GPU clouds.
Mechanically, VXLAN encapsulation wraps the original tenant Ethernet frame in a UDP/IP/Ethernet stack with an 8-byte VXLAN header carrying the VNI. The outer IP header uses the VTEP (VXLAN Tunnel Endpoint) IP addresses - on Spectrum-X, every Spectrum-4 leaf switch acts as a VTEP. When TenantA's DGX node sends a frame, the ingress leaf encapsulates it with the tenant's VNI, routes it across the IP underlay (BGP on the spine), and the egress leaf decapsulates it before delivering to the destination host. Spectrum-4 performs both the encapsulation and decapsulation entirely in the forwarding ASIC at line rate - there is no software VTEP path and no added latency beyond the normal forwarding pipeline.
BGP-EVPN (Ethernet VPN, RFC 7432 and RFC 8365) is the control plane that distributes VTEP reachability and MAC/IP mappings across the fabric without requiring flood-and-learn. Two EVPN route types are critical for GPU cloud operation. Type-2 (MAC-IP) routes carry a host's MAC address and its associated IP address, allowing remote VTEPs to build their FDB (Forwarding Database) from control-plane information rather than by flooding unknown unicast. Type-5 (IP prefix) routes carry subnet-level prefixes and are used for inter-tenant routing at the IP layer - critically, Type-5 routes can be imported into a remote VRF only if the route-target matches, which is the enforcement mechanism for routing isolation.
Spectrum-4 uses symmetric IRB (Integrated Routing and Bridging) for inter-subnet traffic within a tenant. In symmetric IRB, both the ingress and egress VTEP perform routing. The ingress leaf routes the frame into the L3 VNI (the per-VRF routed tunnel), sends it across the fabric, and the egress leaf routes it out to the destination host. This is the preferred model for GPU fabrics because the same VNI is used in both directions, simplifying traffic engineering and telemetry. Asymmetric IRB - where only the ingress leaf routes and the egress leaf only bridges - creates asymmetric VNI usage that complicates per-tenant utilization accounting and is generally avoided in Spectrum-X deployments.
# Verify VXLAN tunnel state on leaf-01
cumulus@leaf-01:~$ nv show vxlan
Name VNI VTEP Address State MTU
------ ----- ------------- ------ ----
vxlan10 100 10.0.0.1 up 9216
vxlan20 200 10.0.0.1 up 9216
# Show VTEP peers (remote VTEPs in BGP-EVPN)
cumulus@leaf-01:~$ nv show vxlan remote-vtep
Remote VTEP State VNIs
----------- ----- ----
10.0.0.2 up 100, 200
10.0.0.3 up 100, 200
10.0.0.4 up 100, 200