VNI allocation is a discipline that practitioners often treat as an afterthought and then spend significant time unwinding. A 24-bit VNI namespace gives you 16,777,214 usable values - which sounds limitless until you realize that silent misconfiguration (two tenants accidentally sharing a VNI) causes immediate traffic mixing without any routing error, and that this category of bug is extremely difficult to detect without a centralized VNI registry.
The recommended allocation scheme for a Spectrum-X SuperPOD follows a hierarchical range model that separates traffic types, tenants, and administrative functions:
VNI Range Allocation - Spectrum-X SuperPOD Reference Design
============================================================
Range Purpose Size
----------- -------------------------------- --------
100-199 TENANT1: L2 compute segments 100 VNIs
200-299 TENANT1: L2 storage segments 100 VNIs
1000 TENANT1: L3 VNI (symmetric IRB) 1 VNI
1100-1199 TENANT2: L2 compute segments 100 VNIs
1200-1299 TENANT2: L2 storage segments 100 VNIs
2000 TENANT2: L3 VNI 1 VNI
... (pattern repeats per tenant, tenant offset = 1000 x tenant_id) ...
9000-9099 Management plane 100 VNIs
9100-9199 OOB overlay 100 VNIs
9900 Default/untagged catch 1 VNI
10000-10999 Shared inference services 1000 VNIs
11000-65534 Reserved / future expansion
This layout has several important properties. L2 and L3 VNIs for each tenant are in adjacent but non-overlapping ranges, making mis-assignment immediately visible in configuration review. The management plane has a distinct high-range allocation that is administratively separate from tenant ranges. The L3 VNI (the per-VRF routed VNI for symmetric IRB) uses a flat 1000 x tenant_id formula, making it trivial to compute programmatically.
# Configure VNI range - NVUE
cumulus@leaf-01:~$ nv set vrf TENANT1 vni 1000
cumulus@leaf-01:~$ nv set vrf TENANT2 vni 2000
# Verify no VNI conflicts across all configured VNIs
cumulus@leaf-01:~$ nv show vxlan
Name VNI Local VTEP Remote VTEPs State
--------- ----- -------------- -------------- -----
vxlan100 100 10.0.0.1 10.0.0.2 up
vxlan101 101 10.0.0.1 10.0.0.2 up
vxlan200 200 10.0.0.1 10.0.0.2 up
vxlan1000 1000 10.0.0.1 10.0.0.2 up
vxlan2000 2000 10.0.0.1 10.0.0.2 up
# Check for duplicate VNIs across bridge domains (should return empty)
cumulus@leaf-01:~$ nv show bridge domain br_default vlan | grep -c "duplicate"
0
VNI range conflicts cause silent traffic mixing that manifests as intermittent connectivity and subtle performance degradation - frames from TenantA's RDMA flows are delivered to TenantB's QPs, which increment error counters without surfacing a clear routing failure. The monitoring chapter (Ch28) covers how to detect this class of error via NetQ.
The other critical VNI planning consideration is MTU. VXLAN adds 50 bytes of overhead (8-byte VXLAN header + 8-byte UDP + 20-byte outer IP + 14-byte outer Ethernet). RoCEv2 training traffic runs at 4,096-byte or 8,192-byte RDMA message sizes. The underlay MTU must be set to at least 9,216 bytes (jumbo frames) to accommodate the largest tenant frames plus VXLAN overhead without fragmentation. Every underlay interface - including the spine uplinks - must be configured at jumbo MTU; a single 1,500-byte MTU link anywhere in the path will silently fragment VXLAN frames, causing catastrophic RDMA performance degradation.
# Set jumbo MTU on all fabric-facing interfaces
cumulus@leaf-01:~$ nv set interface swp1-4 link mtu 9216
cumulus@leaf-01:~$ nv set interface vlan10 ip mtu 9216
cumulus@leaf-01:~$ nv set interface vlan20 ip mtu 9216
cumulus@leaf-01:~$ nv config apply
# Verify MTU consistency
cumulus@leaf-01:~$ nv show interface --view=brief | awk '{print $1, $5}' | grep swp
swp1 9216
swp2 9216
swp3 9216
swp4 9216