Standard eBGP distributes traffic equally across all ECMP paths -- it assumes all paths are equivalent. In a real AI fabric, they are not. A five-stage Clos fabric may have spine-to-super-spine links of different speeds. A spine that has lost one of its four downlinks to a destination leaf now has 75% of the bandwidth it had before. Sending equal traffic to a reduced-capacity spine causes congestion that flows back as PFC pause frames.
The BGP Link Bandwidth Extended Community solves the capacity signalling problem. When a spine advertises a route to a leaf, it encodes the total forwarding capacity toward the destination leaf as a community attribute. If Spine1 normally has four 400 Gbps links to Leaf4 but one has failed, it advertises the Server5 prefix with a bandwidth community value of 1200 (three links x 400 Gbps) instead of 1600. Leaf1 sees all four spines and their respective bandwidth community values. Rather than distributing traffic equally, it performs weighted ECMP proportional to those values: Spine1 gets 3/16 of the traffic, the three healthy spines get 4/16 each. The ASIC implements this via a bucket indirection table where the number of buckets pointing to each next-hop reflects the weight. This mechanism is called Weighted Packet Spraying (WPS) in some vendor implementations.
The bandwidth community mechanism works reactively: it responds to failures after they happen. BGP DPF (Deterministic Path Forwarding, IETF draft draft-wang-idr-dpf) goes a step further -- it partitions the fabric into logical routing planes before any failure, guaranteeing that specific workloads can never compete for the same physical path.
The core idea is fabric coloring. The physical three-stage Clos fabric -- which has many equal paths between any leaf pair -- is divided into multiple logical fabrics by coloring the eBGP sessions. Session coloring happens at peering establishment: the eBGP session between Leaf1 and Spine1 is configured as "black." The session between Leaf1 and Spine2 is configured as "gray." Only routes matching a session's color are advertised over that session.
Route coloring works as follows: Server5 is connected to Leaf4. Leaf4 originates the route to Server5 as uncolored (matching all colors). When Leaf4 advertises this route to Spine1 over the black session, Spine1 records the route as "black." When Spine1 re-advertises to Leaf1 over the black session, Leaf1 receives a black route. Similarly, Leaf4 advertises to Spine2 over the gray session; Spine2 re-advertises to Leaf1 over the gray session; Leaf1 receives a gray route. Leaf1 now has two routes to Server5: one black (through Spine1) and one gray (through Spine2). It can configure the black route as active and the gray as backup, or use both simultaneously for different traffic classes.
Traffic-to-color mapping is the final piece. Several mechanisms can attach a color to an outgoing flow:
Filter-Based Forwarding (FBF) / Policy-Based Routing: A traffic classifier on the leaf switch matches flows by source/destination QPair range, source UDP port range, or DSCP value. Matched flows are redirected to the black or gray logical fabric path instead of the default ECMP path. This is the most operationally straightforward approach and requires no server-side changes.
On-box agent: A software agent running on the switch probes each logical fabric using active measurements and distributes new workloads to the fabric with the best measured quality. Requires ASIC-level RoCEv2 flow awareness to classify flows by QPair.
Central controller: A fabric controller (such as HPE Juniper's CPCE -- Containerized Path Computation Engine) has topology visibility across all nodes, receives streaming telemetry, and pushes forwarding decisions to the switches. The controller can make globally optimal decisions that individual switches cannot see -- for example, rebalancing a tenant's flows across fabrics when a spine degrades, before any switch-local congestion mechanism would trigger.
EVPN-VXLAN correlation: The most powerful form. MAC-VRF instances on the leaf are associated with specific fabric colors. Traffic entering MAC-VRF-black (Tenant A's overlay) is automatically routed over the black physical fabric. Traffic entering MAC-VRF-gray (Tenant B) uses gray. The control plane and data plane isolation are both enforced by the same mechanism.
The multi-tenant scenario this enables: a 128-GPU cluster shared between seven LLM training jobs. The highest-priority job -- a model that must complete a training iteration within a strict time window for a real-time product -- is pinned to the black fabric. That fabric's three spine paths are exclusively its. The other six jobs share the remaining gray fabric. The high-priority job's JCT is predictable because it cannot be affected by the other six jobs' traffic, even if they produce massive AllReduce bursts at exactly the wrong moment.