The frame arrives at the leaf switch on a front-panel port. In a BasePOD, the leaf's front-panel ports connect to DGX nodes; its uplink ports connect to spines. The leaf switch performs the first routing decision.

L2 on the server-facing port, L3 on the uplinks

In a pure-routed Clos AI fabric (the BasePOD reference design), the server-facing ports (swp1-swp32 on a Spectrum-X leaf) are L2 access ports. The uplink ports (swp33-swp64, connecting to spines) are L3 routed ports with no VLAN, no bridging. In an EVPN-VXLAN fabric the server-facing ports are also access ports but carry VLAN tags that identify tenants; the routing happens at the VXLAN overlay level. For this chapter, the pure-routed model is assumed throughout.

When the frame from the ConnectX-7 arrives on swp1:

1. The switch reads the destination MAC. If it matches a locally-connected device's MAC -- stop, deliver locally (intra-leaf traffic).
2. If the destination MAC is the switch's own MAC (i.e., the gateway MAC that the ConnectX-7 ARPed for), this is a routed packet. The switch strips the Ethernet header and performs an IP route lookup.
3. IP route lookup on destination IP (e.g., 10.2.1.1 -- the remote GPU's HCA IP): result is next-hop = one of the spine switches, via one of the uplink ports.
4. ECMP hash computation to select which uplink.

The ECMP hash: what fields, why BTH matters

Standard ECMP hashes the 5-tuple: src IP, dst IP, IP protocol, src UDP port, dst UDP port. This produces reasonable entropy for general traffic. But RoCEv2 AllReduce traffic has a problem: all packets between the same GPU pair use the same src IP, dst IP, and dst UDP port (4791). The only variation is the src UDP port -- which the ConnectX-7 generates from a hash of the QPN, and which is often similar across flows between the same pair.

The fix is to include the RoCEv2 BTH QPair number in the hash. Spectrum-X switches can be configured to parse UDP payload for the BTH header and include the Destination QPN field (bytes 5-7 of the BTH, right after the opcode and partition key). This is configured with:

nv set system forwarding ecmp-hash roce enable

With BTH QPair hashing enabled, two simultaneous AllReduce operations between the same GPU pair on different QPs will hash to different spines -- perfect entropy for AI traffic.

ARP: where it runs and what it produces

When HCA0 on Node A first comes up, it ARPs for its default gateway (the leaf switch's IP on the server-facing subnet). The leaf switch answers with its own MAC. HCA0 caches this: 10.1.0.1 is-at aa:bb:cc:dd:ee:ff. All subsequent packets from HCA0 use aa:bb:cc:dd:ee:ff as the Ethernet destination -- regardless of the IP destination. This is standard L3 routing. The leaf switch's ARP table, conversely, holds 10.1.1.1 is-at 94:6d:ae:xx:xx:xx (the ConnectX-7's IP mapped to its MAC). The leaf uses this when forwarding return traffic back to the GPU.

Frame rewrite at the leaf

After the ECMP hash selects uplink swp36 (connecting to SpineC), the leaf rewrites the Ethernet header:

Before (arriving from GPU):
  Src MAC: 94:6d:ae:aa:bb:cc  (ConnectX-7 on Node A)
  Dst MAC: aa:bb:cc:11:22:33  (Leaf switch swp1 gateway MAC)

After (departing toward SpineC):
  Src MAC: aa:bb:cc:44:55:66  (Leaf switch swp36 port MAC)
  Dst MAC: aa:bb:cc:77:88:99  (SpineC swp12 port MAC, learned from BGP next-hop ARP)

The IP header is unchanged. The TTL is decremented from 64 to 63.

Capture B shows the frame as it arrives on swp1 (before the L3 lookup). Capture C shows the same frame as it leaves on swp36 (after the MAC rewrite and TTL decrement). Compare the two: the IP header and BTH-carrying payload are byte-for-byte identical; only the Ethernet header has changed.

What the leaf's routing table looks like under BGP vs IS-IS

Under eBGP (the standard):

Destination: 10.2.1.1/32 (remote GPU's HCA IP)
  Via: 10.0.0.3 (SpineC loopback, BGP next-hop)
  Protocol: BGP, distance 20
  ECMP: 4 paths (via SpineA, SpineB, SpineC, SpineD)

Under IS-IS Flex Algo 128 (low-latency plane):

Destination: 10.2.1.1/32
  Via: SpineA and SpineB only (SpineC excluded by delay constraint)
  Protocol: IS-IS, Flex Algo 128
  Metric: delay-based (SpineC excluded: delay 8us > 5us threshold)

The packet itself does not change based on protocol -- the routing table lookup produces a next-hop either way. The difference is which spines are eligible. Under Flex Algo 128, the ECMP hash runs over fewer paths, potentially producing different distribution, but the frame that leaves the leaf is identical format regardless of which protocol chose the next-hop.