The frame arrives at SpineC's downlink port (swp12). The spine's role is simpler than the leaf's -- it has no server-facing L2 ports, no ARP entries for GPU servers. Every port on a spine is an L3 routed port.
# show ip route 10.2.1.1
B 10.2.1.0/24 [20/0] via 10.0.2.1 (Leaf4 loopback)
ECMP:
* swp1 (via Leaf4 downlink-1)
* swp2 (via Leaf4 downlink-2)
* swp3 (via Leaf4 downlink-3)
* swp4 (via Leaf4 downlink-4)What the spine's routing table contains -- and what it does not
The spine does not have per-GPU-IP routes in most eBGP deployments. It has per-leaf-subnet routes:
# On SpineC, show ip route
10.1.0.0/24 via 10.0.1.1 (Leaf1 loopback) # Leaf1's server subnet
10.2.0.0/24 via 10.0.2.1 (Leaf4 loopback) # Leaf4's server subnet -- destination
10.3.0.0/24 via 10.0.3.1 (Leaf3 loopback) # Leaf3's server subnet
...
Or in eBGP with per-host routes (eBGP redistributing connected interfaces):
10.2.1.1/32 via 10.0.2.1 (Leaf4) # GPU's /32 host route
Either way, the lookup resolves to a next-hop that is a leaf loopback IP, reachable via an ECMP group of downlink ports toward Leaf4.
ECMP at the spine: a second independent hash
The spine computes another ECMP hash to choose which of its downlinks to Leaf4 to use. This hash uses the same fields as the leaf's hash -- 5-tuple + optional BTH QPair. Because the hash function is deterministic and both switches see the same packet, the spine will consistently forward the same flow toward the same downlink. This consistency is important: if per-packet spraying (RSHP) is not enabled, all packets of one RDMA Write go via the same spine downlink, ensuring in-order delivery at the destination.
With per-packet RSHP enabled on the spine, each packet takes the best available downlink independently. The destination ConnectX-7 then reorders packets in its reorder buffer before DMAing the payload to GPU HBM.
Second Ethernet rewrite at the spine
Before (arriving from Leaf1):
Src MAC: aa:bb:cc:44:55:66 (Leaf1 swp36 port MAC)
Dst MAC: aa:bb:cc:77:88:99 (SpineC swp12 port MAC)
After (departing toward Leaf4):
Src MAC: aa:bb:cc:ff:00:11 (SpineC swp8 port MAC -- downlink to Leaf4)
Dst MAC: aa:bb:cc:22:33:44 (Leaf4 swp36 port MAC -- SpineC-facing uplink)
TTL decremented from 63 to 62.
DLB/GLB operation at the spine
If Dynamic Load Balancing is active, the spine's ASIC monitors the byte count on each downlink in real time. If downlink swp8 toward Leaf4 is at 90% utilisation and swp9 is at 40%, new flows are redirected to swp9. With GLB (Global Load Balancing), SpineC also receives heartbeat quality updates from Leaf4's uplinks -- it can see that Leaf4's uplink toward Leaf2 is congested and avoid routing via Leaf4 altogether, selecting a less-loaded spine instead.
Capture D is taken at the spine ingress. The key observation: the IP header is unchanged from Capture A -- same src IP, dst IP, DSCP 46. Only the MACs and TTL have changed.