Skip to content

Chapter 16: The GPU Compute Network -- Packet Anatomy End to End · Part 6 of 12

Act 5 -- Transit: the spine switch

The frame arrives at SpineC's downlink port (swp12). The spine's role is simpler than the leaf's -- it has no server-facing L2 ports, no ARP entries for GPU servers. Every port on a spine is an L3 routed port.

Spine switch -- pure L3 transit forwarding
The spine has no MAC table entries for GPU servers. Every port is a routed interface. Toggle protocol to see how the routing table differs.
eBGP routing table
# show ip route 10.2.1.1
B  10.2.1.0/24 [20/0] via 10.0.2.1 (Leaf4 loopback)
   ECMP:
     * swp1 (via Leaf4 downlink-1)
     * swp2 (via Leaf4 downlink-2)
     * swp3 (via Leaf4 downlink-3)
     * swp4 (via Leaf4 downlink-4)
BGP distance 20. Next-hop = Leaf4 loopback (BGP next-hop). Spine only knows Leaf4's subnet -- not individual GPU /32s (in most configs).
Ethernet rewrite at spine egress
Ingress frame (from Leaf1)
dst MAC: SpineC swp12
src MAC: Leaf1 swp36
IP: 10.1.1.1 -> 10.2.1.1
TTL: 63
Egress frame (toward Leaf4)
dst MAC: Leaf4 swp36 <- rewritten
src MAC: SpineC swp8 <- rewritten
IP: 10.1.1.1 -> 10.2.1.1 (unchanged)
TTL: 62 <- decremented

What the spine's routing table contains -- and what it does not

The spine does not have per-GPU-IP routes in most eBGP deployments. It has per-leaf-subnet routes:

# On SpineC, show ip route
10.1.0.0/24  via 10.0.1.1 (Leaf1 loopback)   # Leaf1's server subnet
10.2.0.0/24  via 10.0.2.1 (Leaf4 loopback)   # Leaf4's server subnet -- destination
10.3.0.0/24  via 10.0.3.1 (Leaf3 loopback)   # Leaf3's server subnet
...

Or in eBGP with per-host routes (eBGP redistributing connected interfaces):

10.2.1.1/32  via 10.0.2.1 (Leaf4)   # GPU's /32 host route

Either way, the lookup resolves to a next-hop that is a leaf loopback IP, reachable via an ECMP group of downlink ports toward Leaf4.

ECMP at the spine: a second independent hash

The spine computes another ECMP hash to choose which of its downlinks to Leaf4 to use. This hash uses the same fields as the leaf's hash -- 5-tuple + optional BTH QPair. Because the hash function is deterministic and both switches see the same packet, the spine will consistently forward the same flow toward the same downlink. This consistency is important: if per-packet spraying (RSHP) is not enabled, all packets of one RDMA Write go via the same spine downlink, ensuring in-order delivery at the destination.

With per-packet RSHP enabled on the spine, each packet takes the best available downlink independently. The destination ConnectX-7 then reorders packets in its reorder buffer before DMAing the payload to GPU HBM.

Second Ethernet rewrite at the spine

Before (arriving from Leaf1):
  Src MAC: aa:bb:cc:44:55:66  (Leaf1 swp36 port MAC)
  Dst MAC: aa:bb:cc:77:88:99  (SpineC swp12 port MAC)

After (departing toward Leaf4):
  Src MAC: aa:bb:cc:ff:00:11  (SpineC swp8 port MAC -- downlink to Leaf4)
  Dst MAC: aa:bb:cc:22:33:44  (Leaf4 swp36 port MAC -- SpineC-facing uplink)

TTL decremented from 63 to 62.

DLB/GLB operation at the spine

If Dynamic Load Balancing is active, the spine's ASIC monitors the byte count on each downlink in real time. If downlink swp8 toward Leaf4 is at 90% utilisation and swp9 is at 40%, new flows are redirected to swp9. With GLB (Global Load Balancing), SpineC also receives heartbeat quality updates from Leaf4's uplinks -- it can see that Leaf4's uplink toward Leaf2 is congested and avoid routing via Leaf4 altogether, selecting a less-loaded spine instead.

Capture D is taken at the spine ingress. The key observation: the IP header is unchanged from Capture A -- same src IP, dst IP, DSCP 46. Only the MACs and TTL have changed.

Capture D -- SpineC transit (ingress from Leaf1)
SpineC mirror port -- downlink side of swp12 (facing Leaf1)
Frame arriving at the spine transit switch. Two MAC rewrites have occurred (at source NIC, at Leaf1). TTL is 63. The spine will perform another IP lookup and MAC rewrite before forwarding to Leaf4.
CHANGED
PRESERVED
KEY FIELD
Wireshark -- Capture D
>Destinationaa:bb:cc:77:88:99aa bb cc 77 88 99PRESERVED
>Sourceaa:bb:cc:44:55:66aa bb cc 44 55 66PRESERVED