Skip to content

Cluster Design and the End-to-End Data Path · Part 6 of 7

Act 6 — Tracing Every Path End to End

Building the model…

One RDMA write crossing scalable units: rail-3 leaf, spine, rail-3 leaf. Traffic between same-rail GPUs inside one scalable unit never leaves its leaf.

The design is only credible if you can follow a packet through it. Six paths cover essentially everything a cluster does.

Collective traffic (GPU to GPU). Node NIC → rail leaf → (spine, only if the peer is outside the leaf's group) → rail leaf → peer node NIC. It never leaves the compute fabric, never touches a gateway, never sees a firewall, and is confined to the tenant's partition. If a collective is observed crossing anything else, that is a defect, not a design.

Tenant egress to the internet. Server bond → frontend leaf anycast gateway on the tenant's virtual network → VXLAN to the gateway node's anycast address → source NAT into that tenant's public pool → default route → equal-cost across both border leaves → edge firewall inspection → best transit path.

Inbound to a tenant service address. Transit → edge firewall policy and intrusion prevention → the service /32 learned by BGP, originated by the gateway nodes, reached by equal-cost across the border pair → destination NAT at the gateway → VXLAN into the tenant's virtual network → server.

Load-balanced inference endpoint. As above, with the gateway's layer-4 balancer spreading across backend pool members inside the tenant's VPC. This is the path most likely to need application-aware policy at the edge, and the one where inspection throughput — not interface speed — is the number that matters.

Storage and dataset ingest. In a converged design this rides the same north-south path. Large public dataset pulls are worth pinning to a routed public virtual network that bypasses NAT entirely, which keeps the gateway's state tables lean and stops a bulk transfer from consuming session capacity that tenant traffic needs.

Operator management to the internet. Management switch → border leaf management VRF → a dedicated firewall policy in a separate management context → out. Tenant VRFs cannot route to the management VRF: no leaking is configured and the edge policy is default-deny between them.

END TO END
Follow the packet, hop by hop
01server bond
02frontend leaf anycast GW
03VXLAN to gateway anycast
04source NAT to tenant pool
05ECMP across border pair
06edge firewall
07transit
The gateway owns tenant address mapping; the firewall sees already-public sources and does policy only. No double NAT, clean flow records on both sides.

Failure domains, stated explicitly. An MLAG pair is one maintenance domain — upgrade one member at a time, with the automation draining traffic first. A firewall cluster is one policy domain. The compute fabric is one subnet-manager domain, made redundant by subnet-manager HA rather than by duplication. Site-level events — utility power, a WAN duct cut — are out of scope for a single-availability-zone design, and the border tier's free cages are exactly where a second site's interconnect lands on the day that changes.


Read aloudPage 6