The Ethernet side of the cluster is two tiers of the same switch, doing two different jobs.
The frontend leaf pair attaches the servers. Each node's dual-port 200 GbE adapter lands one port on each of two MLAG-paired leaves, bonded with LACP so the pair is active/active — up to 400 Gb/s per server aggregate across flows, with any single flow limited to one 200 Gb/s member. On a 64-cage 800 Gb/s switch, 72 server links at 200 Gb/s consume 18 cages in four-way breakout:
| Purpose | Cages | Mode | Logical ports |
|---|---|---|---|
| Server downlinks | 18 | 4 × 200G breakout | 72 × 200G |
| MLAG peer link | 2 | 800G | 2 × 800G |
| Uplinks to border | 4 | 800G | 4 × 800G (two to each border leaf) |
| Used / free | 24 / 40 |
The oversubscription number, and why it is fine. 72 servers × 400 Gb/s = 28.8 Tb/s of host-facing capacity against 6.4 Tb/s of uplink — 4.5:1 north-south. In an enterprise fabric that would be aggressive. Here it is healthy, because east-west collective traffic — the demanding, synchronous, latency-critical traffic — does not use this fabric at all. What crosses these uplinks is provisioning, control plane, telemetry, tenant internet traffic and (in a converged design) storage. If storage or tenant north-south grows, the ratio is a purchasing decision, not a redesign: every additional pair of 800 Gb/s uplinks per leaf moves it, and there are forty free cages to take them.
The overlay. EVPN-VXLAN with an anycast gateway on both leaves and a shared anycast VTEP address per MLAG pair. Tenant virtual networks stretch across the pair; the underlay is eBGP unnumbered on every fabric link with a per-switch private ASN and BFD on every session. The automation platform provisions VNIs and virtual networks per tenant — nothing in the overlay should be hand-configured, because hand-configured overlay state is how two tenants end up sharing a VNI.
The border leaf pair is the service edge. It is the only tier that talks to the security edge, the gateway nodes, and the management network, and it exists so that the frontend leaves never have to.
| Neighbour | Links per border leaf |
|---|---|
| Frontend leaf pair | 2 × 800G to each = 4 × 800G |
| MLAG peer | 2 × 800G |
| Gateway / NAT service nodes | 2 × 400G (one cage, broken out) |
| Edge firewalls | 2 × 400G |
| Out-of-band switches | one 100G to each management switch |
That is roughly fifteen cages used out of sixty-four. The headroom is deliberate: a second compute pod's frontend pair, a dedicated storage tier, a data-centre interconnect to a second site, or an exchange port all land on the border tier without touching anything else.
One warning about the management uplinks. A 100 GbE management switch's host lanes are NRZ, while a modern 800 Gb/s cage is PAM4. "Break out a 400G module into 100G" is not automatically a supported path between them — the optics have to explicitly provide compatible signalling and gearboxing at both ends. Reserve the cages in the budget, qualify the exact optical pair with the vendor, and do not let a plausible-sounding substitution into the bill of materials unqualified. This is the single most common late-stage surprise in a design like this.