Rail-optimised fat-tree design is arithmetic, not art. Given a node count and a switch radix, there is usually exactly one clean answer, and finding it is a matter of writing down four numbers.
Start from the endpoint count. 72 nodes × 8 GPUs = 576 compute endpoints at 800 Gb/s.
Apply rail optimisation. GPU and NIC n of every node connects to leaf switch n. With eight rails you have eight leaf switches, each terminating exactly one port from every node — 72 host-facing ports per leaf.
Size the leaf uplinks for non-blocking. A 1:1 leaf needs as many uplinks as downlinks: 72 up, 72 down, 144 ports per leaf. On a 144-port 800 Gb/s switch — the radix of the current-generation InfiniBand director-class platform used for this role — that is exactly full utilisation with nothing stranded.
Divide the uplinks among spines. 72 uplinks ÷ 4 spines = 18 × 800 Gb/s from each leaf to each spine. Each spine then terminates 8 leaves × 18 = 144 ports, which is again exactly full radix.
| Tier | Switches | Ports used per switch | Links |
|---|---|---|---|
| Leaf (rail) | 8 | 72 down + 72 up = 144 | 576 host + 576 uplink |
| Spine | 4 | 144 down | (the same 576 uplinks) |
Twelve switches, 1:1 non-blocking, no stranded ports. This is what a design looks like when the node count and the radix are chosen together, and it is worth noticing how narrow the window is: 74 nodes instead of 72 does not cost you two more ports, it costs you the entire clean topology, because the leaf can no longer terminate one port per node and still have symmetric uplinks. Reconciling the node count against the radix is a day-one decision, not a detail.
Fabric services on top of the wiring. Subnet management runs in a high-availability pair on the out-of-band network, so a subnet manager failure never depends on the fabric it manages. Adaptive routing is on fabric-wide, and in-network reduction (SHARP-class offload) is enabled if the workload profile justifies it. Tenant isolation is expressed as partition keys, provisioned from the same automation object that creates the tenant's virtual networks on the Ethernet side — one tenant, one VPC, one partition, created in a single operation.
Cabling arithmetic is part of the design, not a follow-up. Host-side links are single 800 Gb/s transceivers or active copper into the leaf. Leaf-to-spine links are typically twin-port modules that carry two 800 Gb/s channels in one cage, which collapses 72 uplink ports into 36 physical cages per leaf. Get this wrong and the port budget that looked exact on paper does not physically fit in the chassis.
What deliberately stays off this fabric: no Ethernet gateways, no storage, no tenant north-south. Everything that is not a collective rides the frontend. This keeps the compute fabric a pure, schedulable interconnect and keeps its failure domain independent of anything a tenant can influence.