Fat-tree needs switches. A 32-node DGX BasePOD runs 8 leaf switches and 32 spine switches before you count the GPUs. Every hop through the fabric is a hop through dedicated silicon.
A torus eliminates the switches entirely. Each compute node connects directly to a fixed set of neighbours -- the nodes immediately adjacent to it in every dimension of a grid. The node is the router.
Start with the simplest case: a 1D ring. Eight nodes in a circle, each connected to the node on its left and the node on its right. Maximum path length from any node to any other node is N/2 hops -- in an 8-node ring, that's 4 hops.
Extend to two dimensions: a 2D torus. Now each node has four neighbours -- north, south, east, west -- and the edges wrap around (the rightmost column connects back to the leftmost, the top row connects back to the bottom). An 8x8 grid of nodes has 64 nodes, and the maximum path between any two is 4+4 = 8 hops in the worst case, but typical shortest paths are much shorter.
The topology scales by adding dimensions. A 3D torus (as used in IBM Blue Gene /L and /P, and Cray XT3) adds a Z axis. A 5D torus was used in Blue Gene/Q. Fujitsu's K computer -- which was the world's fastest in 2011 -- used a 6D torus called Tofu.
Three properties made this compelling:
-
No switch cost. The network silicon is integrated into the node or the CPU package. At petascale, eliminating switch ASICs from 100,000+ nodes is a meaningful budget difference.
-
Predictable hop count. In an N-dimensional torus with K nodes per dimension, the worst-case hop count is N x (K/2). For a problem that fits the geometry, the path length is bounded and deterministic.
-
Locality awareness. Scientific codes that use stencil computation -- iterative solvers, fluid dynamics, weather modelling -- communicate almost entirely with nearest neighbours. A torus maps this perfectly: the application's communication pattern matches the hardware topology exactly.