The scientific codes that made torus topology successful share a structural property: they communicate locally. A node needs data from its immediate neighbours in the simulation mesh, and those neighbours map directly onto the torus neighbours. Hop count is low because the communication pattern has locality.

AllReduce -- the operation that sits at the heart of every distributed training step -- has no locality at all.

In a ring-AllReduce over N GPUs, every GPU must communicate with every other GPU, and the result must be consistent across all of them before the next forward pass can begin. It is not stencil communication. It is a global synchronisation barrier that touches every participant.

On a torus, this has a specific and painful consequence: distant nodes have high hop count. If you have a 16x16 2D torus (256 GPUs), the worst-case hop count between any two nodes is 8+8 = 16 hops. On a fat-tree BasePOD with 256 GPUs, the worst case is 4 hops (leaf -> spine -> spine -> leaf in a 3-stage tree).

More hops means more latency. More latency in a barrier operation means longer step time. In a 256-GPU training run doing AllReduce 100 times per second, the extra latency compounds rapidly.

But hop count is only the first problem. The second is bisection bandwidth.

A torus has a bisection cut equal to 2 x K^(d-1) x bandwidth-per-link, where K is the nodes per dimension and d is the number of dimensions. In a 16x16 2D torus with 8 GB/s per link, the bisection bandwidth is 2 x 16 x 8 = 256 GB/s for the entire 256-node cluster.

A fat-tree with full bisection provides bisection bandwidth equal to half the total compute bandwidth. For 256 DGX H100 nodes each with 3.2 Tbps of compute fabric bandwidth, a full-bisection fat-tree provides approximately 409 Tbps across the bisection.

The torus bisection and the fat-tree bisection are not even the same order of magnitude. For an all-to-all workload, this is the whole game.

The third problem is fault tolerance and routing. Torus routing is deterministic: given source and destination, there are a small number of shortest paths, and they all pass through the same intermediate nodes. A single failed link in the wrong place can strand a subset of nodes with no alternate route. Fat-tree's ECMP and adaptive routing find alternate paths through any leaf or spine switch -- the topology is designed for redundancy.