A flat 2D torus has a physical problem. Nodes at opposite ends of a dimension are logically adjacent (the edges wrap), but physically they may be metres apart in a rack. The wrap-around links are long, expensive, and lossy.

The folded torus solves this by reordering the nodes physically without changing the logical connectivity. In a folded 1D ring, you fold the sequence in half: node 0 sits next to node N/2, node 1 sits next to node N/2+1, and so on. Every logical neighbour is now a physical neighbour. The long wrap-around cables disappear; all links become short.

The folded 2D torus applies the same principle in both dimensions. Physically, the rack layout is reordered so that the mesh tiles fold back onto themselves. It looks odd on a floor plan but dramatically reduces cable length and latency variance.

Blue Gene /Q used folding extensively in its rack layout. The Tofu interconnect in Fujitsu's K computer used a three-dimensional folded torus to keep cable lengths uniform across a machine spanning thousands of racks.

This matters because latency variance is performance variance. If some node pairs communicate in 2 hops with short links and other pairs communicate in 2 hops with long cables, the training step time is set by the slowest pair. Folding keeps the worst case tightly bounded.