Skip to content

Chapter 7: Topology Design -- From One Node to a SuperPOD

HPC Networking FoundationsIntermediate60 min read

Every chapter so far has shown you a single leaf switch, a handful of DGX nodes, a spine or two. You have learned to read counters on that switch, configure PFC on its ports, diagnose load balancing failures across its uplinks. The fabric has felt local -- something you can touch by SSHing into two or three devices.

Real AI training clusters do not feel that way. A DGX SuperPOD has 32 nodes, 8 leaf switches, and 8 spine switches -- 16 switches total, 256 GPUs, thousands of cables. A hyperscale cluster is 4,000 nodes, hundreds of switches, and a cabling plant that fills multiple data center rows. The operational problems you have been diagnosing on one switch happen simultaneously on all of them.

This chapter zooms out. It shows how the single-leaf topology you have been working with scales into a complete cluster fabric -- and why the design decisions made at the topology level determine whether every chapter's worth of PFC, ECN, and load balancing configuration actually has a chance to work.

By the end you will be able to look at a topology diagram, calculate whether it can support AllReduce at line rate, identify where the bottlenecks will appear, and explain why NVIDIA chose specific port counts and stage configurations for their reference architectures.