Why radix dominates every other spec
The single most important number when selecting a switch for an AI fabric is port count at the required speed. As the fat-tree math shows, doubling radix from 32 to 64 ports increases cluster capacity 8x (k^3 scales with the cube of radix). A 32-port switch requires 8 stages to build a 65K-server cluster; a 64-port switch requires 3 stages. Fewer stages = fewer hops = lower AllReduce latency = better JCT.
Current options in the AI fabric market:
| Switch | Port count | Port speed | Total bandwidth | Primary use |
|---|---|---|---|---|
| Spectrum-X SN5600 | 64 | 400G | 25.6 Tb/s | Current AI training standard |
| Spectrum-X SN5400 | 64 | 400G | 25.6 Tb/s | Cost-optimised alternative |
| QM9700 (InfiniBand) | 40 | 400G NDR | 16 Tb/s | IB training clusters |
| SN5800 | 128 | 400G | 51.2 Tb/s | Emerging for larger clusters |
For 800G deployments (next-generation DGX systems), the port count halves but throughput per port doubles -- the cluster-size math stays roughly the same per rack, but cable plant complexity is reduced because fewer ports carry the same total bandwidth.
Real hardware comparison: QM9700 InfiniBand switch (left) vs Spectrum-X SN5600 Ethernet switch (right). Both 1U form factor. Both 400G. Physically similar but fundamentally different protocols and management architectures.
![NVIDIA QM9700 InfiniBand switch and Spectrum-X SN5600 Ethernet switch side by side] Source: NVIDIA networking product datasheet pages Image placeholder: Search "NVIDIA QM9700 datasheet" and "NVIDIA SN5600 datasheet" for official front-panel images of both switch types.
Buffer depth: the hidden performance variable
Two switches with identical port counts and speeds can perform very differently in an AI fabric depending on their buffer depth -- the total SRAM available for queuing in-flight packets.
Deep-buffer switches (Spectrum-X SN5600: ~64 MB shared buffer):
- Can absorb large AllReduce bursts without ECN or PFC firing
- Packets queue in the switch SRAM rather than being held back by PFC
- Higher base latency per packet (serialisation through the buffer)
- Better tolerance for bursty, in-cast traffic patterns
- PFC and ECN are used sparingly -- as intended backup mechanisms
- Trade-off: slightly higher per-packet latency (nanoseconds)
Shallow-buffer switches (various Broadcom Tomahawk-based switches: ~4-8 MB shared buffer):
- Minimal queuing space -- buffers fill and overflow quickly
- PFC must fire almost immediately during any burst
- Very low base latency when the fabric is uncongested
- Require excellent load balancing (sDLB/per-packet spraying) -- without it, buffers overflow before PFC can respond
- Require aggressive ECN thresholds -- must mark early because the buffer fills fast
- Trade-off: fragile under bursty AllReduce traffic without perfect load balancing
The practical implication: A shallow-buffer switch in a poorly load-balanced fabric will drop packets during every AllReduce barrier -- no amount of PFC tuning will save it because the buffer overflows faster than PFC can respond. A deep-buffer switch in the same fabric will absorb the burst, let PFC handle the overflow, and protect losslessness. The choice of switch buffer depth directly determines how much operational margin you have in your congestion management configuration.
Switch buffer depth vs AllReduce burst size
Adjust cluster size and synchronisation window to see if buffers are sufficient
Matching switch selection to traffic pattern
The AllReduce burst calculation tells you the minimum buffer depth you need:
Burst size = (number of nodes per rail) x (link speed) x (AllReduce synchronisation window)
Example: 32-node cluster, 400G links, 1ms synchronisation window
Burst = 32 x 400 Gb/s x 0.001s = 16 Gb = 2 GB
Minimum buffer per port to absorb this burst:
2 GB / 32 ports (paths the burst can spread across) ~= 64 MB per leaf switch
A switch with 64 MB total shared buffer can absorb this burst.
A switch with 8 MB total shared buffer will overflow after:
8 MB / (400 Gb/s / 8) = 0.16ms -- well before the 1ms synchronisation window ends.
This calculation is the quantitative argument for deep-buffer switches in large AI training clusters. Shallow-buffer switches are better suited for inference workloads -- many small requests, low burst amplitude, latency-per-request is the critical metric rather than burst absorption.
Sizing the same tree for HGX B300
The arithmetic above is written for 400 GbE endpoints. An HGX B300 node presents eight ConnectX-8 SuperNIC ports at 800 Gb/s, so every rate in the calculation doubles while every count stays the same. Three practical consequences follow.
Radix matters more than it used to. A switch that offered 128 ports at 400 GbE offers 64 at 800 GbE from the same ASIC. If your leaf sizing assumed the 128-port figure, halve it before you count nodes.
Cage count becomes a real constraint. A full-radix leaf can run out of physical OSFP cages before it runs out of logical ports, which is why leaf-to-spine links are commonly built from twin-port modules carrying two 800 Gb/s channels in one cage. Count cages, not just ports.
The oversubscription conversation does not change. Rail-optimised, 1:1 non-blocking is still the target for the compute fabric, and the ratio arithmetic in this act works identically -- it is the absolute numbers that move. Chapter 31 works a full 72-node, 576-GPU B300 design through from endpoint count to spine port budget.