Why radix dominates every other spec

The single most important number when selecting a switch for an AI fabric is port count at the required speed. As the fat-tree math shows, doubling radix from 32 to 64 ports increases cluster capacity 8x (k^3 scales with the cube of radix). A 32-port switch requires 8 stages to build a 65K-server cluster; a 64-port switch requires 3 stages. Fewer stages = fewer hops = lower AllReduce latency = better JCT.

Current options in the AI fabric market:

For 800G deployments (next-generation DGX systems), the port count halves but throughput per port doubles -- the cluster-size math stays roughly the same per rack, but cable plant complexity is reduced because fewer ports carry the same total bandwidth.

Real hardware comparison: QM9700 InfiniBand switch (left) vs Spectrum-X SN5600 Ethernet switch (right). Both 1U form factor. Both 400G. Physically similar but fundamentally different protocols and management architectures.

![NVIDIA QM9700 InfiniBand switch and Spectrum-X SN5600 Ethernet switch side by side] Source: NVIDIA networking product datasheet pages Image placeholder: Search "NVIDIA QM9700 datasheet" and "NVIDIA SN5600 datasheet" for official front-panel images of both switch types.

Buffer depth: the hidden performance variable

Two switches with identical port counts and speeds can perform very differently in an AI fabric depending on their buffer depth -- the total SRAM available for queuing in-flight packets.

Deep-buffer switches (Spectrum-X SN5600: ~64 MB shared buffer):

• Can absorb large AllReduce bursts without ECN or PFC firing
• Packets queue in the switch SRAM rather than being held back by PFC
• Higher base latency per packet (serialisation through the buffer)
• Better tolerance for bursty, in-cast traffic patterns
• PFC and ECN are used sparingly -- as intended backup mechanisms
• Trade-off: slightly higher per-packet latency (nanoseconds)

Shallow-buffer switches (various Broadcom Tomahawk-based switches: ~4-8 MB shared buffer):

• Minimal queuing space -- buffers fill and overflow quickly
• PFC must fire almost immediately during any burst
• Very low base latency when the fabric is uncongested
• Require excellent load balancing (sDLB/per-packet spraying) -- without it, buffers overflow before PFC can respond
• Require aggressive ECN thresholds -- must mark early because the buffer fills fast
• Trade-off: fragile under bursty AllReduce traffic without perfect load balancing

The practical implication: A shallow-buffer switch in a poorly load-balanced fabric will drop packets during every AllReduce barrier -- no amount of PFC tuning will save it because the buffer overflows faster than PFC can respond. A deep-buffer switch in the same fabric will absorb the burst, let PFC handle the overflow, and protect losslessness. The choice of switch buffer depth directly determines how much operational margin you have in your congestion management configuration.

Matching switch selection to traffic pattern

The AllReduce burst calculation tells you the minimum buffer depth you need:

Burst size = (number of nodes per rail) x (link speed) x (AllReduce synchronisation window)

Example: 32-node cluster, 400G links, 1ms synchronisation window
  Burst = 32 x 400 Gb/s x 0.001s = 16 Gb = 2 GB

Minimum buffer per port to absorb this burst:
  2 GB / 32 ports (paths the burst can spread across) ~= 64 MB per leaf switch

A switch with 64 MB total shared buffer can absorb this burst.
A switch with 8 MB total shared buffer will overflow after:
  8 MB / (400 Gb/s / 8) = 0.16ms -- well before the 1ms synchronisation window ends.

This calculation is the quantitative argument for deep-buffer switches in large AI training clusters. Shallow-buffer switches are better suited for inference workloads -- many small requests, low burst amplitude, latency-per-request is the critical metric rather than burst absorption.