Skip to content

Spectrum-X Architecture and the AI Factory Platform · Part 4 of 9

Act 4 — Scalable Unit and BasePOD Architecture

The Scalable Unit (SU) is the fundamental deployment building block of Spectrum-X. Every larger architecture — BasePOD, SuperPOD — is composed of integer multiples of SUs. Standardising on the SU enables factory-style pre-cabling, repeatable NVUE configuration templates, and predictable failure domains.

A standard SU contains: 8 DGX nodes (H100, H200, or B200), 2 SN5600 leaf switches for the training fabric, and 1 SN4600C storage leaf switch for the storage fabric. Each DGX H100 or H200 has 8 ConnectX-7 NICs running at 400 GbE each. This gives a total of 64 NIC ports per SU that must be cabled to the 2 leaf switches (32 ports each — half the 64-port SN5600 capacity).

The rail-optimised wiring pattern is what makes the SU's failure domain properties work. NIC 0 from every DGX connects to leaf-01. NIC 1 from every DGX also connects to leaf-01. NIC 2 and NIC 3 from every DGX connect to leaf-02. NIC 4 and NIC 5 connect to leaf-01. NIC 6 and NIC 7 connect to leaf-02. The pattern interleaves 4 NICs from each DGX across each leaf switch. This ensures that a single leaf switch failure removes 4 of 8 NICs from each node simultaneously — all nodes degrade equally, rather than some nodes losing all connectivity. This graceful degradation is a conscious design choice: a 50%-degraded cluster can still run training jobs at reduced efficiency.

BasePOD aggregates 4 Scalable Units: 32 DGX nodes, 8 SN5600 leaf switches, 4 SN4600C storage switches. The leaf switches connect upward to a spine layer — typically 4 SN5600 spines for a BasePOD, providing non-blocking full-mesh connectivity between all leaf pairs. Total active port count in a BasePOD training fabric: 256 DGX NIC ports downlinks + 32 spine uplinks per leaf × 8 leaves. The 2:1 oversubscription ratio at the spine is acceptable because not all 32 nodes in a BasePOD communicate simultaneously at 100% utilisation — ring-AllReduce traffic patterns locality means most collective bandwidth stays within a single SU.

Physical cabling constraints shape deployment decisions. Within a single rack, passive copper QSFP112-DD cables reach 2 metres at 400 GbE with full signal integrity. Cross-rack connections within the same row require active copper cables (ACC) up to 5 metres, or short-reach QSFP-DD 400G SR optics for fibre runs. Spine uplinks crossing multiple rows require multimode OM4 fibre with 400G-SR8 optics (OSFP breakout). A BasePOD with 4 SUs spread across 4 racks generates approximately 288 cross-rack fibre cables. Pre-tested fibre trunk assemblies — pre-terminated MPO-12 to LC breakout — are standard practice to avoid field-termination errors.

INTERACTIVE
Spectrum-X Scalable Unit Topology
Fail switch:
DGX-017/7 NICs activeDGX-027/7 NICs activeDGX-037/7 NICs activeDGX-047/7 NICs activeDGX-057/7 NICs activeDGX-067/7 NICs activeDGX-077/7 NICs activeDGX-087/7 NICs activeleaf-01SN5600NICs 0,1,4,5leaf-02SN5600NICs 2,3,6,7storage-01SN4600CSPINE ↑BasePOD
# Verify rail wiring on leaf-01 — should see all 8 DGX nodes, NICs 0,1,4,5
leaf-01# nv show interface | grep -E "swp[0-9]+" | grep -v "swp3[0-9]"
Interface  State  Speed   MTU   RemoteHost         RemotePort
swp1       up     400G    9216  dgx-01.rail0       ConnectX-7/0
swp2       up     400G    9216  dgx-02.rail0       ConnectX-7/0
swp3       up     400G    9216  dgx-03.rail0       ConnectX-7/0
swp4       up     400G    9216  dgx-04.rail0       ConnectX-7/0
swp5       up     400G    9216  dgx-05.rail0       ConnectX-7/0
swp6       up     400G    9216  dgx-06.rail0       ConnectX-7/0
swp7       up     400G    9216  dgx-07.rail0       ConnectX-7/0
swp8       up     400G    9216  dgx-08.rail0       ConnectX-7/0
# swp9-16: NICs 1 from each DGX (same leaf)
# swp17-32: spine uplinks

# Count total links per leaf
leaf-01# nv show interface brief | grep "^swp" | grep " up " | wc -l
32