Now the fabric proper, one scalable unit at a time, with no traffic crossing the spine. A scalable unit is the reference architecture's building block — a fixed set of GPU nodes plus their leaf switches, sized so that rail-local collectives stay on the leaf tier. Sizes vary by architecture revision; 32-node and 64-node units are both common. Use the size your deployment's RA specifies and make the Slurm partitions match it exactly, or the results describe something other than what you think.
Four collectives, every unit, every time. All-reduce is the headline number but it is not sufficient on its own. All-gather and reduce-scatter dominate the traffic of sharded-optimiser and fully-sharded parallelism, and they must be measured directly rather than inferred. All-to-all is the harshest pattern and the best probe for incast and congestion behaviour.
# Full unit, all GPUs -- 64 nodes x 8 GPUs = 512 GPUs
$ srun -p su01 -N 64 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
./build/all_reduce_perf -b 8 -e 16G -f 2 -g 1 -n 20 -w 5
# Same line for each of:
# all_gather_perf reduce_scatter_perf
# All-to-all sweeps from a larger floor -- small messages are not the point
$ srun -p su01 -N 64 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
./build/alltoall_perf -b 1M -e 8G -f 2 -g 1 -n 20 -w 5
Record the whole curve, not just the peak. The message-size sweep from 8 B to 16 GB is diagnostic in a way the single peak number is not. A healthy fabric produces a curve that rises smoothly and plateaus. A sawtooth, or a collapse in the mid-range, is congestion control misbehaving — the fabric is finding a message size at which its rate-control loop oscillates, and averaging that away into a peak figure throws out the most useful evidence you collected.
Capture switch telemetry over the same window: per-port utilisation, ECN marks, CNP counts, PFC pause counters, buffer occupancy, and adaptive-routing statistics. A pass with heavy pause activity is not really a pass; it is a fabric that is meeting its number by throttling, and it will not hold up under a real workload's mixed traffic.
| Unit-level test | Criterion |
|---|---|
| all_reduce peak bus bandwidth | ≥ 90 % of the platform reference at unit scale; curve monotonic to plateau |
| all_gather / reduce_scatter peak | ≥ 90 % of reference, measured directly |
| all_to_all sustained | ≥ 85 % of reference; zero PFC storm events; ECN behaviour nominal |
| Rail locality | Telemetry confirms traffic stays on the leaf tier; any spine crossing inside a unit is a wiring or routing defect |
| Endurance (30 – 60 min loop) | ≤ 2 % drift; no link flaps; no NIC or optic thermal throttle |
The rail-locality check deserves its own paragraph because it is the one that most often gets skipped. Run an all-reduce on rail-local communicators and watch the spine ports. If anything at all crosses the spine during an intra-unit collective, you have a wiring error the rail audit missed or a routing configuration that is not doing what you think. The performance number may still pass; the defect is real and it will scale badly.