Now cross the spine on purpose, and do it as a ladder rather than a single jump to full scale. Doubling unit count at each step produces a scaling curve, and the curve's shape is more informative than any of its individual points.
# Scaling ladder -- one unit, two, four, ... to the full cluster
$ for N in 64 128 192 250; do
srun -N $N --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
./build/all_reduce_perf -b 8 -e 16G -f 2 -g 1 -n 20 -w 5 \
| tee results/ladder_${N}n.log
done
# Spine stress: full-cluster all-to-all
$ srun -N 250 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
./build/alltoall_perf -b 1M -e 8G -f 2 -g 1 -n 20 -w 5
# Interference: two jobs, disjoint unit sets, launched together
$ srun -p su-group-A ... ./build/all_reduce_perf -b 8 -e 16G -f 2 -g 1 -n 20 -w 5 &
$ srun -p su-group-B ... ./build/all_reduce_perf -b 8 -e 16G -f 2 -g 1 -n 20 -w 5 &
| Cluster-wide test | Criterion |
|---|---|
| Scaled all-reduce at each ladder step | ≥ 90 % of reference at that scale; degradation gradual and reference-shaped |
| Full-cluster all-to-all | ≥ 85 % of reference; adaptive routing spreads load — no spine port above ~1.15× mean utilisation; zero PFC storms |
| Multi-job interference | Each concurrent job within 5 % of its isolated result |
| Link-failure behaviour (recommended) | Drain one spine link mid-run: routing recovers, the collective completes, post-recovery bandwidth within 3 % of nominal |
| Cluster endurance (2 h mixed loop) | ≤ 2 % drift, no flaps, telemetry envelope unchanged |
The interference test is the one with commercial weight. A multi-tenant GPU cloud is selling the promise that one customer's job does not degrade another's. Two concurrent collectives on disjoint unit sets, each landing within 5 % of its isolated result, is that promise measured. If the number is worse, the finding is not "the fabric is slow" — it is that congestion control or traffic isolation is not doing its job, and it is worth far more to know that during acceptance than after the second tenant arrives.
Spine utilisation spread is the adaptive-routing verdict. Under a full-cluster all-to-all, pull per-port counters from every spine and compute the distribution. Even load means the routing engine is doing what it claims. A hot port at 1.4× the mean means flows are being pinned — either the feature is off, the mode is wrong, or the endpoint cannot tolerate the reordering that per-packet spreading produces and something has quietly fallen back.
Gate 2 passes when all of the above hold. Training begins here and not before.