The last gate before measurement is the one people rush, and it is the one that produces the most confusing failures — because a scheduler misconfiguration looks exactly like a fabric problem when it hits a collective.

Align Slurm partitions or topology labels to physical scalable-unit boundaries. Validation runs "one SU at a time" so that intra-unit traffic never crosses the spine; if a partition spans two leaf groups because it was built from a hostname range rather than the wiring, then a supposedly leaf-local test is quietly measuring spine behaviour, and the result is both wrong and unexplainable. Build the node lists from the topology map, then have a second person check them against the rail audit output.

Pin the container runtime. Enroot and Pyxis need consistent configuration on every node — a writable rootfs, root remapping, and the site's hook set installed — and the environment the containers inherit must carry the fabric settings. On a multi-rail node, the single most common cause of a multi-node collective failing or silently selecting the wrong interface is a missing HCA list:

# /etc/enroot/enroot.conf  (identical on every compute node)
ENROOT_ROOTFS_WRITABLE  yes
ENROOT_REMAP_ROOT       yes

# /etc/enroot/environ.d/50-fabric.env -- verify device names with ibdev2netdev
NCCL_IB_HCA=mlx5_0,mlx5_1,mlx5_2,mlx5_3,mlx5_4,mlx5_5,mlx5_6,mlx5_7

# Confirm PMIx is available -- the benchmark launchers depend on it
$ srun --mpi=list
MPI plugin types are...
   pmi2
   pmix

# Smoke test the whole stack: scheduler, container, GPUs, fabric, one node
$ srun -p gpu -N 1 --ntasks-per-node=8 --gpus-per-node=8 --mpi=pmix \
    --container-image=./images/nccl-tests.sqsh \
    all_reduce_perf -b 512M -e 512M -f 2 -g 1

Record every non-default fabric environment variable in the manifest. An untracked tuning flag is the reason two teams cannot reproduce each other's numbers, and it is also the reason a result stops being reproducible six months later when nobody remembers what was set.

Exit criteria for G6: partitions match physical scalable units, a single-node containerised collective completes, and the environment applied to benchmark jobs is written down and version-controlled.