Before any collective runs, qualify the links themselves. This step is cheap, fast, and it is the only one that isolates a single optic to a single port. Once you are running collectives, a bad link is a slow ring and the search space is the whole unit.

The tool is perftest — ib_write_bw for bandwidth, ib_write_lat for latency — run pairwise, rail-aligned, so that every NIC on every node is exercised at least once on every rail. Two rotations (neighbour pairing, then a shift by one) means each node is tested against two different peers per rail, which distinguishes "this node is bad" from "this cable is bad".

# Server side, node A, rail 0
$ ib_write_bw -d mlx5_0 -a --report_gbits

# Client side, node B, same rail, pointed at node A
$ ib_write_bw -d mlx5_0 -a --report_gbits nodeA

# Under Slurm, driven across a unit's node list, one rail at a time
$ srun -p gpu -w cn-0041 -N1 -n1 ib_write_bw -d mlx5_3 -a --report_gbits &
$ sleep 3
$ srun -p gpu -w cn-0042 -N1 -n1 ib_write_bw -d mlx5_3 -a --report_gbits cn-0041

The criterion is ≥ 95 % of line rate at large message sizes, with zero retransmissions and zero CRC errors during the run. On an 800 Gb/s compute link that means roughly 760 Gb/s or better on the average-bandwidth column. A link that returns 700 is not "slightly slow" — it is a defect with a physical cause, and it will be found now with a two-minute test or later with a two-hour argument.

Automate this. Running eight rails across two rotations on a 64-node unit is a few hundred pair tests; a script that emits one CSV row per pair — unit, rotation, rail, server, client, peak Gb/s, percentage of line rate, pass or fail — turns the whole gate into a grep FAIL and makes the result archivable as evidence.

Gate criterion: zero failing links across the entire cluster.