Once every unit has been measured individually, compare them. This is the highest-yield analysis in the entire validation and it costs nothing beyond arithmetic you already have the data for.

The criterion: the coefficient of variation of peak all-reduce bus bandwidth across all units must be ≤ 3 %, and no single unit may sit more than 5 % below the best unit.

The reasoning is worth internalising. A cluster-wide average is a very effective way to hide a localised defect. One unit running 6 % slow because of a marginal optic, a crossed rail, or a leaf switch that missed a configuration change will move a 250-node average by a fraction of a percent — well inside anything you would notice. But that unit is a real place where real jobs will land, and every training step scheduled there pays the 6 % for the life of the cluster. Worse, because the scheduler moves jobs around, the symptom is not "this unit is slow"; it is "sometimes the job is slow", which is a far harder ticket to close.

# Consistency, straight from the parsed results
$ python3 - <<'PY'
import csv, statistics as st
peaks = {}
for r in csv.DictReader(open('csv/su_collectives.csv')):
    if r['collective'] == 'all_reduce_perf':
        peaks[r['su']] = max(peaks.get(r['su'], 0), float(r['peak_busbw_gbps']))
vals = list(peaks.values()); best = max(vals)
cv = st.pstdev(vals) / st.mean(vals) * 100
print(f"units={len(vals)}  mean={st.mean(vals):.1f}  CV={cv:.2f}%")
for su, v in sorted(peaks.items(), key=lambda kv: kv[1]):
    flag = "  <-- OUTLIER" if (best - v) / best * 100 > 5 else ""
    print(f"  {su}  {v:8.1f} GB/s  {(v/best)*100:5.1f}% of best{flag}")
PY

units=4  mean=412.8  CV=3.91%
  su03     388.2 GB/s   91.4% of best  <-- OUTLIER
  su01     412.0 GB/s   97.0% of best
  su04     426.9 GB/s  100.5% of best
  su02     424.1 GB/s   99.8% of best

An outlier is not a tuning problem. Go back to that unit's link qualification CSV, its optic temperatures, its rail map, and its switch configuration diff — in that order. In practice the cause is a physical-layer defect roughly four times out of five.

Gate 1 passes when every unit meets every row of its own criteria and the consistency criterion holds. Only then is it worth crossing the spine.