Every cluster acceptance, however it is dressed up contractually, comes down to two questions asked in a fixed order.
Is the fabric itself sound? Answered with collective benchmarks and low-level point-to-point tests, run first inside a single scalable unit and then deliberately across the spine. This question is about links, optics, wiring, routing behaviour and congestion control. It is answered with NCCL bus bandwidth and perftest numbers.
Does the fabric deliver at workload level? Answered with real training runs at increasing scale, compared against published reference figures for the same recipe, precision and GPU count. This question is about whether the fabric's measured goodness survives contact with a workload that also has to move optimiser state, checkpoint, and read data.
The order is not stylistic. A training benchmark that lands 12 % low tells you almost nothing on its own — the cause could be a marginal optic on one node, a congestion-control setting, a dataloader stall, or a genuinely undersized topology. The same 12 % after a clean collective pass narrows the field to a handful of candidates. Sequencing the questions is what makes a failure diagnosable.
- Link qualification ≥ 95% of line rate on every rail, zero CRC/retransmit
- all_reduce, all_gather, reduce_scatter ≥ 90% of reference
- all_to_all ≥ 85% of reference, no PFC storms
- Cross-unit CV ≤ 3%; no unit more than 5% below the best
The gates that structure the work:
| Gate | Content | Blocks |
|---|---|---|
| 0 | Manifest frozen, fabric feature state verified, node health 100 %, rail audit clean | All measurement |
| 1 | Every scalable unit passes its collective criteria; cross-unit consistency holds | Cluster-wide collectives |
| 2 | Cluster-wide collectives, interference and endurance pass | Any training run |
| 3 | Training benchmarks meet target at unit and full scale, weak scaling holds | Acceptance sign-off |
Two working rules apply throughout, and both are worth agreeing with the customer before anything is measured. Do not tune during acceptance — acceptance measures a configuration, not a search. If a setting must change after Gate 0, the manifest is re-frozen and every affected gate is re-run. And do not compare across versions — a result is only meaningful against the reference published for that exact recipe version, precision, scale and platform generation. Recipe refreshes legitimately move baselines, which is why the version is pinned at kickoff and held for the entire campaign.