A checklist can be worked in any order. A bring-up cannot. Every step in commissioning an AI cluster produces evidence that the next step depends on, and skipping ahead does not save time — it moves a defect from a domain where one person can find it in an hour to a domain where five people argue about it for a week.
The discipline that makes this tractable is the gate: a named point with written exit criteria, evidence attached, and an explicit statement of what it blocks. Gates are how you keep a 250-node build from turning into a diffuse cloud of half-finished work, and they are also how you keep the vendor conversation honest, because a gate that has not passed is not an opinion.
| Gate | What it proves | What it blocks |
|---|---|---|
| G0 — Physical | Racks landed, power live, cooling loop commissioned and balanced, structured cabling in and labelled | Any power-on |
| G1 — Out-of-band | Every BMC discovered, addressed, reachable, and inventoried; console path independent of the production fabric | Any in-band work |
| G2 — Firmware baseline | One frozen manifest applied fleet-wide: BIOS, BMC, GPU VBIOS, NIC firmware, switch OS, DPU image | Node health testing |
| G3 — Physical layer | All compute links up at rate, rail audit clean, 24-hour error soak with zero symbol/CRC growth | Fabric feature work |
| G4 — Node health | 100 % of nodes pass health checks: GPU count and health, NVLink topology, PCIe width and speed, clean logs | Any collective run |
| G5 — Fabric features | Adaptive routing, congestion control, MTU, ECN/PFC verified and captured as config evidence | Acceptance measurement |
| G6 — Scheduler and runtime | Slurm partitions aligned to scalable units, container runtime working, fabric environment pinned | Benchmark execution |
Everything downstream of G6 is the subject of the next chapter. The rest of this one is how each gate is actually passed.
One rule underpins all of them: freeze before you measure. The single most common way a commissioning effort loses a fortnight is that somebody upgrades a driver, a switch image, or a NIC firmware in the middle of the process to fix one node. The fleet is now non-uniform, every comparison across scalable units is invalid, and the only honest recovery is to re-freeze and re-run. Write the manifest down at G2, put it under change control, and treat any change to it as re-opening the gate.