Firmware is where fleet uniformity is won or lost. Every layer has a version, every version interacts with the others, and the number that matters is not "is it current" but "is it identical everywhere and approved together".

Build the manifest as a single file, checked into the same repository as the cluster configuration, listing an exact version for each of:

• System BIOS and BMC firmware
• GPU VBIOS and the GPU driver
• CUDA runtime and NCCL version, and the container images they arrive in
• SuperNIC / HCA firmware and the OFED or DOCA host stack
• DPU image where BlueField is present
• Switch OS on every Ethernet and InfiniBand switch
• Subnet manager or fabric-manager version
• Cooling controller firmware, if it participates in shutdown logic

Then apply it, then verify it — the verification is the gate, not the application. A single node that missed a NIC flash because its BMC was busy will pass every functional test and fail cross-unit consistency two weeks later at a point where nobody is looking at firmware any more.

# Verify the manifest across the fleet; anything that prints is a deviation
$ pdsh -g compute 'nvidia-smi --query-gpu=driver_version,vbios_version \
      --format=csv,noheader | sort -u' | dshbak -c

$ pdsh -g compute 'for d in /sys/class/infiniband/*; do
      printf "%s %s\n" "$(basename $d)" "$(cat $d/fw_ver)"; done | sort -u' \
  | dshbak -c

# Switch side, Cumulus
$ ansible -i fabric.ini leaf,spine -m shell -a 'nv show system | grep -i version'

# One-line drift check against the frozen manifest
$ ./scripts/verify-manifest.sh manifest/2026-09-frozen.yaml
  gpu.driver      570.172.08   OK  (250/250 nodes)
  nic.firmware    28.44.1036   DRIFT: cn-0173 reports 28.43.2026
  switch.os       5.14.0       OK  (36/36 switches)

Exit criteria for G2: zero drift rows, the manifest committed with a hash, and a written statement of what re-opens the gate.