Two gates run in parallel here: the nodes have to be individually sound, and the fabric has to be configured the way the reference architecture says.
Node health checks (NHC) should be automated and run as a Slurm prolog for the rest of the cluster's life, not just at bring-up. The list is short and non-negotiable:
- Correct GPU count enumerated, all reporting healthy, no pending ECC errors and no rows pending retirement
- NVLink topology matching the expected pattern — every peer visible, no degraded links
- PCIe link width and speed at spec on every device, GPUs and NICs alike
- Every HCA present, at rate, with the expected device ordering
- Clean
dmesg: no Xid errors, no PCIe AER corrections accumulating, no thermal events - Clock synchronisation within tolerance across the fleet
- Local NVMe present and healthy; shared filesystem mounted and writable
# GPU inventory and error state
$ nvidia-smi --query-gpu=index,name,memory.total,ecc.errors.uncorrected.volatile.total \
--format=csv
$ nvidia-smi topo -m # NVLink peer matrix -- every pair should show NV#
$ nvidia-smi -q | grep -A3 "Retired Pages\|Remapped Rows"
# PCIe negotiated width/speed for GPUs and NICs
$ lspci -vv -s $(lspci | grep -i 'NVIDIA\|Mellanox' | cut -d' ' -f1 | head -1) \
| grep -E "LnkCap:|LnkSta:"
# Whole-fleet pass/fail, one line per failing node
$ pdsh -g compute 'nhc -c /etc/nhc/nhc.conf' 2>&1 | grep -v 'PASS' | dshbak -c
Fabric feature verification is the other half. The reference architecture for the deployed generation specifies a feature state, and "we think it is on" is not evidence. Capture the actual running configuration from every switch and every NIC, diff it against the intended state, and archive the result — it becomes part of the acceptance pack and, later, the thing you diff against when someone asks whether anything changed.
The items that matter for a Spectrum-X Ethernet compute fabric are adaptive routing enabled and in the correct mode, congestion control enabled per RA guidance, RoCE lossless configuration and ECN thresholds consistent end to end, and one MTU across every hop. For a Quantum-X800 InfiniBand fabric the equivalents are the subnet manager running in HA with the correct routing engine, adaptive routing enabled, SHARP configured if it will be used, and partition keys provisioned to match the tenancy model.
# Ethernet compute fabric -- feature state, captured as evidence
leaf-01# nv show router adaptive-routing
leaf-01# nv show qos roce
leaf-01# nv show interface swp1 link | grep -i mtu
leaf-01# nv config show > /evidence/leaf-01-$(date +%F).cfg
# InfiniBand fabric -- SM state, routing engine, and a full fabric sweep
$ ibdiagnet --get_phy_info --pm_pause_time 60 -o /evidence/ibdiagnet-$(date +%F)
$ sminfo
$ ibnetdiscover | grep -c "^Switch" # matches the design count?
Exit criteria for G4 and G5: 100 % of nodes pass NHC with no exceptions granted, and a captured configuration export from every fabric device that matches the intended state item by item.