The instinct on delivery day is to power on the GPUs. Resist it. The out-of-band network is the only path that still works when the production fabric is misconfigured, the OS will not boot, or a firmware flash goes badly — and it is the path you will use for literally every remaining step. Build it first, and build it so that it does not depend on anything you have not yet commissioned.
An OOB design for a GPU cluster carries more than BMCs. Plan the port count deliberately:
| Endpoint class | Count for a 72-node build |
|---|---|
| Compute node BMCs | 72 |
| Ethernet fabric switch management | 4 |
| InfiniBand switch management | 12 |
| Firewall and gateway node management | 6 |
| Cluster controller / provisioning nodes | 3 – 6 |
| Subnet manager (UFM) nodes | 2 |
| PDUs, terminal servers, environmental sensors, CDU controllers | ~12 – 15 |
That lands around 115 access ports before anyone asks for something unplanned, which is why the standard answer is six 48-port 1G management switches with 100G uplinks — capacity and failure-domain distribution, in that order. Six switches do not make a single-ported BMC redundant; they make sure that losing one switch costs you twelve nodes' management access rather than seventy-two.
Two design points are worth insisting on. Keep management in its own VRF with no route leaking to tenant space, because a BMC is a full out-of-band computer with power control over the node, and it should be reachable from exactly one place. And retain a console path that does not traverse the production fabric — a terminal server on its own access route. If your OOB switches uplink through the same border leaves that carry tenant traffic, then a border failure has just taken away both the service and the tool you would use to fix it.
# Discovery: sweep the management subnet and inventory what answers Redfish
$ for ip in $(seq -f "10.10.20.%g" 11 82); do
curl -sk -u "$BMC_USER:$BMC_PASS" \
"https://$ip/redfish/v1/Systems/System.Embedded.1" \
| jq -r '[.SerialNumber, .BiosVersion, .Status.Health,
(.ProcessorSummary.Count|tostring),
(.MemorySummary.TotalSystemMemoryGiB|tostring)] | @tsv'
done | tee inventory/bmc-sweep.tsv
# What "good" looks like for one node
CN4J8P3 2.14.3 OK 2 2304
# Power state and boot device, without touching the OS
$ curl -sk -u "$BMC_USER:$BMC_PASS" \
https://10.10.20.31/redfish/v1/Systems/System.Embedded.1 \
| jq '{PowerState, BootSourceOverrideTarget: .Boot.BootSourceOverrideTarget}'
Exit criteria for G1: every expected BMC answers, serial numbers reconcile against the delivery manifest and the rack elevation, DNS or a static map resolves node names to management addresses, and one person can reach a serial console on any device with the production fabric administratively down.