Act 5 established the OOB network's topology and addressing. This Act covers what actually happens on it the first time a rack of nodes is powered on -- because before a node has an OS, the management network is the only thing it has.

The course has treated node bring-up as somebody else's problem up to now. It is worth ten minutes of a network engineer's attention, because one of the four ways to do it makes the switch the source of truth for node naming, and the other three quietly break the rail map.

PXE, and what "unidentified" means

A node with no OS powers on, PXE boots, and broadcasts its MAC address on the provisioning network. The cluster manager -- Base Command Manager in an NVIDIA reference design -- is listening, and its job is to answer a single question: which node object does this MAC belong to?

That mapping is called node identification. Until it succeeds the node is unidentified, and an unidentified node is not provisioned. It sits there having broadcast successfully, on a healthy network, doing nothing.

Note what this makes the head node: the DHCP, DNS and directory authority for the provisioning network. That is not a detail of the cluster manager. It is a service on your management VLAN that you did not configure and cannot move, and it needs to be in the address plan as a fixed point.

The four ways to identify a node, and why only one scales

1. From the booting node's console. Watch it boot, tell it what it is. Works. Does not scale past a handful.

2. device newnodes in cmsh. The cluster manager collects the MACs it has heard and offers them for assignment in the order they arrived. Fast, convenient, and carries a trap covered below.

3. Enter every MAC by hand. Correct, auditable, and miserable at rack scale -- though it is what you fall back to when a node's MAC is already known from an asset system.

4. Switch port-based identification. The cluster manager reads the provisioning switch's forwarding table and maps port to node. Port 12 on oob-rack04-sw1 is node-04-12, whatever MAC happens to be behind it.

Method 4 is the one recommended for large clusters, and it is the one that concerns you directly: it makes the switch port map the authoritative source for node identity. Get the port map right and node names correspond to rack positions permanently -- including after a motherboard swap changes the MAC.

The trap in device newnodes

Method 2 assigns identities in the order the MACs arrive. Power a rack on all at once and they arrive in whatever order the nodes finish POST -- which is not rack order, and is not stable between boots.

The result is a cluster where node017 is physically above node019, and nothing in software knows. Everything still works, right up until the first time somebody reasons about locality: which nodes share a leaf switch, which share a rail, which should be scheduled together for a tightly-coupled job. Chapter 7's rail-optimised design assumes node names track physical position. When they do not, the topology in everyone's head is wrong and the fabric is fine.

The stated mitigation is exactly as tedious as it sounds -- power the nodes on one at a time, leaving about a minute between each -- which is a reasonable description of why method 4 exists.

The commands

# Power on an entire node category
cmsh -c "device; power -c cloned on"

# Assign a MAC to a node object, then commit
cmsh
[cluster]% device
[cluster->device]% use node001
[cluster->device[node001]]% set mac 4e:56:44:41:01:01
[cluster->device[node001]]% commit
# Successfully committed 4 Devices

# Reboot a node group without going through the cluster manager
pdsh -g computenode reboot

That last one looks redundant until you meet the case it exists for: after a licence change, already-provisioned nodes must reboot to pick up new certificates, and the cluster manager will not reboot nodes it no longer considers licensed. pdsh is the way out of that particular loop.

Switches are objects too

Worth knowing because it is the mechanism behind method 4: the cluster manager models Ethernet switches, InfiniBand switches, PDUs and rack sensor kits as first-class device objects, alongside nodes. It can poll them. Which means the same system that owns node identity also holds a model of the management fabric -- and that model is only as good as the port map somebody entered into it.