The OOB management network is high-value. If an attacker reaches it, they have access to power control, firmware update mechanisms, and console access for every node in the cluster. A compromised BMC is a compromised server. Getting the security posture of the OOB network right is not optional.

Physical segmentation -- the first line

The OOB management switch is physically separate from compute and storage switches. Its uplinks go to the management workstation or jump server, not to the production network. There is no route between the OOB 10.0.x.x subnet and the compute 10.10.x.x or storage 10.20.x.x subnets. This is a routing policy decision -- enforce it at the L3 boundary.

If you are using a jump server model (engineers SSH into a bastion host, then SSH to BMC addresses from there), the jump server is the only host with access to the OOB network. The jump server itself should be hardened, MFA-protected, and session-logged.

BMC authentication hardening

DGX BMC factory default credentials are admin / admin and must be changed before the cluster enters production. Unlike some server vendors, NVIDIA DGX does not print per-unit BMC credentials on the service label -- the factory default is uniform across all units and is hardened at first-boot provisioning time:

# IPMI: change BMC password
ipmitool -I lanplus -H 10.0.1.10 -U admin -P oldpass \
  user set password 2 newstrongpassword

# Create a read-only monitoring user (for Prometheus IPMI exporter)
ipmitool -I lanplus -H 10.0.1.10 -U admin -P pass user set name 3 monitor
ipmitool -I lanplus -H 10.0.1.10 -U admin -P pass user set password 3 monitorpass
ipmitool -I lanplus -H 10.0.1.10 -U admin -P pass user priv 3 2  # 2 = USER privilege
ipmitool -I lanplus -H 10.0.1.10 -U admin -P pass user enable 3

# Disable cipher suite 0 (unauthenticated -- a known IPMI vulnerability)
ipmitool -I lanplus -H 10.0.1.10 -U admin -P pass lan set 1 cipher_privs .xxxxxxxxxxxxxxx

IPMI 2.0 with RAKP is the minimum. Disable IPMI 1.5 sessions if the BMC supports it. Prefer Redfish over IPMI where tooling permits -- Redfish uses TLS 1.2+ and standard certificate management rather than IPMI's custom session negotiation.

Switch management hardening

On ONYX and Cumulus Linux switches, the management interface should listen for SSH only:

# Cumulus Linux (NVUE): restrict management plane access
nv set system ssh-server state enabled
# Disable Telnet if somehow enabled
nv set system telnet state disabled
# Restrict SSH access to the management VRF only
nv set service ssh-server default vrf mgmt
nv config apply

# Set a strong SSH host key (if default keys need rotation)
ssh-keygen -A  # regenerate all host keys
systemctl restart ssh

Separating BMC traffic from compute host traffic

This is the most important security boundary in the OOB design. The BMC should not be reachable from the compute fabric. The compute CX7 NICs on each DGX (mlx5_0 to mlx5_7) should have no routes to the 10.0.1.x BMC subnet. The only path to the BMC is through the physical OOB switch.

You enforce this with routing policy -- no route to 10.0.1.0/24 in the DGX host's main routing table, and no path from the compute leaf switches to the OOB management switch. If a DGX host OS is compromised, the attacker should not be able to reach the BMC and escalate to hardware control.