The DGX H100 and H200 use a straightforward management model: the host CPU runs the OS, DCGM, and all management agents. The BMC is a separate controller for power and hardware monitoring only. The storage and management CX7 NICs are pure network adapters.

The DGX B200 changes this. It replaces the two dual-port CX7 storage NICs with two dual-port BlueField-3 DPUs. These are not just NICs -- each BlueField-3 has its own ARM Cortex-A78 CPU complex (sixteen Cortex-A78 cores running at up to 3.0 GHz), its own DDR5 DRAM, and runs its own independent ARM Linux OS.

This matters for management because the BlueField-3's ARM cores are in the data path, not just monitoring it. On the DGX B200:

• NVMe-oF initiator for storage: runs on the BlueField-3 ARM cores, not the host CPU.
• In-band management NIC: the BlueField-3 acts as the network interface for the host's in-band management traffic (SSH, DCGM exporter, monitoring).
• OVS/networking offload: on B200, the BlueField-3 can run Open vSwitch in hardware, offloading switching decisions from the host CPU.

rshim: the host-to-BlueField-3 communication path

The host CPU communicates with the BlueField-3 via rshim -- a PCIe-based communication channel. rshim presents as a set of character devices on the host:

• /dev/rshim0/boot -- write a BFB (BlueField-3 Bootstream Bundle) image here to reflash the BlueField-3 OS.
• /dev/rshim0/console -- character device giving console access to the BlueField-3's ARM Linux.
• /dev/rshim0/misc -- control channel for BF3 operations.

The rshim channel is how you push firmware and OS updates to the BlueField-3 without physical access to the server:

# On the DGX B200 host: push a new BlueField-3 OS image (BFB file)
# Requires rshim module loaded
lsmod | grep rshim

# Push the BFB image (this reflashes the BF3 and reboots it)
bfb-install --rshim /dev/rshim0 --bfb bf3-bfb-24.1-ubuntu22.04.bfb

# Monitor progress via console
minicom -D /dev/rshim0/console

# After reflash, SSH into the BlueField-3's ARM Linux directly
ssh ubuntu@192.168.100.2

The link that has no switch: tmfifo_net0

That 192.168.100.2 is not an OOB address and does not traverse any switch. rshim also presents a virtual Ethernet interface, tmfifo_net0, which tunnels IP over the same PCIe channel. It is a two-host point-to-point link between the x86 host and the Arm cores, and it exists on every BlueField server whether anyone configured it or not.

The DPU side ships preconfigured at 192.168.100.2/30. The host side is not configured out of the box -- you set it to 192.168.100.1/30:

# Ubuntu: /etc/netplan/01-netcfg.yaml
network:
  version: 2
  renderer: networkd
  ethernets:
    tmfifo_net0:
      addresses:
        - 192.168.100.1/30
      dhcp4: false

sudo netplan apply
ssh ubuntu@192.168.100.2

Two things follow from this that matter to whoever owns the address plan.

192.168.100.0/30 is reserved, cluster-wide, on every DPU host. It is a per-host island -- the same four addresses repeat on every node, and they never appear on a wire. But if that prefix is also allocated to something real in the plan, the hosts running DPUs will have a route to two different places called 192.168.100.x, and the failure is intermittent and node-specific. Put it on the reserved list in Chapter 19 and leave it there.

It is a path in when nothing else works. If the DPU's OOB address is wrong, its network configuration is broken, or a BFB upgrade left it unreachable, tmfifo_net0 still works, because it is PCIe rather than Ethernet. This is the DPU's equivalent of a serial console.

Predicting the OOB and BMC MAC addresses

The BlueField-3 exposes two logical endpoints -- the DPU's own OOB interface and its BMC -- through one physical RJ45, joined by an internal switch. Both request DHCP independently. One cable, one switch port, two MAC addresses, two leases. When the OOB switch port shows two MACs, that is correct, not a loop.

You can derive both addresses before the card ever boots, which is what lets you pre-populate DHCP reservations:

sudo flint -d /dev/mst/mt41692_pciconf0 q
# Base MAC:        9C63C03C4CA0
# GuidsNumber:     36

The rule:

• OOB MAC = Base MAC + (GuidsNumber - 1)
• BMC MAC = Base MAC + GuidsNumber

With a base of 9C:63:C0:3C:4C:A0 and 36 GUIDs, that gives an OOB address of 9C:63:C0:3C:4C:C3 and a BMC address of 9C:63:C0:3C:4C:C4. Do the arithmetic against the GuidsNumber your own card reports -- it varies by part -- and the two addresses are deterministic from the label.

Reading a BFB upgrade that stalls

bfb-install streams the boot stages to the console as it goes, and the stage it stops at localises the fault without any further investigation:

One operational note that belongs in the change record rather than the runbook: a BFB upgrade resets the DPU credentials to the image defaults and forces a change on first login. A fleet-wide DPU upgrade therefore ends with every DPU on a default credential until somebody re-runs the hardening step. Plan that into the maintenance window, not after it.

BlueField-3 management interface

The BlueField-3 exposes its own management interface (an out-of-band-equivalent path independent of the host OS). In DGX B200 deployments, each BF3 has an oob_net0 interface that connects to the OOB switch alongside the host BMC. From the management workstation, you can reach the BF3's ARM Linux, inspect its NVMe-oF initiator state, check link counters on the storage ports, and upgrade its firmware -- all without the host DGX OS being involved.

# On the BlueField-3 ARM Linux (after SSH to BF3 IP):

# Check NVMe-oF initiator state on the BF3
nvme list-subsys

# BlueField-3 port counters (storage ports)
mlxlink -d /dev/mst/mt41692_pciconf0 -p 1 --show_counters

# Check BF3 firmware version
mlxfwmanager --query

# Upgrade BF3 NIC firmware (separate from OS/BFB upgrade)
mlxfwmanager --update -y /path/to/fw-ConnectX7-rel-28_40_1000.bin

This is the correct home for BlueField-3 engineering content. Ch0 and Ch1 mention that the DGX H100 does NOT have BF3. Ch10 describes the storage architecture difference. Ch18 is where you learn to actually operate the BF3 management layer.