Skip to content

Chapter 18: OOB and Management Network · Part 3 of 11

Act 2 -- Inside the BMC: AST2600 Architecture

Jump to section

1. Introduction2. The Three-Fabric Model Completed3. Inside the BMC: AST2600 Architecture4. IPMI: How It Works at the Protocol Level5. Redfish: IPMI's Successor6. OOB Switch Topology and IP Design7. Switch Management Port vs Data Ports8. UFM Management Plane: How It Reaches Switches9. DGX B200 and BlueField-3: A Different Management Architecture10. Security: Segmenting the Management Plane11. What you now know

Every DGX H100 has a Baseboard Management Controller. It is not running on the host CPU. It is a completely separate SoC (system-on-chip) soldered to the motherboard: the ASPEED AST2600. Understanding this chip is essential for understanding what IPMI and Redfish can actually do.

BMC Architecture
ASPEED AST2600 -- Inside the Baseboard Management Controller
ASPEED AST2600 SoCDual ARM A71.2 GHz, BMC CPUDDR4 RAM512 MB - 1 GBSPI NOR Flash128 MB firmware1GbE PHYOOB NIC2D VGA CtrlKVM-over-IPUSB 2.0 CtrlVirtual mediaHost ServerI2C / SMBusSensor pollingLPC BusPOST capturePCIe vNICIn-band pathOOB Switch (SN2201)Standby power -- always on
click any component to learn what it does

The AST2600 is a dual-core ARM Cortex-A7 running at 1.2 GHz. It has its own DDR4 DRAM (typically 512 MB to 1 GB), its own flash storage (128 MB SPI NOR for the BMC firmware), and its own dedicated 1GbE PHY that connects to the OOB management switch. It boots independently from the host server -- the BMC is powered by the server's standby power rail, which means it is alive as long as the server has AC mains power, even if the host is powered off.

This has a critical operational implication: you can SSH into the BMC while the host is completely off and power-cycle it, read its power consumption, check hardware sensor values, and stream the virtual console output -- all without the host OS being involved at all.

The AST2600 also has a direct connection to the host server's data planes:

  • I2C/SMBus buses: connects to every power regulator, fan controller, temperature sensor, and voltage monitor on the motherboard. The BMC polls these continuously and logs readings with timestamps. This is how IPMI sensor data is generated.
  • LPC bus (Low Pin Count): the legacy path between the BMC and the host CPU. This is how the BMC intercepts the host POST output and presents it on the virtual console.
  • PCIe BMC virtual NIC: in newer platforms, the BMC exposes a virtual NIC to the host over PCIe. Host-to-BMC communication uses this path for in-band management (as opposed to the dedicated 1GbE OOB port for out-of-band).
  • VGA controller: the AST2600 has a built-in 2D graphics controller that intercepts the host server's video output and streams it over the network as the KVM-over-IP console. The host never knows it is being watched.
  • USB 2.0 controller: used for virtual media -- mounting an ISO image from a management workstation as a virtual USB drive on the host. This is how you provision OS images without physically touching the server.

The BMC runs a custom Linux distribution (typically Yocto-based) provided by NVIDIA for the DGX platform. NVIDIA calls their BMC management stack NVSM (NVIDIA System Management). The operating system on the BMC is separate from the DGX OS (Ubuntu) running on the host CPUs.

27% complete

Previous

The Three-Fabric Model Completed

Next

IPMI: How It Works at the Protocol Level