ProDeploy — Learn the fabric that keeps AI clusters alive

This act applies exclusively to DGX B200 clusters. DGX H100 and H200 use ConnectX-7 HCA — a high-performance network adapter with no embedded CPU. BlueField-3 DPU (BF3) is present only in DGX B200. This distinction matters operationally: security architectures described here cannot be deployed on H100/H200 clusters.

BF3 architecture: independent ARM subsystem

The BlueField-3 contains 16 ARM Cortex-A78AE cores running their own independent Linux instance — the DPU OS — completely separate from the host OS (DGX OS). Between the host CPU and the physical network sits the BF3's embedded switch (eSwitch). Every packet transmitted or received by the host passes through this eSwitch, giving the DPU OS full visibility and control over the host's network traffic.

Critically: the host OS cannot override eSwitch rules set by the DPU OS. A compromise of the host OS — or a misconfigured tenant application — cannot disable the DPU's security policies. The isolation is hardware-enforced.

DGX B200 data path:
  GPU memory → PCIe → BF3 eSwitch (DPU OS rules here) → physical 400GbE → fabric
                           ↑
                    This is the enforcement point.
                    DPU OS programs OVS rules here.
                    Host OS cannot bypass this.

Zero-trust microsegmentation via eSwitch OVS

The DPU OS runs Open vSwitch (OVS) on the eSwitch. OVS rules can classify traffic by source MAC, source IP, VLAN, DSCP, or any L2-L4 field, and apply actions: forward, drop, rate-limit, redirect to a monitoring port.

# On the DPU OS (accessed via: ssh admin@<bf3-oob-ip>):

# Block all traffic from this host to a different tenant's subnet:
ovs-vsctl add-br br-host
ovs-vsctl add-port br-host pf0hpf  # host-facing port
ovs-vsctl add-port br-host p0      # network-facing port

# ACL: allow TenantA traffic (src 10.100.0.0/24), deny cross-tenant:
ovs-ofctl add-flow br-host \
  "priority=100,ip,nw_src=10.100.0.0/24,actions=output:p0"
ovs-ofctl add-flow br-host \
  "priority=50,ip,actions=drop"   # deny everything else

# Verify rules are installed:
ovs-ofctl dump-flows br-host

These rules survive a host OS reboot. They survive a kernel panic on the host. They survive a compromised application that attempts to modify networking. The DPU OS runs independently; the host cannot touch it.

DOCA App Shield (B200 only)

DOCA App Shield is a runtime application monitoring service that runs entirely on the DPU OS. It uses ARM performance counters and memory-mapped views of the host's PCIe transactions to monitor host application behaviour — without any agent running on the host OS.

# On DPU OS: start DOCA App Shield monitoring
doca_app_shield --monitor-pid-range 1-65535 \
                --alert-threshold syscall_rate:10000 \
                --output /var/log/app_shield.log &

# View active alerts:
tail -f /var/log/app_shield.log
# Example alert:
# [ALERT] PID 4821 (python3): syscall rate 48,221/sec -- anomalous
# [INFO]  PID 1204 (nccl_all_reduce): normal pattern

IPsec offload

BF3 can terminate IPsec tunnels in hardware at line rate (400 Gbps). All traffic between GPU nodes can be encrypted in transit without consuming any host CPU cycles. This is particularly valuable when tenant data must cross a shared spine layer in a colocation environment.

# Configure IPsec offload on BF3:
doca_ipsec --mode=offload \
           --local-ip=10.100.0.5 \
           --remote-ip=10.100.0.6 \
           --spi=1001 \
           --key=<256-bit-aes-key>

# Verify offload is hardware-accelerated (not software):
doca_ipsec --status
# Expected: Mode: HW_OFFLOAD, Throughput: 400Gbps, CPU: 0%