Per-packet adaptive routing on Spectrum-X creates out-of-order (OOO) packet delivery at the receiver. Understanding why this matters requires a quick look at how RoCEv2 handles error recovery.

Go-Back-N in RoCEv2. ConnectX-7 HCAs implement Go-Back-N (GBN) error recovery. When a receiver detects a sequence gap (packet 42 arrives but packet 41 did not), it sends a Negative Acknowledgement (NAK). The sender must retransmit from packet 41 - the entire outstanding window. For a fabric where packets arrive OOO routinely (as per-packet AR guarantees), the GBN mechanism triggers constantly. The result is catastrophic throughput collapse: measurements show ConnectX-7 all-reduce bandwidth dropping to 40-60% of link rate when per-packet AR is active and OOO rate exceeds 2%.

This is why ConnectX-7 clusters (DGX H100, DGX H200) cannot use per-packet AR mode. They must use per-flowlet AR with a short flowlet timer (100-200 us) to limit OOO exposure.

The BlueField-3 SuperNIC reorder buffer. DGX B200 uses BlueField-3 DPUs in SuperNIC mode. BF3 adds a hardware reorder buffer in the receive path that operates below the QP layer. When packets arrive OOO, BF3 holds them in the reorder buffer, reconstructs the original message sequence, and delivers a correctly ordered stream to the QP. From the QP's perspective, packets always arrive in order - no NAKs are ever triggered.

The reorder buffer depth is configurable. Deeper buffers tolerate larger path-delay differentials (useful when spine-to-spine latency spread is high) but consume BF3 buffer memory. The default depth is sized for a 2-hop fabric (leaf -> spine -> leaf) with differential latency under 10 us, which covers all standard Spectrum-X SuperPOD topologies.

Configuring SuperNIC reorder in NVUE. On each DGX B200 host, the BF3 reorder buffer is enabled through NVUE commands applied to the uplink interfaces:

# Enable SuperNIC reorder buffer on DGX B200 BF3 uplink interfaces
nv set interface eth0 reorder-buffer enable
nv set interface eth1 reorder-buffer enable
nv config apply

Verifying reorder buffer status:

mlxlink -d /dev/mst/mt41692_pciconf0 --show_module | grep -i reorder
# Expected: Reorder Buffer: Enabled

# Check reorder buffer depth (in units of KB)
mlxconfig -d /dev/mst/mt41692_pciconf0 q ROCE_REORDER_BUFFER_SIZE
# Expected: ROCE_REORDER_BUFFER_SIZE 256

Why you cannot run per-packet AR without SuperNIC. The architecture is explicit: per-packet AR (AR mode = per-packet in NVUE) requires that every host NIC in the fabric has a hardware reorder buffer. In a mixed cluster where some nodes are DGX B200 (BF3) and others are DGX H100/H200 (CX7), per-packet AR is not supported. You must use per-flowlet AR for the entire fabric. Spectrum-X validates this at AR enablement time - if it detects CX7-only endpoints, it will override a per-packet AR configuration with a warning.

Comparing recovery behaviour. With per-packet AR and BF3 reorder enabled, a 512-GPU all-reduce that previously took 182ms (with hash ECMP) completes in 134ms - a 26% improvement in collective throughput. The CloudAI benchmark (introduced in Ch25) sees a corresponding improvement in reported all-reduce bandwidth, crossing the 46.5 GB/s acceptance threshold comfortably.