InfiniBand has had per-packet adaptive routing since its design. The Subnet Manager (UFM) programs routing tables that include real-time congestion feedback from every switch port.

How IB adaptive routing works

When a packet arrives at an InfiniBand switch, the forwarding decision is not made by looking up a pre-computed table entry. Instead, the switch measures the current credit count (the proxy for congestion) on each potential output port and forwards the packet to the port with the most available credits -- the least congested port.

This decision happens per-packet, in hardware, at line rate. Every packet from the same QP can take a different path through the fabric. Unlike RSHP on Ethernet, there is no reorder buffer required on the NIC side -- InfiniBand's transport protocol handles out-of-order delivery natively at the QP level.

Checking adaptive routing status on ONYX

# ONYX switch terminal (spine switch -- purple prompt)
show ib counters

IB Port Counters Summary
  Interface  SymbolErrors  LinkErrors  XmtDiscards  VL15Drops
  IB1/1      0             0           0            0
  IB1/2      0             0           0            0
  IB1/3      0             0           0            0

# Check per-port utilisation
show interface ib 1/1

  Input rate:  139,482 Mb/s  (34.9% utilisation)
  Output rate: 138,917 Mb/s  (34.7% utilisation)

show interface ib 1/2

  Input rate:  138,944 Mb/s  (34.7% utilisation)
  Output rate: 139,201 Mb/s  (34.8% utilisation)

On a correctly operating IB fabric with adaptive routing enabled, all spine ports should show similar utilisation during AllReduce. The near-equal distribution (34.9% vs 34.7%) is adaptive routing working correctly.

UFM routing algorithm selection

UFM controls which routing algorithm the SM uses. The choice of algorithm affects both performance and failure recovery behaviour:

# UFM server terminal
show ufm routing

Current algorithm:   DFSSSP
Adaptive routing:    enabled
Reroute on failure:  immediate

FTREE (fat-tree): Designed specifically for fat-tree topologies. Distributes routes evenly across all paths by construction. Good baseline for most DGX deployments.

DFSSSP (Destination-First Shortest Sub-optimal Symmetric Path): Handles asymmetric topologies and link failures more gracefully. Recommended when the fabric has experienced link failures or has a heterogeneous topology.

SSSP (Shortest Sub-optimal Symmetric Path): Useful for mixed InfiniBand generations where some switches run HDR and others NDR.

The routing algorithm does not affect per-packet adaptive routing -- that happens in hardware regardless of algorithm. The algorithm determines the base routing table that adaptive routing adjusts around.