Skip to content

Adaptive Routing and Per-Packet Spraying on Spectrum-X · Part 8 of 8

Taking stock

Building the model…

Packet spraying splits every flow across every uplink; the SuperNIC puts the packets back in order before they reach GPU memory.

Adaptive Routing on Spectrum-X addresses the fundamental limitation of hash-based ECMP in synchronized AI collective workloads. The key takeaways:

  • At scale (512+ GPUs), hash ECMP produces 18-25% load variance across spines; AR reduces this below 3%
  • Spectrum-4 AR engine samples queue depth at sub-microsecond intervals and makes per-packet forwarding decisions at line rate with zero added latency
  • Per-packet AR requires SuperNIC reorder buffers on every host - BlueField-3 on B200, ConnectX-8 on HGX B300; DGX H100/H200 (CX7) clusters must use per-flowlet mode with a 100 us flowlet timer
  • Resilient hashing protects 75% of active flows from disruption during single-link failures in a 4-spine fabric
  • AR validation requires telemetry: NetQ per-port utilization spread <5%, CloudAI benchmark >=46.5 GB/s, WJH showing near-zero RoCE OOO NAKs
  • IB AR uses a centralized SM-based approach with credit-based OOO prevention; Spectrum-X AR is distributed with host-side reorder - different tradeoffs suited to different deployment models
Read aloudPage 8