Skip to content

Adaptive Routing and Per-Packet Spraying on Spectrum-X · Part 8 of 8

Chapter 26 Summary

Jump to section

1. The Problem with Flow-Based ECMP in AI Fabrics2. Spectrum-4 Adaptive Routing Engine3. BlueField-3 SuperNIC Reorder Engine (DGX B200 Only)4. Configuring Adaptive Routing on Spectrum-X (NVUE Full Walkthrough)5. Resilient Hashing: Protecting Flows During Link Failures6. Telemetry for AR Validation7. AR Comparison: Spectrum-X vs InfiniBand Adaptive Routing8. Summary

Adaptive Routing on Spectrum-X addresses the fundamental limitation of hash-based ECMP in synchronized AI collective workloads. The key takeaways:

  • At scale (512+ GPUs), hash ECMP produces 18-25% load variance across spines; AR reduces this below 3%
  • Spectrum-4 AR engine samples queue depth at sub-microsecond intervals and makes per-packet forwarding decisions at line rate with zero added latency
  • Per-packet AR requires BF3 SuperNIC reorder buffers on DGX B200 hosts; DGX H100/H200 (CX7) clusters must use per-flowlet mode with a 100 us flowlet timer
  • Resilient hashing protects 75% of active flows from disruption during single-link failures in a 4-spine fabric
  • AR validation requires telemetry: NetQ per-port utilization spread <5%, CloudAI benchmark >=46.5 GB/s, WJH showing near-zero RoCE OOO NAKs
  • IB AR uses a centralized SM-based approach with credit-based OOO prevention; Spectrum-X AR is distributed with host-side reorder - different tradeoffs suited to different deployment models
100% complete

Previous

AR Comparison: Spectrum-X vs InfiniBand Adaptive Routing

Chapter complete

Back to curriculum