RoCE on Spectrum-X is not a single configuration command ”” it is a coordinated stack of six interdependent prerequisites, each with a specific failure mode if misconfigured. The correct deployment sequence is: MTU first, DSCP trust second, PFC on priority 3 only, ECN thresholds sized to BDP/4 (practical: 500KB min / 1.5MB max), lossless headroom carved from the 48 MB Spectrum-4 buffer, and CNP on TC6 SP with PFC disabled. The NVUE qos roce shorthand applies all defaults in one line, but production deployments require manual control for NCCL DSCP 46 traffic and custom threshold tuning.

Verification requires ib_write_bw ≥ 46.8 GB/s, ib_write_lat p99 ≤ 1.2µs, and CloudAI all-reduce ≥ 95% of line rate before a deployment is signed off. The four common failure scenarios ”” ECN not marking, PFC deadlock, MTU mismatch, CNP storm ”” each have a specific counter signature and a specific fix. Mastering the counter-to-root-cause mapping is one of the fastest ways to troubleshoot a misbehaving RoCE deployment.

Next chapter: Chapter 26 covers Adaptive Routing and Per-Packet Spraying on Spectrum-X.