Before a single RDMA byte flows across your Spectrum-X fabric, six conditions must hold simultaneously ”” and they must be verified in the order listed. Miss any one, and your RoCE traffic either silently degrades or fails outright without a clear error message.
The ordered prerequisite checklist:
1. MTU = 9000 bytes end-to-end (jumbo frames)
This is the non-negotiable foundation. RoCEv2 sends large messages split across back-to-back full-size frames. If any hop ”” NIC, leaf, spine, storage leaf ”” uses a default MTU of 1500 bytes, the effective bandwidth collapses to ~13% of theoretical. Worse, it's silent: ib_write_bw will run without error and report a BDP-limited number. Every interface participating in the RoCE path must be set to 9000 or 9216 bytes.
2. DSCP trust boundary configured
The NIC stamps RoCEv2 frames with DSCP 26 (binary 011010). The leaf switch must trust this value and map it to Traffic Class 3 (TC3). If the switch overwrites DSCP values ”” the default on most platforms ”” all RoCE traffic lands in TC0 (the lossy default queue) and receives no PFC protection. Configure nv set interface swp1-32 qos trust dscp before anything else.
3. PFC enabled on priority 3 Priority Flow Control (PFC) provides the lossless channel. Priority 3 maps to TC3 where DSCP 26 lands. Enabling PFC on priority 3 allows the switch to send IEEE 802.1Qbb PAUSE frames to the upstream NIC when the TC3 queue approaches its headroom limit. Without this, any congestion episode drops RDMA frames, which triggers Go-Back-N retransmissions and latency spikes on the order of milliseconds.
4. ECN enabled on egress queues for TC3
Explicit Congestion Notification is the primary congestion signal for DCQCN (DC Quantized Congestion Notification). When TC3 queue depth exceeds min_threshold, the Spectrum-4 ASIC marks the ECN bits in the IP header of outgoing frames at wire speed ”” no CPU involvement. The receiving NIC generates a Congestion Notification Packet (CNP) and returns it to the sender, which then rate-limits. ECN prevents the queue from filling to the level where PFC would trigger, making PFC the last-resort backstop rather than the primary mechanism. See Chapter 5 for PFC/ECN fundamentals and Chapter 21 for DCQCN parameter deep-dive.
5. Lossless queue carved from lossy budget
The Spectrum-4 ASIC has 48 MB of packet buffer shared across all ports and queues. TC3 must have guaranteed headroom ”” buffer reserved to absorb in-flight frames while a PFC PAUSE propagates upstream and takes effect. Formula: headroom = (cable_propagation_delay + switch_processing_delay) × link_rate. At 400GbE with a 5m DAC cable, this is approximately 100 KB per port. If the lossless queue is not explicitly carved from the lossy budget, buffer overflow in a congestion event will cause the switch to drop lossless frames before PFC triggers ”” defeating the purpose.
6. CNP DSCP 48 trusted as TC6 Congestion Notification Packets must be trusted as DSCP 48, mapped to TC6. TC6 is configured as Strict Priority and is never PFC-paused. This is critical: if CNP packets hit a PFC pause, the DCQCN feedback loop breaks. The congested queue that DCQCN is trying to drain is the same queue that holds CNP packets if you misconfigure the trust map. A CNP stuck in a paused queue means the sender never receives the rate-reduction signal, and the congested queue fills until PFC fires, creating a deadlock.
Common Day-1 mistakes:
- MTU mismatch: set on the DGX NIC but not on the leaf switch uplinks
- DSCP mismatch: switch in
dscptrust mode but the DSCP-to-TC map not applied, so DSCP 26 lands in TC0 - PFC on wrong priority: PFC enabled on priority 0 (the default in some templates) instead of priority 3
- ECN threshold above buffer:
min_thresholdset to 10 MB on a switch with 48 MB total, meaning ECN never marks