Skip to content

RoCE Configuration and Operations on Spectrum-X · Part 3 of 9

Act 3 ”” QoS Architecture on Spectrum-X

The Spectrum-4 ASIC implements 8 Traffic Classes (TC0”“TC7) and 8 PFC priorities (0”“7). Each TC has an independent scheduling discipline, buffer allocation, and ECN/PFC policy. The mapping between DSCP values (set by NICs in the IP header), switch priorities (used by 802.1p/PFC), and traffic classes (internal ASIC queues) must be consistent across every node in the fabric.

Standard AI fabric QoS mapping:

Traffic ClassDSCPSchedulingPFCECNDescription
TC00”“25, 27”“47, 49”“63DWRR weight=16offoffDefault lossy (TCP, storage management)
TC110DWRR weight=8offoffCheckpoint-to-storage
TC326SPononRoCEv2 AI training (lossless)
TC546DWRR weight=32offoffNCCL high-priority collectives
TC648SP highestoffoffCNP feedback (must never pause)

Scheduling disciplines:

Strict Priority (SP): The SP queue is always served before any DWRR queue. TC6 at SP-highest means CNP packets are dequeued before any other traffic class, every single scheduling cycle. TC3 at SP means RoCEv2 training frames are always served before TC0 lossy traffic.

Deficit Weighted Round Robin (DWRR): Used for lossy queues (TC0, TC1, TC5). Each queue gets bandwidth proportional to its configured weight. A TC0 weight of 16 and TC5 weight of 32 means NCCL traffic gets twice the lossy bandwidth of default TCP traffic in a congestion scenario.

Why CNP must be SP and never PFC-paused: This is one of the most operationally important topics in RoCE troubleshooting. The DCQCN mechanism works as follows: the switch marks ECN bits on TC3 frames → the receiving NIC detects the CE mark → the NIC generates a CNP and sends it back to the sender → the sender's NIC rate-limiter reduces the injection rate. If CNP (TC6) is in the same queue as RoCEv2 data (TC3), or if PFC is enabled on TC6, the following deadlock occurs: TC3 fills → PFC fires → pause propagates upstream → the upstream switch also pauses → CNP packets generated by the downstream NIC cannot drain because the return path is paused. DCQCN never fires the rate-limiter. TC3 stays full. Deadlock is complete.

Buffer resource query:

leaf-01# cl-resource-query
  Per-port buffer headroom (TC3, lossless):
    swp1:  headroom=98304 bytes  xoff=81920  xon=32768
    swp2:  headroom=98304 bytes  xoff=81920  xon=32768
  Total lossless buffer carved: 6.25 MB / 48 MB
  Lossy buffer remaining: 41.75 MB

leaf-01# nv show qos scheduler
TC   Discipline  Weight  Priority
---  ----------  ------  --------
0    DWRR        16      -
1    DWRR        8       -
3    SP          -       2
5    DWRR        32      -
6    SP          -       3  (highest)
ANIMATED SIMULATION
QoS Scheduler — SP vs DWRR Under Load
Fabric Load: 60%← ECN marking active on TC3
0% idle100% full congestion
STRICT PRIORITY (served first)
TC6CNP Feedback
3%
TC3RoCEv2 TrainingPFCECN
18%
DWRR (served proportionally)
TC5NCCL Collectivesw=32 (57%)
47%
TC0Default Lossyw=16 (29%)
55%
TC1Checkpoint Storagew=8 (14%)
60%
Hover over any TC · Drag slider to simulate congestion