The Spectrum-4 ASIC implements 8 Traffic Classes (TC0”“TC7) and 8 PFC priorities (0”“7). Each TC has an independent scheduling discipline, buffer allocation, and ECN/PFC policy. The mapping between DSCP values (set by NICs in the IP header), switch priorities (used by 802.1p/PFC), and traffic classes (internal ASIC queues) must be consistent across every node in the fabric.
Standard AI fabric QoS mapping:
| Traffic Class | DSCP | Scheduling | PFC | ECN | Description |
|---|---|---|---|---|---|
| TC0 | 0”“25, 27”“47, 49”“63 | DWRR weight=16 | off | off | Default lossy (TCP, storage management) |
| TC1 | 10 | DWRR weight=8 | off | off | Checkpoint-to-storage |
| TC3 | 26 | SP | on | on | RoCEv2 AI training (lossless) |
| TC5 | 46 | DWRR weight=32 | off | off | NCCL high-priority collectives |
| TC6 | 48 | SP highest | off | off | CNP feedback (must never pause) |
Scheduling disciplines:
Strict Priority (SP): The SP queue is always served before any DWRR queue. TC6 at SP-highest means CNP packets are dequeued before any other traffic class, every single scheduling cycle. TC3 at SP means RoCEv2 training frames are always served before TC0 lossy traffic.
Deficit Weighted Round Robin (DWRR): Used for lossy queues (TC0, TC1, TC5). Each queue gets bandwidth proportional to its configured weight. A TC0 weight of 16 and TC5 weight of 32 means NCCL traffic gets twice the lossy bandwidth of default TCP traffic in a congestion scenario.
Why CNP must be SP and never PFC-paused: This is one of the most operationally important topics in RoCE troubleshooting. The DCQCN mechanism works as follows: the switch marks ECN bits on TC3 frames → the receiving NIC detects the CE mark → the NIC generates a CNP and sends it back to the sender → the sender's NIC rate-limiter reduces the injection rate. If CNP (TC6) is in the same queue as RoCEv2 data (TC3), or if PFC is enabled on TC6, the following deadlock occurs: TC3 fills → PFC fires → pause propagates upstream → the upstream switch also pauses → CNP packets generated by the downstream NIC cannot drain because the return path is paused. DCQCN never fires the rate-limiter. TC3 stays full. Deadlock is complete.
Buffer resource query:
leaf-01# cl-resource-query
Per-port buffer headroom (TC3, lossless):
swp1: headroom=98304 bytes xoff=81920 xon=32768
swp2: headroom=98304 bytes xoff=81920 xon=32768
Total lossless buffer carved: 6.25 MB / 48 MB
Lossy buffer remaining: 41.75 MB
leaf-01# nv show qos scheduler
TC Discipline Weight Priority
--- ---------- ------ --------
0 DWRR 16 -
1 DWRR 8 -
3 SP - 2
5 DWRR 32 -
6 SP - 3 (highest)