Cumulus Linux 5.x ships NVUE (NVIDIA User Experience), a unified configuration API and CLI. For RoCE, NVUE provides both a shorthand command that applies all recommended defaults in a single line, and a manual path that gives per-parameter control. Understanding both is important for real-world bring-up and troubleshooting.

The RoCE shorthand (recommended for Day 0):

leaf-01# nv set interface swp1-32 qos roce
leaf-01# nv config apply
leaf-01# nv config save

The qos roce shorthand applies the full NVIDIA-recommended RoCE QoS profile to the specified interfaces in one operation:

• Sets DSCP trust on all specified ports
• Maps DSCP 26 → TC3 (lossless, PFC priority 3)
• Maps DSCP 48 → TC6 (CNP, SP, never paused)
• Enables PFC on priority 3
• Applies ECN thresholds: min_threshold=1.5MB, max_threshold=3MB
• Sets TC3 scheduling to Strict Priority
• Sets TC6 scheduling to Strict Priority (highest)
• Sets TC0 scheduling to DWRR weight 16 (lossy default)

Verification after shorthand:

leaf-01# nv show qos roce
                    operational  applied
------------------  -----------  -------
enable              on           on
roce-mode           lossy        lossy

leaf-01# nv show interface swp1 qos
QoS Interface swp1
  Trust:  dscp
  Traffic Class Summary:
    TC0  DSCP:0-25,27-47,49-63  Mode: DWRR  Weight:16  PFC:off  ECN:off
    TC3  DSCP:26               Mode: SP     PFC:on    ECN:on   Lossless:yes
    TC6  DSCP:48               Mode: SP     PFC:off   ECN:off  Lossless:no

leaf-01# nv show qos trust dscp-map
DSCP  TC   Description
----  ---  -----------
26    3    RoCEv2 training traffic (lossless)
48    6    Congestion Notification Packet (SP)
46    5    NCCL high-priority collectives
10    1    Checkpoint-to-storage
0-9   0    Default lossy

Manual QoS path (for custom thresholds):

# Step 1: Set DSCP trust on each interface
leaf-01# nv set interface swp1-32 qos trust dscp

# Step 2: Set DSCP-to-TC mapping
leaf-01# nv set qos map dscp 26 traffic-class 3
leaf-01# nv set qos map dscp 48 traffic-class 6

# Step 3: Enable PFC on priority 3 only
leaf-01# nv set interface swp1-32 qos pfc priority 3
leaf-01# nv set interface swp1-32 qos pfc enable on

# Step 4: Set ECN thresholds for TC3
leaf-01# nv set qos ecn profile roce min-threshold 500000
leaf-01# nv set qos ecn profile roce max-threshold 1500000
leaf-01# nv set interface swp1-32 qos egress-queue-mapping traffic-class 3 ecn on

# Step 5: Apply and save
leaf-01# nv config apply
leaf-01# nv config save

What shorthand sets vs manual: The shorthand applies NVIDIA-recommended defaults validated across DGX reference deployments. The manual path gives control over ECN thresholds, DSCP map additions (DSCP 46 for NCCL, DSCP 10 for checkpoint), and scheduling weights for DWRR classes. For production deployments with NCCL traffic, the manual path is preferred because DSCP 46 is not in the shorthand default map.

TCAM resource verification:

leaf-01# cl-resource-query
  TCAM Resource Usage:
  ACL entries:     1024 / 16384 (6.25%)
  QoS entries:     48   / 512   (9.37%)
  DSCP map:        8    / 64    (12.5%)
  Forwarding:      2048 / 32768 (6.25%)

leaf-01# nv set interface swp1-32 qos roce

leaf-01# nv config apply

leaf-01# nv config save

leaf-01# nv show qos roce

leaf-01# nv show interface swp1 qos