This is the checklist an engineer works through when bringing up a new DGX node on a Spectrum-X fabric. Every item must be correct before RDMA traffic will be lossless.

Step 1: Jumbo frames (MTU 9000)

Both the DGX NIC and the switch port must have MTU 9000 configured. A mismatch causes fragmentation which kills RDMA performance.

# DGX host terminal
ip link set eth0 mtu 9000
# Persist: add to /etc/network/interfaces or NetworkManager config

# Verify
ip link show eth0 | grep mtu

If the switch port MTU is 1500 (default) and the NIC sends 9000-byte frames, the switch will fragment or drop the frames (depending on the DF bit). RDMA transfers will fail or perform at 1/6th efficiency.

Step 2: DSCP trust mode

The switch must be configured to trust the DSCP marking sent by the DGX NIC. If the switch is in "trust CoS" mode instead of "trust DSCP" mode, it will ignore the NIC's DSCP marking and put RoCEv2 traffic in the wrong queue -- the default queue, which has no PFC protection.

# Leaf switch terminal
# Verify trust mode
show dcb ets

# If trust mode is wrong, configure it (swp1-32 = server-facing ports)
nv set interface swp1-32 qos trust dscp
nv config apply

Step 3: DSCP-to-CoS mapping

The RoCEv2 DSCP value (DSCP 26 in this deployment's QoS policy -- your deployment may use DSCP 46 or another value) must map to the lossless PFC-protected priority. Verify this and set it if wrong:

# Leaf switch terminal
nv show qos mapping

# In this deployment: DSCP 26 -> CoS 3
# If wrong, configure:
nv set qos mapping traffic-class 3 dscp 26
nv config apply

# Also verify on the NIC side -- must match the switch:
mlnx_qos -i eth0 --dscp

Step 4: ETS traffic class configuration

CoS 3 must be in a Strict Priority (SP) queue -- not a weighted queue. RoCEv2 traffic requires deterministic latency, which SP provides. Weighted queuing introduces variable latency as the queue depth changes.

# Leaf switch terminal
show dcb ets

TC3 (RoCE)  3  50%  Strict Priority   <- correct

Step 5: PFC on the lossless priority

# Leaf switch terminal (apply to all server-facing ports)
nv set interface swp1-32 qos pfc enable on
nv set interface swp1-32 qos pfc priority 3
nv set interface swp1-32 qos pfc watchdog action drop
nv set interface swp1-32 qos pfc watchdog interval 200
nv config apply

# DGX host terminal
mlnx_qos -i eth0 --pfc 0,0,0,1,0,0,0,0

Step 6: ECN thresholds

# Leaf switch terminal
nv set qos congestion-control profile roce \
    min-threshold 150000 \
    max-threshold 1500000 \
    probability 100
nv config apply

Step 7: Verify with show roce

After all configuration is applied:

# Leaf switch terminal
show roce

RoCE Configuration -- eth0
  RoCE version:   RoCEv2
  State:          active
  PFC:            enabled (priority 3)    <- must be priority 3
  ECN:            enabled -- DCQCN active <- must be active
  DSCP marking:   26 (RoCE traffic)       <- must be 26
  MTU:            9000                    <- must be 9000
  GID:            fe80::506b:4b03:00a1:b200 <- must be non-zero

All six fields must show the correct values. A single misconfiguration breaks losslessness -- usually silently, until a training job fails under load.