All the packet anatomy described above has a corresponding CLI observation on the production devices. This act maps the packet fields to the commands that reveal them.

At the source ConnectX-7 (DGX Node A)

# Verify HCA is in the correct mode (IB or Ethernet)
ibstat mlx5_0 | grep "Link layer"
# Output: Link layer: Ethernet   (or InfiniBand)

# Check DSCP-to-priority mapping (DSCP 46 must map to the lossless PFC priority)
# BasePOD reference: DSCP 46 -> priority 3. Verify this matches your switch config.
mlnx_qos -i mlx5_0 --dscp

# Verify QP state (INIT -> RTR -> RTS means connected and ready)
ibv_devinfo -d mlx5_0 -v | grep QP

# Monitor per-priority packet counters (shows PFC triggering by priority)
ethtool -S eth3 | grep "rx_prio3\|tx_prio3"
# rx_prio3_pause_duration: 142000   <- PFC pauses received on priority 3

# Show current active QP connections (NCCL AllReduce in progress)
rdma res show qp dev mlx5_0 | grep -c RTS
# Output: 127   (127 active QPs for 128-GPU ring AllReduce)

At the leaf switch

# Verify routing table entry for destination GPU IP
show ip route 10.2.1.1
# Output: 10.2.1.1/32 via 10.0.0.3 (SpineC), 4 ECMP paths

# Check ECMP hash configuration -- is BTH QPair hashing enabled?
show ecmp hash
# roce: enabled   <- critical -- must be enabled for proper entropy

# Verify DSCP-to-queue mapping (DSCP 46 must go to lossless queue)
show qos dscp-map
# 46 -> queue 3 -> PFC-protected

# Monitor interface counters for PFC storms
show interface swp36 counters pfc
# Priority 3 pause frames received: 0   (0 = healthy)
# Priority 3 pause frames sent: 2841    <- sending PFC upstream (congestion present)

# Verify ARP entry for destination NIC
show arp 10.2.1.1
# 10.2.1.1 is-at 94:6d:ae:cc:dd:ee via swp1

At the spine switch

# Verify routing table (per-leaf-subnet or per-GPU-host routes)
show ip route 10.2.0.0/24
# Output: 10.2.0.0/24 via 10.0.2.1 (Leaf4 loopback), 4 ECMP downlinks

# Check DLB status (is adaptive LB active?)
show load-balance dlb
# Mode: flowlet   Active: yes   Inactivity timer: 100 us

# Monitor spine utilisation (should be even across all downlinks)
show interface counters | grep "swp[0-9]"
# swp1: TX 78.3%  swp2: TX 79.1%  swp3: TX 12.4% <- imbalanced!

# If imbalanced: check GLB heartbeat status
show glb status
# Heartbeat interval: 20ms   Last update: 3ms ago   Quality: good

InfiniBand-specific observation commands

# On IB switch: show Linear Forwarding Table
ibswitchinfo -D 0  # Shows LFT for switch at depth 0

# Verify Subnet Manager assigned LIDs
ibstat mlx5_0 | grep "LID"
# LID: 0x0042

# Show per-port receive counters (detects symbol errors, link errors)
perfquery -x 0x0042 1  # Query port 1 on device with LID 0x42

# On UFM management console: show routing algorithm
ufm_rest GET /ufmRest/app/network/routing
# {"algorithm": "fat_tree", "lft_mode": "minhop"}

Connecting CLI output to packet anatomy

Every CLI command above exposes one or more fields from the packet path:

• ibstat link layer -> determines whether LRH or Ethernet header is present (Act 3)
• mlnx_qos DSCP map -> determines which PFC priority DSCP 46 maps to (must match switch config)
• show ip route -> the routing table entry the leaf/spine uses for the IP lookup (Acts 4, 5)
• show ecmp hash roce: enabled -> confirms BTH QPair is in the ECMP hash (Act 4)
• show qos dscp-map -> confirms DSCP 46 goes to the lossless PFC queue (Act 3)
• show interface counters pfc -> reveals whether PFC is triggering and for which priority (Act 3, Ch5)
• rdma res show qp -> shows active QPs and their state (Act 2)
• ethtool -S rx_prio3_pause -> counts PFC pause frames received at the NIC (Act 3, Ch5)

When a training job slows down unexpectedly, this is the diagnostic sequence: start at the NIC (ibstat, rdma res show qp, ethtool -S), move to the leaf switch (show interface counters, show ip route, show ecmp hash), move to the spine (show interface counters, show glb status). The packet anatomy in this chapter tells you exactly what each command is measuring and which layer of the packet it corresponds to.