Links are confirmed up on both ends. Now you look at what traffic is doing on those links. Counter commands also have a two-end perspective: the switch sees the fabric view (all flows aggregated at the port), and the DGX NIC sees its own individual experience (what was sent to it, what it received, what it was told to pause).
You need both perspectives because they can tell different stories -- and the difference between those stories is itself diagnostic.
Traffic counters — two perspectives on the same link
leaf-switch # show interface counters Interface eth0 Output drops: 0 ← looks clean PFC pause frames: 847 ← moderate pauses Buffer util: 45% ← buffer looks fine
The switch shows only 847 pause frames total and no drops. The switch view looks almost clean. Buffer utilisation is moderate. Nothing alarming.
dgx-node-a:~$ ethtool -S eth0 rx_pfc_pause_frames: 94,283 ← 94K pauses in last poll interval tx_pfc_pause_frames: 0 rx_ecn_marked: 0 tx_dropped: 0
The NIC tells a completely different story. 94,283 rx_pfc_pause_frames in the last poll interval. The NIC is being paused almost continuously. It is not dropping packets because PFC is working — but it is barely sending anything.
This is a pause storm hidden from the switch view. The switch sees 847 total pauses because each pause is brief — but they arrive so frequently that the NIC is held idle most of the time. The application sees near-zero throughput. A monitoring system watching only the switch would miss this entirely.
This is why you must check ethtool on the DGX even when the switch looks clean. The NIC counter reveals pause frequency that aggregate switch counters obscure. Check PFC watchdog — it should be firing after 200ms of continuous pause, but repeated short pauses can avoid it.
Switch end: show interface counters -- the fabric view
Device: leaf switch terminal (blue prompt)
show interface counters on the Spectrum-X switch shows you what the switch is doing to the traffic on each port. This is the aggregate view -- drops and pauses from all flows combined, hitting the switch's egress buffer simultaneously.
show interface counters — what each pattern means
leaf-switch # show interface counters
Interface swp1
Input packets: 1,847,293,441
Output drops: 47,291 <- non-zero drops: problem
PFC pause frames: 12,847 <- non-zero pauses: PFC active
Buffer util: 87% <- high buffer: congestion
Output drops: 47,291 -- The switch's output buffer filled and packets were discarded. In a lossless RDMA fabric this must be zero.
PFC pause frames: 12,847 -- The switch sent PAUSE frames to stop the upstream sender. PFC is active. Whether this is good or bad depends on context -- a small count under burst traffic is expected; rapidly growing means a pause storm.
Buffer util: 87% -- The output buffer is nearly full. Approaching 90% is where drops begin.
The key patterns:
- Drops, no pauses -> PFC and ECN both absent or misconfigured. Traffic dropping uncontrolled.
- Pauses, no drops -> PFC working correctly. Lossless. Congestion is present but managed.
- Both drops and pauses -> PFC overwhelmed or on wrong priority. Congestion too severe or misconfigured.
- Neither -> Clean. No congestion.
If you are seeing sustained pauses with uneven spine link utilisation (some links hot, others idle), the root cause is likely load balancing -- not PFC misconfiguration. PFC is doing its job. The layer upstream of PFC is failing. Chapter 6 covers load balancing diagnostics.
The switch view answers: is this port healthy at the fabric level
DGX end: ethtool -S eth0 -- the NIC's experience
Device: DGX host terminal (green prompt) -- switch back to the DGX now
ethtool -S eth0 runs on the DGX host. It shows hardware statistics from inside the ConnectX-7 NIC -- what the NIC received, what it sent, and crucially, how many times the switch told it to stop sending.
ethtool -S eth0 — click each counter to understand it
dgx-node-a:~$ ethtool -S eth0
NIC statistics:
rx_pfc_pause_frames: 12847 <- switch told this NIC to stop sending
tx_pfc_pause_frames: 8293 <- this NIC told the switch to stop sending to it
rx_ecn_marked: 0 <- no ECN signals being received
tx_dropped: 47291 <- NIC dropped packets on transmit
link_speed: 400Gb/s
link_state: up
rx_pfc_pause_frames: 12847 -- The switch sent 12,847 PAUSE frames to this NIC. Notice this matches the PFC pause frames on the switch side -- they are counting the same events from opposite ends of the cable.
tx_pfc_pause_frames: 8293 -- This NIC sent 8,293 PAUSE frames to the switch. The NIC's own receive buffer was filling and it needed the switch to slow down.
rx_ecn_marked: 0 -- No ECN-marked packets received. Even though drops are occurring, the switch is not sending early congestion signals. ECN is not configured -- the switch is dropping instead of marking.
The story the NIC tells that the switch cannot:
Now imagine the switch showed no drops but training throughput is still degraded. The NIC's rx_pfc_pause_frames might be growing at 10,000 per second. The switch is successfully preventing drops through PFC -- but the NIC is being held in a pause state almost continuously. The application sees near-zero throughput. The switch report looks clean. Only the NIC counter reveals the pause storm.
The two-counter rule:
Always read show interface counters (switch) and ethtool -S eth0 (DGX) together. The switch tells you what the fabric is doing. The NIC tells you what the DGX experienced. They corroborate each other when everything is working and diverge -- productively -- when there is a subtle failure.