This is the most diagnostically important section in this chapter. IB error counters are the primary tool for identifying cable and transceiver problems before they cause training failures.
show ib counters -- the full view
switch # show ib counters
show ib counters — click each counter type
Uncorrectable physical layer bit errors. The most sensitive early warning of cable and transceiver problems.
Root cause: Dirty optical connector, cable near minimum bend radius, failing transceiver, electromagnetic interference.
IB Port Counters Summary
Interface SymbolErrors LinkErrors RcvErrors XmtDiscards VL15Drops
--------- ------------ ---------- --------- ----------- ---------
IB1/1 0 0 0 0 0
IB1/2 0 0 0 0 0
IB1/3 147 2 0 0 0 <- problem
IB1/4 0 0 0 0 0
IB1/5 0 0 0 0 0
Understanding each counter type:
SymbolErrors -- The count of bad or uncorrectable symbol errors at the physical layer. A symbol error occurs when a received bit pattern does not match any valid InfiniBand symbol. This is the most sensitive indicator of physical layer problems.
- Zero: perfect physical layer
- 1-100: watch carefully, may be a marginal cable or transceiver
- 100+: replace the cable or transceiver immediately
- Rapidly growing: an active signal integrity problem -- likely a faulty optical transceiver running at elevated temperature, or a dirty connector
LinkErrors -- The count of link layer recoveries. When symbol errors accumulate beyond a threshold, the InfiniBand link layer attempts to recover by cycling the port through Init/Armed/Active. Each recovery is a LinkError.
- Zero: no recoveries needed
- 1-5: occasional marginal events, monitor
- Growing: the link is unstable and cycling repeatedly -- training jobs will see timeouts during each recovery
RcvErrors -- Receive-side packet errors that passed physical layer but failed packet-level validation (CRC errors, packet length errors). These indicate problems higher up than the physical layer -- possible firmware bugs, switch buffer problems, or software configuration issues.
XmtDiscards -- Packets the switch discarded on transmit because the output buffer was full. In InfiniBand's credit-based flow control, discards should never happen because credits prevent buffer overflow. If you see XmtDiscards, something is wrong with the credit flow mechanism -- possibly a credit loop, a routing loop, or a misconfigured VL (Virtual Lane).
VL15Drops -- Drops on Virtual Lane 15, which carries InfiniBand management traffic (SM communication, diagnostic packets). VL15 drops mean the SM cannot communicate reliably with the fabric -- this is serious. It can cause LID reassignment failures, routing table corruption, and SM failover.
The counter baseline workflow
Error counters are cumulative -- they never decrease unless you explicitly clear them. This means you cannot tell from a non-zero counter whether the errors are happening right now or happened three months ago.
Best practice: establish a baseline and monitor deltas.
# Record current counter values
switch # show ib counters
# (note the values in a spreadsheet or monitoring system)
# Wait 5 minutes under normal traffic
# Check again -- compare deltas
switch # show ib counters
If SymbolErrors increased by 500 in 5 minutes, you have an active problem. If SymbolErrors is 500 but has not increased in 24 hours, the problem may have been a temporary physical disturbance (someone bumped a cable rack) that has self-corrected.
Clearing counters:
switch # clear counters ib 1/3
Clear counters only after you have recorded the baseline. Clearing counters to "make the problem go away" is the single most destructive thing you can do during a diagnostic -- you erase the evidence.
What counters tell you about root cause
Counter patterns and what they reveal
Active link instability. SymbolErrors exceeded threshold causing link layer recovery. Each recovery interrupts traffic briefly — training jobs will see intermittent NCCL timeouts.
- 1.Failing optical transceiver (most common)
- 2.Damaged cable — internal fiber break
- 3.Loose QSFP connector not fully seated
- 4.ESD damage to transceiver or NIC
- →Replace the optical transceiver first (faster, lower risk)
- →If problem persists, replace the cable
- →Reseat both ends of the cable before replacing
- →Schedule a maintenance window — active link cycling risks training job failures
Growing SymbolErrors + Zero LinkErrors: Early-stage physical layer degradation. The link is still stable enough to avoid recovery, but the physical signal quality is marginal. Causes: dirty optical connector, cable near bend radius limit, transceiver approaching end of life. Action: clean connectors, check cable routing, replace transceiver.
Growing SymbolErrors + Growing LinkErrors: Active link instability. The link is cycling through recovery. Every recovery causes a brief traffic interruption. In a training cluster, this manifests as intermittent AllReduce timeouts. Action: replace the cable first (more common failure), then the transceiver.
Non-zero XmtDiscards: Credit flow problem. This should never happen in InfiniBand. Possible causes: routing loop (SM routing table error), VL credit misconfiguration, or switch ASIC bug. Action: check UFM for routing errors, try restarting UFM and letting it recompute routes. If persistent, engage NVIDIA support.
Non-zero VL15Drops: SM communication is impaired. The SM cannot reliably discover and manage the fabric. Possible causes: SM server overloaded, management network packet loss, SM software bug. Action: check UFM health, restart SM if safe, ensure management network is clean.