Load balancing problems have specific counter signatures. Knowing what to look for lets you distinguish a load balancing problem from a PFC/ECN configuration problem before you start tuning the wrong layer.
The spine utilisation check
The first diagnostic for a load balancing problem is not PFC counters -- it is spine link utilisation. Open terminal sessions on two or more spine switches and compare link utilisation across uplinks.
# Spine switch terminal (open multiple sessions)
show interface counters
# Note: on a spine, swp1-swpN are downlinks to leaf switches
# Healthy load balancing -- all links similar:
Interface swp1 (-> Leaf 0): Buffer util: 28%
Interface swp2 (-> Leaf 1): Buffer util: 31%
Interface swp3 (-> Leaf 2): Buffer util: 29%
Interface swp4 (-> Leaf 3): Buffer util: 27%
# Load balancing failure -- hot links + idle links:
Interface swp1 (-> Leaf 0): Buffer util: 89% <- hot
Interface swp2 (-> Leaf 1): Buffer util: 91% <- hot
Interface swp3 (-> Leaf 2): Buffer util: 4% <- idle
Interface swp4 (-> Leaf 3): Buffer util: 3% <- idle
In the second output, the fabric has the same aggregate bandwidth -- but two links are approaching saturation while two are nearly empty. This is the flowlet problem in production. AllReduce traffic has hashed onto swp1 and swp2. swp3 and swp4 are available but unused.
The fix is not to add more bandwidth to swp1 and swp2. The fix is to enable per-packet load balancing (sDLB / RSHP) so traffic distributes across all four links.
Correlating with leaf counter patterns
When you see hot spine links, check the leaf switches feeding them to understand which flows are the source.
# Leaf switch terminal
show interface counters
# swp1-32 = server-facing ports, swp33+ = spine-facing uplinks
# High utilisation on uplinks feeding hot spine ports:
Interface swp33 (-> Spine swp1):
Input packets: 6,234,918,441
Buffer util: 88%
PFC pause frames: 23,847
Interface swp34 (-> Spine swp2):
Input packets: 6,198,441,223
Buffer util: 86%
PFC pause frames: 19,231
Interface swp35 (-> Spine swp3): <- feeds idle spine port
Input packets: 298,441
Buffer util: 2%
PFC pause frames: 0
The leaf switch is sending almost all its AllReduce traffic through swp33 and swp34. swp35, which connects to the idle spine port, carries almost nothing. DLB or sDLB on this leaf switch would have detected the utilisation imbalance and distributed traffic across all three uplinks.
The diagnostic pattern that separates LB failure from PFC misconfiguration
This is the key diagnostic decision that the previous chapters set up:
| Counter pattern | Root cause | Fix |
|---|---|---|
| PFC pauses growing, drops zero, spine links even | Normal AllReduce burst, PFC working | No action -- healthy |
| PFC pauses growing, drops zero, some spine links hot | Load balancing failure | Enable DLB/sDLB |
| Drops growing, PFC pauses zero | ECN or PFC not configured | Configure ECN + PFC |
| Drops growing despite PFC, spine links even | Oversubscription or capacity limit | Capacity planning |
| Drops growing despite PFC, some spine links hot | LB failure causing local oversubscription | Fix LB first |
The spine link utilisation comparison is the diagnostic that routes you to the correct layer. Without it, PFC activity looks the same whether the cause is a load balancing failure or a genuine congestion spike -- and you will tune the wrong thing.