Lab 3 presents a specific scenario: AllReduce throughput has dropped 25% across the cluster. No packet drops are visible. PFC pause frames are present but modest. The question is not whether PFC or ECN is misconfigured -- both are correctly configured. The question is why the fabric is not distributing traffic efficiently.

The scenario

A 16-node training cluster is running data-parallel training. The monitoring system shows AllReduce bandwidth at 75% of expected. JCT has increased by 35%. No alerts have fired -- no drops, no PFC storms, no link failures.

Your task:

1. Confirm the issue is load balancing, not congestion management
2. Identify which spine links are hot and which are idle
3. Check the current load balancing mode on the leaf switches
4. Enable per-packet load balancing
5. Verify that spine utilisation has equalised

The diagnostic path

Step 1: Check leaf interface counters -- confirm no drops, modest pauses

# Leaf switch terminal
show interface counters

Interface swp1 (-> DGX Node A)
  Output drops:      0
  PFC pause frames:  847     <- modest, expected during burst

Interface swp33 (-> Spine)
  Output drops:      0
  PFC pause frames:  2,341   <- somewhat elevated
  Buffer util:       61%     <- elevated but not critical

No drops, modest pauses. This rules out PFC/ECN misconfiguration as the primary cause.

Step 2: Check spine utilisation -- identify hot vs idle links

# Spine switch terminal
show interface counters

Interface swp1 (-> Leaf Rail 0):   Buffer util: 72%  <- hot
Interface swp2 (-> Leaf Rail 1):   Buffer util: 68%  <- hot
Interface swp3 (-> Leaf Rail 2):   Buffer util:  8%  <- idle
Interface swp4 (-> Leaf Rail 3):   Buffer util:  6%  <- idle

Hot and idle links simultaneously. This is the load balancing failure signature.

Step 3: Check current load balancing mode

# Leaf switch terminal
show dcb load-balance

Load-balance mode:  hash     <- static ECMP
Adaptive:           disabled
Per-packet:         disabled

Static ECMP is confirmed. AllReduce flows with low-entropy headers are hashing to swp1 and swp2 on the spine. swp3 and swp4 are not being used.

Step 4: Enable per-packet load balancing

# Leaf switch terminal
enable load-balance per-packet    <- ProDeploy simulator command

Step 5: Verify

# Spine switch terminal
show interface counters

Interface swp1 (-> Leaf Rail 0):   Buffer util: 31%  <- balanced
Interface swp2 (-> Leaf Rail 1):   Buffer util: 29%  <- balanced
Interface swp3 (-> Leaf Rail 2):   Buffer util: 28%  <- balanced
Interface swp4 (-> Leaf Rail 3):   Buffer util: 27%  <- balanced

Spine links are now within 4% of each other. AllReduce bandwidth recovers to full expected throughput.