Spectrum-X Ethernet switches support multiple load balancing modes. The correct configuration for an AI fabric is RSHP with per-packet distribution enabled.

Checking current load balancing mode

# Leaf switch terminal (blue prompt)
show ecmp load-balance

Load-balance mode:  hash
Hash fields:        src-ip, dst-ip, src-port, dst-port, protocol
Adaptive:           disabled
Per-packet:         disabled
RSHP:               disabled

This output shows standard SLB (ECMP hash). Adaptive: disabled means no real-time utilisation feedback. Per-packet: disabled means all packets in a flow go to the same path.

Enabling DLB (Dynamic Load Balancing)

# Leaf switch terminal
nv set interface swp1-32 qos load-balance adaptive enable on
nv config apply

# Verify
show ecmp load-balance

Load-balance mode:  adaptive
Adaptive:           enabled
Local utilisation:  monitored
Per-packet:         disabled

DLB monitors output port utilisation and redirects new flows to less-loaded ports. This handles the simultaneous flowlet problem -- new AllReduce flows are routed away from congested links. It does not fully solve the sequential flowlet problem (old flows stay pinned) or the fundamental low-entropy problem (all flows may still look identical).

Enabling RSHP (per-packet spraying)

RSHP requires ConnectX-7 NICs with reorder buffer support. Verify the DGX nodes have ConnectX-7 or later before enabling.

# Leaf switch terminal
nv set interface swp1-32 qos load-balance per-packet enable on
nv set interface swp1-32 qos load-balance rshp enable on
nv config apply

# Verify
show ecmp load-balance

Load-balance mode:  per-packet
Adaptive:           enabled
Per-packet:         enabled
RSHP:               enabled
Reorder tolerance:  200 microseconds

With RSHP enabled, the switch distributes individual packets across all equal-cost paths. The ConnectX-7 NIC receives out-of-order packets but reassembles them before presenting to the RDMA engine. The Reorder tolerance is the maximum time the NIC will wait for out-of-order packets before treating them as lost -- 200 microseconds at 400G is sufficient for typical leaf-spine latency variation.

Verifying load balancing is working

After enabling RSHP, re-check spine utilisation:

# Spine switch terminal
show interface counters

# Expected after RSHP:
Interface swp1 (-> Leaf 0):   Buffer util: 29%
Interface swp2 (-> Leaf 1):   Buffer util: 27%
Interface swp3 (-> Leaf 2):   Buffer util: 28%
Interface swp4 (-> Leaf 3):   Buffer util: 31%

All spine links should now show similar utilisation during AllReduce. The variance across links during active training should be less than 10%. Larger variance indicates that RSHP is not fully distributing traffic -- check that all leaf switches have RSHP enabled (a single leaf without RSHP will still hash all its flows to the same uplinks).