Understanding load balancing requires understanding the specific traffic pattern that makes it hard. AllReduce creates in-cast congestion -- a pattern where many senders simultaneously send to a single destination port.
In-cast congestion — the AllReduce traffic pattern at the switch
The in-cast has reached its peak. The leaf switch's output buffer is filling rapidly. The switch sends PFC PAUSE frames to slow the 32 senders. Senders pause briefly, buffer drains slightly, senders resume. This stop-start cycle continues for the duration of the AllReduce. This is expected and healthy — PFC is working. Without RSHP, some spine uplinks are handling 90% of the traffic while others are idle.
Buffer at 80-90%. PFC active on all 32 input ports simultaneously. This is the in-cast signature: PFC activity on input ports, congestion on output uplinks.
Output drops: 0 PFC pause frames: 4,823 (and growing) Buffer util: 87%
What in-cast looks like at the switch
During a rail-optimised AllReduce, every GPU 0 on every DGX node simultaneously sends its gradient tensor to the same leaf switch -- Leaf Switch 0. That leaf switch receives 32 simultaneous 400G flows (one from each DGX node in a 32-node cluster) all destined for the same egress ports.
The aggregate input rate during this moment is 32 x 400G = 12.8 Tb/s. The leaf switch's uplinks to the spine can carry 8 x 400G = 3.2 Tb/s (assuming 8 uplinks). The switch must queue the 9.6 Tb/s excess while forwarding at full uplink rate.
This is not a misconfiguration. It is the expected traffic pattern of AllReduce, and it repeats on every synchronisation barrier -- thousands of times per hour. The switch's buffer management, ECN threshold, and PFC headroom are all sized specifically for this burst pattern. Getting the sizing wrong means drops on every AllReduce step.
How in-cast shows in counters
In-cast congestion has a diagnostic signature that distinguishes it from other congestion types:
Drops concentrated on specific egress ports: In in-cast, the congestion is on the output side of the switch -- the uplinks toward the spine, or the ports toward specific destination nodes. Input ports are not congested. If you see drops on specific egress ports while input ports show normal utilisation, you are looking at in-cast.
Simultaneous PFC activity across multiple input ports: Because many senders are contributing to the in-cast, the switch sends PFC PAUSE frames out on multiple input ports simultaneously -- one to each DGX node that is contributing traffic. You will see PFC pause frames growing on multiple input interfaces at the same time.
Burst-shaped counter growth: In-cast from AllReduce is bursty. Counter values spike during the AllReduce barrier and then stabilise between barriers. If you sample show interface counters twice 5 seconds apart, you may see large counter increments during a training step and near-zero increments between steps.
# Leaf switch terminal (blue prompt)
show interface counters
# What in-cast looks like:
Interface swp1 (-> DGX Node A) <- input port
Input packets: 847,293,441
Output drops: 0 <- input side clean
Interface swp33 (-> Spine Switch) <- uplink output port
Input packets: 6,234,918,441
Output drops: 47,291 <- drops on uplink: in-cast
PFC pause frames: 12,847
Buffer util: 87%
The drops are on the uplink toward the spine (swp33+), not on the server-facing ports (swp1-32). This is the in-cast signature.