Understanding GPU collective communication patterns is not optional context for a network engineer in an AI cluster — it is directly causal to switch configuration decisions. The traffic pattern drives the congestion point drives the buffer sizing drives the ECN threshold drives the DCQCN parameters. Getting this chain wrong produces training throughput that collapses at scale even with perfectly cabled hardware.

The dominant NCCL collective in LLM training is All-Reduce, which synchronises gradient tensors across all GPUs at the end of each backward pass. NCCL implements this as ring-AllReduce by default. In ring-AllReduce, GPUs are arranged in a logical ring. Each GPU simultaneously sends a chunk of its gradient tensor to the next GPU and receives a chunk from the previous GPU. After 2×(N-1) communication steps, every GPU holds the fully reduced gradient. The key property: at any instant, each GPU sends exactly one chunk and receives exactly one chunk — bandwidth is used uniformly, and the algorithm is bandwidth-optimal at O(2(N-1)/N × data_size × bandwidth).

Tree AllReduce arranges GPUs in a binary tree. Reduce phase: leaf GPUs send to parent GPUs up the tree. Broadcast phase: root broadcasts down. Tree AllReduce completes in O(2 × log₂N) steps regardless of message size, making it latency-optimal for small gradient tensors. At large message sizes, ring-AllReduce wins on bandwidth. NCCL switches automatically based on message size.

Incast at synchronisation barriers is the pathology to design around. When all 8 GPUs in a DGX complete their backward pass simultaneously — which they do, because they are processing the same mini-batch — all 8 NICs transmit All-Reduce traffic at line rate simultaneously. All 8 flows target the same logical destination (a ring-AllReduce neighbour, or a parameter server if using All-to-All). At the leaf switch, the single egress queue for the destination port receives 8 × 400 GbE = 3.2 Tbps of traffic attempting to exit through one 400 GbE port. Even with ECMP spreading across spine paths, incast congestion at the spine-to-leaf downlink is unavoidable during synchronisation barriers.

This is the architectural moment where Spectrum-4's 48 MB buffer operates. The 48 MB absorbs the incast burst while DCQCN ECN marking signals senders to reduce their rate. With only 12 MB of buffer (Tomahawk 4), the burst exceeds the tail-drop threshold before DCQCN can respond, producing packet loss and triggering Go-Back-N retransmissions — which add more traffic to an already-congested queue in a feedback loop.

Key NCCL tuning knobs that interact with fabric design: NCCL_MIN_NCHANNELS sets the minimum number of NCCL communication channels (ring instances) per GPU pair. More channels = more parallelism = more simultaneous flows, which spreads incast across more queue entries at the switch but increases the number of active QPs consuming switch buffer. NCCL_IB_QPS_PER_CONNECTION sets the number of RoCEv2 QPs per NCCL channel — higher values produce more granular adaptive routing paths but increase BF3 reorder buffer pressure. Cross-reference Chapter 8 for NCCL algorithm detail and Chapter 25 for QP configuration.

# Measure NCCL All-Reduce bandwidth before and after AR tuning
dgx-01$ mpirun -np 8 --hostfile /etc/nccl-hostfile \
  nccl-tests/build/all_reduce_perf \
  -b 1G -e 4G -f 2 -g 1 -c 1
#                             In-place                    Out-of-place
# Size    Count  Type  Redop  Time(us)  Algbw(GB/s)   Time(us)  Algbw(GB/s)
1073741824  256M  float  sum    1243.8      864.3         1251.2      857.7
2147483648  512M  float  sum    2481.4      865.8         2494.6      861.1
4294967296   1G   float  sum    4963.2      865.5         4978.8      863.9
# 865 GB/s per direction = 93% of theoretical 400GbE × 8 NIC = 928 GB/s
# With standard ECMP (no AR) this number drops to ~55% at 8-DGX scale

# View NCCL transport selection
dgx-01$ NCCL_DEBUG=INFO mpirun -np 2 ./nccl_test 2>&1 | grep -E "transport|channel"
[0] NCCL INFO Channel 00: 0[0x03] -> 1[0x81] [send] via NET/IB/0
[0] NCCL INFO Channel 01: 0[0x03] -> 1[0x81] [send] via NET/IB/1
[0] NCCL INFO NET/IB : Using interface mlx5_0:1 (direct device)
[0] NCCL INFO NET/IB : Using interface mlx5_1:1 (direct device)