NCCL implements three collective algorithms. Understanding which one NCCL chooses -- and why -- connects the topology knowledge from Chapter 7 directly to the performance numbers you see in nccl-tests.
NCCL AllReduce algorithms -- interactive comparison
Reduce-scatter: share + sum shards around ring
AllGather: broadcast reduced shards
Small clusters (<=32 nodes), low-latency balanced fabrics
Large clusters -- step count grows linearly with N
NCCL_ALGO=RING(leave unset in production -- NCCL auto-selects)Ring AllReduce
Arranges all N GPUs in a virtual ring. The AllReduce proceeds in two phases:
Reduce-scatter phase: Each GPU sends a shard of its gradient tensor to the next GPU in the ring, receiving a shard from the previous GPU and adding it to its own. After N-1 steps, each GPU holds one reduced shard -- the complete sum of all GPUs' contributions to that slice.
AllGather phase: Each GPU sends its reduced shard to the next GPU in the ring, forwarding what it receives. After N-1 more steps, every GPU holds the complete reduced tensor.
Communication volume: Each GPU sends and receives exactly 2 x (N-1)/N x tensor_size bytes. For large N this approaches 2 x tensor_size -- essentially constant regardless of GPU count. Every GPU is always sending and receiving simultaneously -- near-perfect bandwidth utilisation.
When ring is optimal: Small clusters (<= 32 nodes) with balanced, low-latency fabrics. BasePOD topology (2-stage, 4 hops) with ring AllReduce achieves near-line-rate busbw because the ring's N-1 steps all have the same hop count.
When ring degrades: Large clusters. At 256 GPUs, ring requires 255 steps. The tail latency of those steps compounds -- the last GPU in the ring sees the full accumulated latency of all preceding steps. A single slow NIC or congested link stalls the entire ring at every step.
Tree AllReduce (Binary Tree)
Arranges GPUs in a binary tree hierarchy. Reduction happens in log2(N) steps -- dramatically fewer than ring's N-1.
For 256 GPUs: 8 steps instead of 255. Each step cuts the number of active senders in half. At the root, one GPU holds the complete sum and broadcasts it back down the tree in another log2(N) steps.
The problem: At each tree level, half the GPUs are idle -- waiting for the other half to finish. Bandwidth utilisation is approximately 50% of ring's efficiency. The root GPU is a bottleneck -- all traffic must flow through it.
Double-Binary Tree (DBT)
Runs two binary trees simultaneously, with roots at different GPU positions. The key insight: the GPUs that are idle in Tree 1 (waiting at each level) are active in Tree 2 (contributing at those same levels). Every GPU is active in at least one tree at every step.
Result: logarithmic step count (log2(N) steps, like tree) AND near-optimal bandwidth utilisation (like ring). DBT is NCCL's default for large AllReduce operations and is the algorithm that makes SuperPOD-scale training practical.
The topology connection from Chapter 7: NCCL does not know your switch topology -- it only knows which NICs are available and the latency it measures to each peer. But understanding hop count from Chapter 7 tells you which algorithm NCCL should prefer:
| Cluster | Stage count | Hop count | Optimal algorithm |
|---|---|---|---|
| 8 nodes, single node | 0 (NVLink) | 0 | NVLink (automatic) |
| 32 nodes, BasePOD | 2-stage | 4 hops | Ring (low step count dominates) |
| 256 nodes, SuperPOD | 3-stage | 8 hops | DBT (log steps dominate) |
| 2048 nodes | 5-stage | 16+ hops | DBT + SHARP in-network reduction |
When NCCL_ALGO is not set, NCCL auto-selects based on measured latency and message size. You can force a specific algorithm: NCCL_ALGO=RING for benchmarking ring performance, NCCL_ALGO=TREE or NCCL_ALGO=COLLNET_RING for SHARP-aware collectives.