NCCL (NVIDIA Collective Communications Library, pronounced "nickel") is the software layer between the training framework and the network hardware. When PyTorch executes an AllReduce, it calls NCCL. NCCL decides which NICs to use, which algorithm to run, how to manage the Queue Pairs, and what to do when something goes wrong.
The full stack -- from training framework to fabric
Click any layer to see what it owns and what it hands to the layer below
- +Algorithm selection (ring / tree / DBT)
- +GPU topology detection (NVLink vs external fabric)
- +Queue Pair establishment and lifecycle
- +Error detection: timeouts, QP failures
- +busbw metric -- performance reporting
- +Fallback transport selection (RDMA -> socket)
- xPFC configuration
- xECN thresholds
- xDCQCN rate control parameters
- xSwitch load balancing mode
- xPhysical link state
- xFabric topology (it only measures, does not know)
The ownership boundary
NCCL owns:
- Detecting which GPUs are present and how they are connected (NVLink intra-node vs external fabric inter-node)
- Selecting the AllReduce algorithm based on detected topology and message size
- Managing Queue Pairs -- the RDMA connection objects -- between all participating GPUs
- Error detection: QP failures, packet loss detection, timeout thresholds
- Reporting: the
busbwmetric that tells you how efficiently the collective operation ran
NCCL does not own:
- PFC configuration -- that is the switch and the ConnectX-7 driver
- ECN thresholds and DCQCN -- that is the switch ASIC and NIC firmware
- Load balancing -- that is the switch forwarding plane
- Physical link state -- that is ibstat and the RDMA driver
This boundary is the most important thing to understand when debugging. NCCL is sensitive to fabric quality but cannot fix fabric problems. When NCCL shows degraded performance, the cause is almost always in the layers NCCL does not control -- and the diagnostic commands from Chapters 3 through 7 are what you use to find it.
NCCL initialisation -- what happens before the first gradient
Before any gradient tensor moves, NCCL runs an initialisation sequence when the training job starts:
- GPU detection: Probes all available GPUs and their NVLink connectivity
- NIC probing: Scans for RDMA-capable network interfaces (looks for
mlx5_*devices via the RDMA verbs API) - QP establishment: Creates Queue Pairs between every GPU pair that will communicate across the external fabric -- on a 32-node cluster this means creating QPs between 256 GPUs x 255 peers = 65,280 connections
- Algorithm negotiation: Based on detected topology, selects the optimal AllReduce algorithm
- Validation test: Runs a small collective to verify all paths work before training begins
If NIC probing in step 2 finds no RDMA devices, NCCL does not fail -- it falls back to socket-based TCP transport. This is the most common cause of "NCCL is using 2 GB/s instead of 400 GB/s" and it produces no error, just a warning in the debug log that most engineers miss.