Skip to content

Chapter 8: NCCL -- The Application Layer of AI Networking · Part 9 of 9

Taking stock

Building the model…

Ring AllReduce across eight ranks, step by step. Brightness shows how many ranks have contributed to each chunk.

NCCL is the translation layer between the training framework and the fabric. It owns algorithm selection, QP management, and performance reporting -- but it does not own the fabric conditions that determine whether those operations succeed.

You know the three AllReduce algorithms -- ring (optimal for small clusters, near-100% bandwidth), tree (logarithmic steps but 50% bandwidth), and double-binary tree (logarithmic steps AND near-100% bandwidth) -- and how the topology knowledge from Chapter 7 maps to which algorithm NCCL should use.

You know the environment variables that determine whether NCCL even finds your RDMA NICs. NCCL_IB_HCA matching the actual device names from rdma link show is the most common fix for "NCCL is running at 1% bandwidth" with no error message. NCCL_SOCKET_IFNAME pointing to the management interface, not the RDMA interface, is the most common cause of QP establishment failures under load.

You know how to read the nccl-tests output table -- the difference between algbw and busbw, why busbw is the number that matters, why #wrong > 0 is a severity-1 incident, and what "expected busbw" looks like for different cluster sizes and topologies.

And you know the correlation workflow: starting from a NCCL symptom, which CLI command to run first, what that command tells you, and how it routes you to the Chapter 3-7 diagnostic that resolves the root cause.

Continue to Chapter 9 ->

Read aloudPage 9