To understand the packet, understand what creates it. NCCL's AllReduce operation produces a specific sequence of RDMA write operations. Each write becomes a sequence of packets. The mechanism that drives this is the Queue Pair.

A Queue Pair (QP) is a pair of circular ring buffers -- the Send Queue (SQ) and the Receive Queue (RQ) -- that live in pinned, physically-contiguous memory on the host (or in GPU HBM, with GPUDirect RDMA). The ConnectX-7's DMA engine reads Work Queue Entries (WQEs) from the SQ at line rate, builds the corresponding packets, and places them on the wire. The RQ holds pre-posted Receive Work Requests (RWRs) that tell the HCA where in memory to DMA incoming payload.

When NCCL initialises a communicator between 128 GPUs, it creates one QP per remote peer per GPU. For a 128-GPU ring AllReduce, each GPU maintains 127 remote QPs. Each QP has:

• A local QPN (Queue Pair Number) -- a 24-bit identifier that is unique within the HCA. This is what appears in the RoCEv2 BTH header's destination QPN field.
• A remote QPN -- the peer HCA's QPN for the corresponding Receive Queue.
• A remote GID (Global Identifier) -- the 128-bit IPv6-format address of the peer HCA port. In RoCEv2, the GID maps directly to the peer's IP address via a specific encoding.
• An RKEY (Remote Key) -- a capability token that authorises writing to a specific Memory Region (MR) on the remote side.

When NCCL posts an RDMA Write, it fills a WQE specifying: remote address, remote RKEY, local source VA (virtual address in pinned GPU HBM), length, and signalling flags. The HCA DMA-reads the source data from GPU HBM, constructs the RoCEv2 packet stream, and DMAs it onto the wire. When the last packet of the RDMA Write reaches the destination HCA, that HCA DMAs the payload into the destination GPU HBM at the address the sender specified -- and the destination CPU never wakes up.

This is why the kernel network stack is irrelevant: there is no socket, no sendmsg(), no kernel buffer copy. The path is: GPU kernel -> CUDA -> NCCL -> libibverbs -> HCA firmware -> wire.

How this differs from a normal TCP connection

In a TCP connection, the kernel manages buffers, the kernel performs the socket system call, the kernel's NIC driver submits DMA descriptors. The CPU is involved at every stage. An iperf3 test at 400G would require the CPU to process millions of interrupts per second and copy data between socket buffers and application memory.

In RDMA, the CPU posts a single Work Request and reads a single completion entry when it is done. The HCA handles everything in between. A 400G RDMA write of a 128 MB tensor involves zero CPU copies and zero CPU interrupts during the transfer itself.