The ConnectX-7 does not receive a ready-made packet from software and transmit it. It constructs the packet itself, field by field, from the information in the WQE and the QP context. Understanding this pipeline is what makes RoCEv2 behaviour comprehensible.

Stage 1: WQE dequeue

The HCA's send engine polls the Send Queue ring buffer at the head pointer. When it finds a valid WQE (indicated by the ownership bit), it reads the scatter-gather list -- the list of memory regions to transmit. For a large RDMA Write, NCCL may post one WQE with a 128 MB scatter-gather entry. The HCA will segment this into multiple packets, each at MTU (typically 4200 bytes for RoCEv2 with jumbo frames enabled -- verify with mlnx_qos --pfc and ip link show MTU).

Stage 2: Payload DMA from GPU HBM

The HCA DMA-reads the payload from the GPU HBM address specified in the WQE scatter-gather list. This is GPUDirect RDMA: the HCA's PCIe DMA engine transfers directly from GPU memory to the HCA's internal transmit buffer, bypassing system RAM entirely. On a DGX H100, this uses the NVLink-connected NVSwitch to bridge from the GPU to the ConnectX-7's PCIe endpoint.

Stage 3: BTH construction (IB/RoCE transport layer)

The HCA builds the Base Transport Header. For an RDMA Write operation:

BTH fields:
  Opcode:        RDMA Write First (0x06), Middle (0x07), Last (0x08), Only (0x00)
  Partition Key: 0xFFFF (default partition)
  Destination QPN: 24-bit remote QP number
  Packet Seq Num:  24-bit PSN, incremented per packet
  Acknowledgement: not requested on middle packets (reduces ACK traffic)

The PSN (Packet Sequence Number) is how the destination HCA detects packet loss and out-of-order delivery. If PSN arrives out of order, the HCA either holds it in a reorder buffer (if RSHP/adaptive routing is enabled on the Spectrum-X NIC) or marks it as a NAK (Negative Acknowledgement) and triggers retransmission. The PSN is also what the switch's adaptive routing -- per the QPair field in the BTH -- uses for entropy in the ECMP hash. More on this in Act 4.

Stage 4: UDP and IP header construction (RoCEv2 mode)

In RoCEv2 mode, the HCA wraps the BTH in UDP/IP headers:

UDP header:
  Source port:      varies (generated from hash of QPN -- provides flow entropy)
  Destination port: 4791 (fixed -- the IANA-assigned RoCEv2 port)
  Length:           payload + BTH + UDP header

IP header:
  Source IP:        local HCA port IP (e.g. 10.1.1.1 for mlx5_0)
  Destination IP:   remote HCA port IP (e.g. 10.2.1.1 for remote GPU's mlx5_0)
  Protocol:         17 (UDP)
  DSCP:             46 (EF -- Expedited Forwarding, to be mapped to the lossless PFC priority at the NIC and switch)
  TTL:              64
  Don't Fragment:   set (RDMA cannot fragment -- if MTU mismatch exists, connection fails)

The DSCP value 46 is what enables Priority Flow Control to work -- but the mapping from DSCP value to PFC priority queue is a site-configurable parameter, not a fixed property of RoCEv2. NVIDIA's Spectrum-X BasePOD reference configuration maps DSCP 46 to PFC priority 3; some deployments use priority 5; others use different mappings entirely. What is invariant is that DSCP 46 must map to a lossless (PFC-protected) priority queue, and this mapping must be configured identically on the NIC and every switch in the fabric. The chapter uses priority 3 throughout as the BasePOD reference value. When the switch's egress queue for the RoCEv2 priority fills beyond the XOFF threshold, it sends a PFC pause frame to the upstream device. The HCA, on receiving a PFC pause for that priority, pauses sending all traffic in that priority class -- not just this flow. This is why a single congested flow affects all flows in the same priority class (the head-of-line blocking problem Ch5 describes).

Stage 5: Ethernet frame construction

The IP packet is wrapped in an Ethernet frame:

Ethernet header:
  Destination MAC:  leaf switch swp1 MAC (learned from ARP on interface bring-up)
  Source MAC:       ConnectX-7 port MAC (burned-in hardware address, e.g. 94:6d:ae:xx:xx:xx)
  EtherType:        0x0800 (IPv4)
  802.1Q VLAN tag:  absent in a routed fabric (VLAN tagging only on access ports, not uplinks)

The destination MAC is the leaf switch's MAC -- not the remote GPU's MAC. This is fundamental L3 routing: the NIC ARPs for the leaf switch's gateway IP, gets the leaf switch's MAC, and uses that as the L2 destination for every packet regardless of the remote IP. The remote GPU's MAC never appears in the frame that leaves the NIC.

Stage 6: DSCP-to-PFC priority mapping at the HCA

The ConnectX-7 maps DSCP values to 802.1p priority bits (when VLAN tagging is used) or to internal priority queues (when untagged). The NVIDIA Spectrum-X BasePOD reference mapping for RoCEv2:

DSCP 46 (EF) -> PFC priority 3 -> lossless queue (PFC-protected)  [BasePOD reference]
DSCP 0  (BE) -> PFC priority 0 -> lossy queue (best-effort)

This mapping is site-configurable and must match identically at the NIC (via mlnx_qos) and at every switch in the fabric. NVIDIA's factory default DSCP trust map may use priority 5 rather than 3 for DSCP 46 in some configurations -- always verify with mlnx_qos -i mlx5_0 --dscp and show qos dscp-map on the switch. A mismatch -- NIC marking DSCP 46 to priority 3, switch mapping DSCP 46 to priority 5 (unprotected) -- causes RoCEv2 traffic to be dropped silently under congestion.

Stage 7: PAM-4 encoding and wire

The Ethernet frame is serialised onto the 400G OSFP transceiver as 8 lanes x 50 Gbps each, using PAM-4 modulation (4 signal levels, 2 bits per symbol). At this point, it is an optical or electrical signal on a DAC cable heading toward the leaf switch.

Below is the packet as it leaves the ConnectX-7 -- captured at the NIC egress point. This is Capture A: the frame exactly as it enters the cable.