The frame arrives at Leaf4's uplink port. Leaf4 performs the same lookup as the originating leaf: IP route lookup finds 10.2.1.1/32 is directly connected via swp1 (the port where Node B's HCA0 is connected). L2 lookup finds 10.2.1.1 -> 94:6d:ae:cc:dd:ee (the remote ConnectX-7's MAC) in the ARP table.
Final Ethernet rewrite:
Src MAC: aa:bb:cc:bb:cc:dd (Leaf4 swp1 gateway MAC)
Dst MAC: 94:6d:ae:cc:dd:ee (Node B's ConnectX-7 MAC -- destination GPU's NIC)
TTL: 61 (decremented three times: once at Leaf1, once at SpineC, once at Leaf4).
The frame is delivered to the destination ConnectX-7. This is Capture E -- the frame as it arrives at the destination NIC. At this point, the entire Ethernet journey is visible: the original Ethernet header from Capture A has been rewritten three times (at Leaf1, at SpineC, at Leaf4), but the IP header, UDP header, BTH, and payload are byte-for-byte unchanged from when they left the source NIC.
What the destination ConnectX-7 does
The HCA's receive pipeline:
- Frame validation: check FCS, check destination MAC matches local port MAC.
- Protocol decode: strip Ethernet, check IP dst matches local IP, check UDP port = 4791 -> RoCEv2.
- QP lookup: read Destination QPN from BTH -> look up local QP context table. QPN 0x0042 -> maps to a specific QP that Node A established with Node B during NCCL initialisation.
- PSN check: expected PSN = current PSN + 1? If yes, proceed. If out of order and RSHP enabled: hold in reorder buffer. If out of order and no RSHP: send NACK, trigger retransmission.
- RKEY validation: for RDMA Write, the incoming packet carries a virtual address and RKEY. HCA validates RKEY against the local Memory Region table. If valid, proceed.
- DMA to GPU HBM: HCA DMA-writes the payload to the virtual address specified in the RDMA Write header, translated through the MR's page table to a physical GPU HBM address.
- Completion: when the last segment (opcode "RDMA Write Last" or "Only") is received, the HCA posts a completion entry to the destination's Completion Queue. NCCL's background thread sees this completion and knows the write succeeded.
The full latency budget
| Stage | Contribution |
|---|---|
| GPU HBM -> PCIe DMA -> HCA TX buffer | ~500 ns |
| HCA packet construction (BTH, headers, FCS) | ~200 ns |
| PAM-4 serialisation (400G, 4200B frame) | ~84 ns |
| DAC cable propagation (3m copper) | ~15 ns |
| Leaf1 L3 lookup + MAC rewrite | ~400-600 ns |
| Spine L3 lookup + MAC rewrite | ~400-600 ns |
| Leaf4 L3 lookup + MAC rewrite | ~400-600 ns |
| Propagation (DAC + fibre, ~30m total) | ~150 ns |
| Destination HCA RX pipeline + DMA | ~400 ns |
| Total | ~2.6-3.2 us end-to-end |
InfiniBand NDR achieves ~600 ns end-to-end for the same hop count because it eliminates the UDP/IP header processing overhead, uses credit-based flow control (no PFC latency jitter), and runs a simpler switch forwarding pipeline optimised for IB. The table in Act 7 will break this down precisely.