Every chapter so far has described what happens around the packet -- the protocols, the topology, the routing table, the congestion control mechanism. This chapter goes inside the packet itself.
A gradient tensor leaves GPU 0 on DGX Node A. Roughly 1.4 microseconds later, it arrives at GPU 0 on DGX Node B. Between those two moments, a specific sequence of hardware operations rewrites the frame's headers three times, makes three routing decisions, runs two ECMP hashes, and enforces PFC priority on two switch ports -- all at line rate, all in silicon, without a single CPU instruction.
If you are a network engineer, you have spent your career understanding packets. You know what a Wireshark decode tree looks like. You know what "TTL decremented by 1" means at a transit router. You know the difference between a MAC table lookup and a routing table lookup. This chapter uses exactly that knowledge to take you inside the AI fabric -- translating the familiar disciplines of L2/L3 networking into the specific, concrete mechanics of GPU-to-GPU communication at 400 gigabits per second.
By the end, you will be able to answer questions that stump most practitioners: What IP address does the GPU traffic use as its source? Does the fabric do ARP? What is in the BTH header and why does it matter for load balancing? What changes when you swap InfiniBand for RoCEv2? What changes when you replace a ConnectX-7 with a Gaudi 3 accelerator that has no separate NIC at all?
How this chapter is structured -- read this first
Acts 1 through 6 trace a single RoCEv2 packet from source GPU to destination GPU, step by step, in RoCEv2 mode only. Every Wireshark capture in those acts is a RoCEv2 frame. Every routing decision is an Ethernet IP lookup. Every congestion mechanism is PFC and ECN.
Act 7 then runs the identical GPU-to-GPU RDMA Write over InfiniBand NDR and shows, layer by layer, exactly what is different. The RDMA programming model (CUDA, NCCL, libibverbs, Queue Pairs, BTH opcodes) is identical in both cases -- only the network headers below the BTH change. If you see a reference to libibverbs or to the "IB verbs API" in Acts 1-6, note that libibverbs is the standard RDMA programming interface used for both InfiniBand and RoCEv2 -- it is not exclusive to InfiniBand. The "IB" in "libibverbs" refers to the historical origin of the API in the InfiniBand standards body, not to a transport restriction.
Act 8 covers alternative NIC stacks (Gaudi 3, MI300X, Slingshot) and Act 9 is the full comparison matrix across all fabrics.