A GPU-to-GPU RDMA Write is not "just a packet." It is a precisely-engineered sequence of hardware operations that bypass the operating system entirely, construct headers at line rate in NIC firmware, traverse three Ethernet rewrites across two switch hops, and deliver payload directly to GPU memory on the far side -- all without a single CPU instruction touching the data.
The fundamental insight for a network engineer: most of your existing knowledge applies directly. ARP still runs (in RoCEv2 mode). TTL still decrements. ECMP still hashes the 5-tuple. MAC addresses are still rewritten at every L3 hop. What is new is what lives above UDP: the BTH header whose QPair field provides the extra entropy that ECMP needs, the RDMA semantics that make the payload self-routing to a specific GPU memory address, and the PFC mechanism that keeps the lossless queue from dropping the packet when a spine is congested.
InfiniBand removes the UDP/IP layer and replaces it with a simpler, faster lookup (LFT array vs IP route table), native per-packet adaptive routing, and credit-based flow control without pause frames. The BTH is identical. The GPU-visible API is identical. The operational simplification comes at the cost of a proprietary switch stack and a dependency on the Subnet Manager for every addressing decision.
Alternative fabrics -- Gaudi 3's integrated NIC, Slingshot's credit-based Ethernet, MI300X with Pensando -- all use the same BTH transport model but different network layers beneath it. Understanding the packet anatomy at each layer lets you reason about what changes when the hardware changes, without being surprised by undocumented behaviour.