Now run the same GPU-to-GPU write over an InfiniBand NDR fabric (QM9700 switches, ConnectX-7 in IB mode). The CUDA call is identical. NCCL is identical. The QP model is identical. What changes is every header below the BTH.

L1: unchanged

The physical layer is the same. 400G OSFP, 8 lanes x 50G PAM-4. The transceiver is the same part number. The DAC cable is the same. The only physical difference is that IB uses NDR signalling which includes different link training sequences at link-up -- invisible to the packet analyser once the link is established.

L2: InfiniBand Local Routing Header (LRH) replaces Ethernet

In InfiniBand, there is no Ethernet frame. The L2 header is the LRH (Local Routing Header), a 64-bit header:

LRH fields:
  Virtual Lane (VL):    4 bits  -- equivalent to 802.1p priority
  Link Version:         4 bits  -- always 0
  Service Level (SL):   4 bits  -- equivalent to DSCP (determines QoS treatment)
  Link Next Header:     2 bits  -- indicates GRH present (1) or not (0)
  Destination LID:      16 bits -- Local Identifier of destination port (assigned by SM)
  Packet Length:        11 bits -- in 32-bit words
  Source LID:           16 bits -- Local Identifier of source port

The LID (Local Identifier) is a 16-bit address assigned to each HCA port by the Subnet Manager (UFM) at fabric bring-up. It is not configured by the network administrator -- the SM assigns it automatically as part of the fabric initialisation sweep. There is no ARP: the SM knows the LID of every port and distributes routing tables to every switch via Subnet Management Packets (SMPs).

L3: InfiniBand Global Routing Header (GRH) replaces IP

The GRH is optional but required for inter-subnet routing and is commonly present:

GRH fields:
  IP Version:     6 (always IPv6 format, even in IB)
  Traffic Class:  8 bits -- maps to Service Level for QoS
  Flow Label:     20 bits -- equivalent to TCP flow label
  Payload Length: 16 bits
  Next Header:    0x1B -- InfiniBand transport
  Hop Limit:      1 (intra-subnet) or 64 (inter-subnet)
  SGID:           128 bits -- Global ID of source port (GID = subnet prefix + port GUID)
  DGID:           128 bits -- Global ID of destination port

The SGID and DGID use the same GID encoding as RoCEv2 -- in fact, RoCEv2 reuses IB's GID format to represent IP addresses. In RoCEv2, DGID = 0x0000000000000000ffff<IPv4address> (the RoCEv2 GID encoding of an IPv4 address). In native IB, the DGID uses the subnet prefix assigned by the SM and the port's 64-bit GUID.

What the switch looks up: LID vs IP

An IB switch does not have a routing table in the IP sense. It has a Linear Forwarding Table (LFT): a flat array of 65536 entries (one per possible LID), each containing the output port number for that LID. The LFT is populated by the Subnet Manager's routing algorithm (fat-tree, MINHOP, or DFP) and downloaded to every switch via SMPs.

IB switch LFT lookup:
  Input:  DLID from LRH (e.g., 0x0042)
  Output: port 12 (the port that leads toward the destination)
  Time:   O(1) array lookup -- faster than IP route lookup

There is no ARP. There is no MAC table. There is no TTL decrement. The LRH's DLID is never rewritten in transit -- unlike Ethernet where the MAC is rewritten at every hop, the IB LID is preserved end-to-end. Only the LFT is consulted at each switch.

Adaptive routing: per-packet, in hardware, without hashing

In InfiniBand, adaptive routing is not ECMP hash-based. The QM9700 switch measures the congestion level on each output port in real time (via credit counters in the IB credit-based flow control mechanism). For each incoming packet, the switch selects the least-congested output port toward the destination. This decision is made per-packet in hardware. No hash. No flow state. No RSHP reorder buffer needed at the destination -- IB's transport protocol handles out-of-order delivery natively at the QP level.

This is the fundamental performance advantage of IB: per-packet adaptive routing with native OOO tolerance produces near-perfect link utilisation even under all-to-all communication patterns where hash-based ECMP fails.

BTH: identical in both transports

The BTH header is byte-for-byte identical in IB and RoCEv2:

  Opcode, Partition Key, Destination QPN, PSN -- same encoding, same meaning.

This is by design: RoCEv2 was created by taking IB's transport layer (BTH and above) and putting it on top of UDP/IP instead of the IB link/network layers. Everything above the GRH is transport -- everything below is network fabric.

Side-by-side: what changes, what stays

Capture F is the side-by-side decode of both frames for the same RDMA Write First packet -- RoCEv2 on the left, IB on the right. Every field that differs is highlighted.