Everything in Acts 1 through 9 described a DGX node and its fabric without answering one foundational question: when you look at the compute fabric section of the wiring diagram, it says "InfiniBand NDR or RoCEv2." What determines which one you deploy? What is the actual difference, and where does that difference live in the hardware? The answer is simpler and more surprising than most engineers expect. The ConnectX-7 does not care which protocol it runs. The same physical PCIe card -- the same silicon, the same optical connector, the same cable -- can operate as a native InfiniBand HCA or as a 400GbE Ethernet NIC. This is not a software emulation. The ConnectX-7 ASIC was designed from the ground up to support both transport layers natively. You change the operating mode through a firmware configuration utility (mlxconfig), and after a reboot the card presents a completely different personality to the operating system.

Figure: Official NVIDIA ConnectX-7 image. This is the physical card the chapter is referring to when it explains that the external fabric personality can change even when the hardware form factor does not.

What "mode" actually means at the port level: In InfiniBand mode, the ConnectX-7 speaks the InfiniBand protocol natively. It uses credit-based flow control -- packets are only transmitted when the receiver has confirmed it has buffer space. There is no concept of packet loss at the link layer. The port is addressed by a LID assigned by the Subnet Manager, not by a MAC or IP address. In Ethernet mode (RoCEv2), the ConnectX-7 presents as a standard 400GbE network interface. It has a MAC address. It can be assigned an IP address. RDMA still works -- through a protocol called RoCEv2 that encapsulates RDMA verbs in UDP/IP packets -- but losslessness is no longer built-in. You achieve it through configuration: Priority Flow Control (PFC) and Explicit Congestion Notification (ECN). Misconfigure either and you lose the lossless guarantee. The mode is set using mlxconfig on the DGX host:

# Check current mode of all ConnectX-7 NICs
mlxconfig -d /dev/mst/mt4129_pciconf0 q | grep LINK_TYPE
# Output for InfiniBand mode:
LINK_TYPE_P1     IB(1)
# Output for Ethernet mode:
LINK_TYPE_P1     ETH(2)
# Change to Ethernet mode (requires reboot)
mlxconfig -d /dev/mst/mt4129_pciconf0 set LINK_TYPE_P1=2

This single configuration parameter -- LINK_TYPE -- is the fork in the road that determines what your entire external infrastructure looks like.

What a RoCEv2 packet looks like on the wire

When a ConnectX-7 in Ethernet mode sends an RDMA transfer, the resulting packet has a specific layered header structure:

Outer Ethernet header   (src MAC, dst MAC)
IP header               (src IP, dst IP, DSCP = 26 for RoCEv2 traffic)
UDP header              (dst port = 4791, src port = derived from QP number)
InfiniBand BTH          (Base Transport Header -- opcode, QP number, PSN)
InfiniBand payload      (data or AETH for acknowledgements)

Several fields in this header explain things you will see in diagnostic output:

UDP destination port 4791 -- This is the IANA-assigned port for RoCEv2. Every RDMA packet has dst port 4791. When you configure PFC on CoS 3 (DSCP 26 -> CoS 3 via the DSCP trust map), you are protecting all traffic with this destination port. show roce confirming DSCP marking: 26 tells you RDMA packets are being correctly classified.

UDP source port derived from the QP number -- Each Queue Pair gets a unique UDP source port. This means each QP is a distinct UDP flow. ECMP hashing (which uses the 5-tuple including src port) treats each QP as a separate flow and can distribute QPs across different paths. This is the mechanism by which a DGX node with 255 active QPs can distribute AllReduce traffic across all available switch paths -- each QP potentially takes a different ECMP path.

InfiniBand BTH -- opcode, QP number, PSN -- The Base Transport Header carries:

• The opcode (RDMA Write, RDMA Read, Send, Acknowledge)
• The destination QP number (identifies which queue pair receives this packet)
• The Packet Sequence Number (PSN) -- a monotonically increasing counter per QP

The PSN is critical for reliability. The receiver expects PSNs in order. If a packet is dropped and the PSN sequence has a gap, the receiver sends a NACK. The sender retransmits from the missing PSN. If the retransmission fails or the gap is too large, the QP enters error state -- which is the root cause of ibstat showing State: Init or State: Error on a rail after a link flap.

DSCP 26 -- RoCEv2 traffic is conventionally marked DSCP 26 (binary 011010) by the ConnectX-7 NIC. DSCP 26 maps to CoS 3 on the switch. CoS 3 is where PFC and ECN are configured. This chain -- DSCP 26 -> CoS 3 -> PFC protection -- is the mechanism that makes RoCEv2 lossless. A break anywhere in the chain (wrong DSCP marking, wrong DSCP-to-CoS map, PFC on wrong CoS) silently defeats the lossless mechanism. Chapter 5 covers exactly how to verify each link in this chain.

What changes when you choose

The ConnectX-7 mode selection cascades through every layer of the external infrastructure. The DGX node itself -- GPUs, NVLink, NVSwitch, CPUs, RAM, storage -- is completely unchanged. NVLink does not care what protocol the external NICs are running. CUDA does not care. The training job does not care, as long as NCCL can initialise the communication. What changes is everything outside the DGX chassis:

The switches change entirely. InfiniBand NICs must connect to InfiniBand switches. You cannot plug an IB-mode ConnectX-7 into a standard Ethernet switch and expect it to work -- the electrical signalling is compatible (both use the same QSFP56/OSFP cables) but the protocol is incompatible. You need QM9700 leaf switches and Q3400 spine switches running NVIDIA ONYX. RoCEv2 NICs connect to Ethernet switches. You need SN5600 leaf switches running Cumulus Linux or SONiC, and standard Ethernet spine switches. The switch operating system changes. ONYX for InfiniBand. Cumulus Linux or SONiC for Ethernet. These are fundamentally different operating systems with different CLI commands, different configuration paradigms, and different diagnostic tools. Chapter 4 covers ONYX. Chapters 3 and 5 cover Cumulus/Spectrum-X. The management philosophy changes. InfiniBand requires a Subnet Manager (UFM) that centrally manages the entire fabric -- assigns addresses, computes routing, programs every switch. This was covered in Chapter 1 and Chapter 4. You do not configure InfiniBand switches individually. Ethernet is distributed -- each switch is configured independently using standard networking principles. PFC and ECN must be configured consistently across every switch and every port that carries RoCEv2 traffic. The losslessness mechanism changes. InfiniBand: credit-based, built into the protocol, cannot be misconfigured away. RoCEv2: requires PFC (pauses the sender before buffers overflow) and ECN (marks packets to signal congestion before pausing is needed). Both must be correctly configured on every switch port. A single misconfigured port can cause packet drops that stall an entire training job. This is what Labs 1 and 2 diagnose and fix. What stays exactly the same:

• The DGX node hardware (GPUs, NVLink, NVSwitch, CPUs, storage)
• DGX OS -- it runs the same regardless of NIC mode
• CUDA and the training framework
• The physical cables (both modes use the same QSFP56/OSFP optical cables)
• The rail-optimised wiring topology -- 1 NIC per GPU, each on a different switch
• NVLink intra-node communication
• The storage fabric (always Ethernet, always separate from the compute fabric)
• The management network
• ibstat -- it works in both modes because it probes the RDMA layer, not the transport That last point is worth dwelling on. ibstat is part of the MLNX-OFED driver stack, installed on every DGX node regardless of mode. It can interrogate a ConnectX-7 in either mode. In Ethernet/RoCEv2 mode, ibstat output shows Link layer: Ethernet and the LID-related fields (Base lid, SM lid) will show 0 -- there is no InfiniBand Subnet Manager assigning LIDs, because LIDs do not exist in Ethernet mode. If you see non-zero LID values, the NIC is actually in InfiniBand mode, not Ethernet mode.

The practical decision framework

Neither choice is universally better. The decision depends on what you are building and what you already have. Choose InfiniBand when:

• Absolute maximum training performance is the priority
• You are building a dedicated AI training cluster with no legacy infrastructure
• You need SHARP in-network compute to accelerate AllReduce at scale
• You want losslessness guaranteed by the protocol, not by configuration Choose RoCEv2 when:
• You already have Ethernet infrastructure and want to reuse it
• You need multi-tenancy -- Ethernet is easier to segment and isolate for multiple users
• Your operational team knows Ethernet and the additional complexity of InfiniBand is a risk
• You are building a smaller cluster where the performance gap is acceptable
• Cost is a constraint -- Ethernet switches are generally cheaper than equivalent InfiniBand switches The honest performance gap: In controlled benchmarks on identical hardware, a well-configured RoCEv2 deployment on Spectrum-X with RSHP adaptive routing comes within 5-10% of InfiniBand performance on most training workloads. At hyperscale (thousands of nodes), InfiniBand's advantage widens. At the scale of a typical enterprise AI cluster (16-128 DGX nodes), the difference is often acceptable. This is why the industry has genuinely split into two camps rather than one winning. It is not marketing. It is a real tradeoff between protocol simplicity and operational familiarity.