Not every AI cluster uses NVIDIA DGX hardware. Different accelerators, different interconnects, and different NIC architectures produce different packet flows. Understanding these alternatives is increasingly important as the AI infrastructure market diversifies.

Gaudi 3 (Intel/Habana): the accelerator IS the NIC

Gaudi 3 integrates 24 x 100GbE ports directly onto the accelerator die -- there is no separate ConnectX-7, no separate PCIe NIC, no separate GPU HBM-to-NIC DMA path. The accelerator package is simultaneously the compute engine and the network adapter.

What changes in the packet path:

• There is no PCIe bus between compute and NIC. Data that the network stack reads is in Gaudi's on-die SRAM or HBM directly accessible to the integrated network engine.
• The network engine implements RoCEv2 natively but uses a different queue pair management model -- Gaudi's HCCL (Habana Collective Communications Library) manages QPs via a different verbs-like API than libibverbs.
• Packet headers are structurally identical to ConnectX-7 RoCEv2 -- same BTH, same UDP/IP, same DSCP marking. A Wireshark decode of a Gaudi 3 packet looks identical to a ConnectX-7 packet.
• The switch sees exactly the same RoCEv2 traffic and applies the same PFC/ECN/ECMP logic.
• What is missing: GPUDirect Async (which depends on NVIDIA's proprietary PCIe peer-to-peer signalling), SR-IOV VFs (Gaudi 3 does not expose NIC virtualisation in the same way), and the ConnectX-7's specific adaptive routing participation.

Real-world deployment: Intel Gaudi 2/3 clusters at scale

Intel-based Gaudi clusters (deployed at several European cloud providers and in AWS DL2q instances) use a 2.4 Tbps all-to-all fabric built from the 24 x 100GbE Gaudi ports (24 x 100G = 2.4 Tbps). The fabric typically uses standard Ethernet switches (Arista, Juniper) rather than Spectrum-X. Because Gaudi's ports are 100GbE rather than 400GbE, the topology typically uses higher radix to achieve equivalent bandwidth -- 24 ports per accelerator vs 8 ports per DGX GPU (though DGX provides 400G per port vs Gaudi's 100G, making the total compute fabric BW 3.2 Tbps for DGX vs 2.4 Tbps for Gaudi per node).

AMD Instinct MI300X with AMD Pensando Elba DPU

AMD's MI300X pairs with the AMD Pensando Elba NIC/DPU rather than NVIDIA's ConnectX-7. The Elba DPU runs a DPDK-based P4 dataplane on its ARM cores rather than the firmware-driven approach of ConnectX-7.

What changes in the packet path:

• The RoCEv2 packet format is structurally identical -- same BTH, same UDP/IP, same DSCP. The difference is in what happens inside the NIC.
• Elba's RDMA engine uses a different RNICM (RNIC Microcode) implementation. QP capacity limits, maximum MR sizes, and maximum message sizes may differ.
• The kernel driver is different (ionic instead of mlx5_core). RCCL (ROCm Collective Communications Library, AMD's equivalent of NCCL) interacts with the RDMA stack via the same libibverbs API, but the underlying driver path differs.
• Adaptive routing capability depends on Elba's implementation. As of MI300X deployments, per-packet spraying is supported but with different MTU and reorder buffer characteristics than ConnectX-7 RSHP.

HPE Slingshot: a different Ethernet -- not RoCEv2, not InfiniBand

The Frontier supercomputer (Oak Ridge National Laboratory -- AMD Instinct MI250X + HPE Cray EX) uses HPE Slingshot 11 interconnect. Slingshot is Ethernet-based but is not RoCEv2 and is not standard Ethernet.

What Slingshot adds to the Ethernet frame:

Slingshot uses a custom Ethernet frame with a vendor-specific header extension:

Standard Ethernet:
  [Dst MAC][Src MAC][EtherType 0x0800][IP][UDP][BTH][Payload]

Slingshot Ethernet:
  [Dst MAC][Src MAC][EtherType 0x0801 (Slingshot-specific)][Slingshot Header][IP][UDP][Cray BTH][Payload]

The Slingshot header carries:

• HRP (Hardware Retry Protocol) control bits: credit-based acknowledgement without PFC
• Adaptive routing tag: the hardware tells the switch which flow this belongs to for adaptive routing decisions
• Congestion control fields: Slingshot uses its own congestion notification mechanism -- not ECN marks in DSCP, but dedicated header fields processed by Slingshot ASICs

Why this matters:

A standard Ethernet switch cannot forward Slingshot frames correctly -- it will see EtherType 0x0801 and may drop it or treat it as an error. Slingshot requires Slingshot-specific switch ASICs (the Rosetta ASIC). The packet format is proprietary.

Slingshot's congestion control is credit-based at the hardware level -- each Slingshot port maintains per-destination credits. When a destination buffer fills, credits are not issued, and the source NIC physically cannot send until credits are replenished. This eliminates PFC storms (there is no backward-propagating pause frame -- the source just cannot send), but requires hardware-enforced credit tables in the switch ASIC.

What stays the same:

The RDMA transport model is the same -- Queue Pairs, Work Requests, Memory Regions, BTH opcodes. A programmer writing NCCL-equivalent code against the Cray RDMA API sees the same abstract model. What differs is the network plumbing that carries the BTH.

NVIDIA Quantum InfiniBand vs ConnectX-7 in RoCEv2 mode: same hardware, different mode

A critical point often overlooked: the ConnectX-7 itself is a dual-mode adapter. The same physical card can operate in InfiniBand mode (connected to a QM9700 IB switch) or Ethernet mode (connected to an SN5600 Spectrum-X switch running RoCEv2). The NIC mode is set explicitly via mlxconfig (shown below) -- the switch port type is matched to the configured NIC mode, not the other way around.

# Check mode on a ConnectX-7
mlxconfig -d /dev/mst/mt4129_pciconf0 query | grep LINK_TYPE
LINK_TYPE_P1    IB(1) or ETH(2)

# Changing mode (requires reboot)
mlxconfig -d /dev/mst/mt4129_pciconf0 set LINK_TYPE_P1=2  # Set to Ethernet

When you change mode from IB to Ethernet, the entire LRH/GRH header disappears and UDP/IP appears instead. The BTH stays. The GPU-to-GPU communication latency increases by approximately 1.5-2 us per hop due to the added header processing overhead and the replacement of IB's credit-based flow control with PFC (which introduces pause latency under congestion).