Before the first packet can be analysed, the interfaces that produce it must be understood. A DGX H100 server is, from a network engineer's perspective, an appliance with four distinct network identities on a single chassis. Each identity uses different hardware, different software stacks, different IP addresses, and different physical ports.

Identity 1: The compute network (8x ConnectX-7 HCA ports)

Eight ConnectX-7 HCAs are installed in the DGX, each providing one 400G OSFP port. These are the ports that carry GPU training traffic -- AllReduce, AllGather, ReduceScatter, point-to-point tensor transfers. In a rail-optimised deployment, GPU n on every DGX node connects to a dedicated leaf switch (Rail n), so GPU0 uses HCA0, GPU1 uses HCA1, and so on through GPU7/HCA7. The leaf switch numbering in a given pod is a deployment variable -- what matters is that each GPU rail has its own dedicated leaf. In this chapter's worked example, GPU0/HCA0 on Node A is connected to Leaf1, and GPU0/HCA0 on Node B is connected to Leaf4.

Each ConnectX-7 port has its own IP address. These are the IPs that appear as source and destination in RoCEv2 packets. They are configured on the HCA's network interface as seen by the host OS -- for example mlx5_0 through mlx5_7 in Linux -- but crucially, the host OS's routing table and ARP table are largely irrelevant once NCCL has established its Queue Pairs. The HCA firmware manages the data path completely independently of the kernel network stack once QPs are set up.

Identity 2: The OOB/management network (2x additional ConnectX-7 ports)

The DGX H100 includes two additional ConnectX-7 HCA ports (beyond the eight compute ports) that serve the out-of-band management and storage fabric. These connect to a separate Spectrum-X or standard Ethernet OOB fabric used for UFM management traffic, NVMe-oF storage, and inter-node management connectivity. Like the compute CX7 ports, these have their own IPs and operate independently of the compute fabric. Chapter 17 and 18 cover storage and management networking in detail.

A note on generations: the DGX B200 and DGX GB200 NVL72 introduce BlueField-3 DPUs for dedicated storage and DPU-offloaded management. In those platforms, the storage identity IS a BlueField-3 running its own ARM Linux with its own IP space -- a fundamentally different architecture. The DGX H200 retains the same dual-port ConnectX-7 storage NIC model as the H100. Everything written in this chapter applies to the DGX H100 and H200; the DGX B200/GB200 storage path differs in the ways Chapter 17 will describe.

Identity 3: The management network (1GbE OOB + BMC)

A dedicated 1GbE out-of-band port on the motherboard connects the host OS to the OOB management switch. This carries SSH, monitoring agents, Docker/containerd management, and NCCL bootstrap traffic. In most deployments NCCL's rendezvous phase (rank-to-rank discovery before QPs are established) uses this interface or whatever the default route resolves to -- controlled by the NCCL_SOCKET_IFNAME environment variable. Chapter 18 covers OOB in detail.

Identity 4: The BMC (1x dedicated IPMI port)

A dedicated IPMI/iDRAC port on the motherboard provides out-of-band management at the hardware level -- power control, serial-over-LAN console, hardware event logs. This has its own MAC and its own IP on the OOB management network. It functions even when the host OS is off or crashed.

What Linux sees vs what CUDA sees

This distinction is operationally critical. When you run ip addr on a DGX H100, you see eight compute HCA interfaces (mlx5_0 through mlx5_7), two OOB/management HCA interfaces, and the BMC interface. Linux assigns IP addresses to these, maintains ARP caches for them, and routes packets through them for CPU-originated traffic (SSH from the host, Prometheus exporter, ping commands).

When NCCL launches an AllReduce operation, it bypasses the kernel stack for data movement. NCCL calls ibv_open_device() to access the HCA via libibverbs -- the RDMA verbs API (historically named from InfiniBand but used identically for RoCEv2 and IB) -- which speaks directly to the HCA firmware through a memory-mapped device file. The HCA reads Work Requests from GPU-pinned memory and DMAs packets onto the wire without a single interrupt to the host CPU. The management/OOB interface IP is used only during NCCL's bootstrap/rendezvous phase (before QPs are established); after that, the QP-to-QP data path is direct hardware-to-hardware through the compute HCAs.