Each DGX server needs IPs on its network interfaces. In an AI fabric, the compute interfaces (the eight CX7 HCAs for RoCEv2) get IPs that are advertised as /32 host routes into BGP. Understanding why /32 and not /24 changes how you design the entire server addressing scheme.

B>  10.10.1.5/32 [20/0] via fe80::spine01, swp33 weight 1
                        via fe80::spine01, swp34 weight 1
B>  10.10.1.6/32 [20/0] via fe80::spine01, swp33 weight 1
                        via fe80::spine01, swp34 weight 1
# Each server /32 has 2 ECMP paths (one per spine)

Why /32 host routes dominate AI fabric server addressing

In an enterprise network, servers typically get a /24 subnet and the switch has a gateway IP in that subnet. AllReduce on a GPU cluster cannot work this way -- you cannot have a shared L2 subnet spanning leaf switches without violating the lossless fabric requirements (PFC domains, ECN thresholds, and the need to control broadcast traffic).

The correct model: each server interface is a /32. It is directly connected to its leaf switch. The leaf switch advertises the /32 into BGP. Remote leaves learn the /32 via BGP and install a host route pointing to the originating leaf's next-hop.

This means:

• No shared L2 domain across leaf switches
• No ARP flooding fabric-wide
• No broadcast storms possible (each server is its own /32 island)
• The leaf switch is a pure L3 router from the server's perspective

RoCEv2 interface addressing for a DGX H100

Each DGX H100 has eight compute CX7 HCAs. In a RoCEv2 fabric, each HCA gets an IP address:

DGX-01 compute interface addresses (BasePOD 1):
  mlx5_0 / eth0:  10.10.1.1/32   (Rail 0, Leaf-01 facing)
  mlx5_1 / eth1:  10.10.1.2/32   (Rail 1, Leaf-02 facing)
  mlx5_2 / eth2:  10.10.1.3/32   (Rail 2, Leaf-03 facing)
  mlx5_3 / eth3:  10.10.1.4/32   (Rail 3, Leaf-04 facing)
  mlx5_4 / eth4:  10.10.1.5/32   (Rail 0, second port if dual-homed)
  mlx5_5 / eth5:  10.10.1.6/32
  mlx5_6 / eth6:  10.10.1.7/32
  mlx5_7 / eth7:  10.10.1.8/32

DGX-02 compute interface addresses:
  mlx5_0 / eth0:  10.10.1.9/32
  mlx5_1 / eth1:  10.10.1.10/32
  ...

With 8 DGX nodes x 8 interfaces = 64 compute /32 addresses per BasePOD. These easily fit in a /24 block (10.10.1.0/24 has 254 usable addresses; 64 addresses consumed, 190 reserved for expansion).

Each leaf switch advertises the /32 addresses for the DGX ports connected to it:

• Leaf-01 advertises: 10.10.1.1/32, 10.10.1.9/32, 10.10.1.17/32, ... (DGX-01 to DGX-08 rail0)
• Leaf-02 advertises: 10.10.1.2/32, 10.10.1.10/32, ... (rail1 ports)

BGP next-hop behaviour with /32 host routes

When DGX-01 sends a packet to DGX-02's IP 10.10.1.10, the DGX-01 NIC looks up the route. The BGP table on DGX-01 shows: 10.10.1.10/32 via 10.10.0.2 (Leaf-02's loopback). But there are 4 paths to Leaf-02's loopback (via each spine). ECMP distributes the traffic across all 4 paths to the same leaf. Within each path, the last hop is the direct connected link between the leaf and DGX-02's NIC.

This is why the loopback is the BGP next-hop, not the P2P link address: in a BGP unnumbered design there is no P2P link address to use. The loopback provides a stable /32 that the spine switches can route traffic toward.