After applying the full RoCE configuration, verification requires two InfiniBand performance benchmarks: ib_write_bw for throughput and ib_write_lat for latency.

Expected performance at 400GbE:

Running ib_write_bw:

# On dgx-node-02 (server):
dgx-02:~$ ib_write_bw -d mlx5_0 --iters 5000 --size 65536

# On dgx-node-01 (client):
dgx-01:~$ ib_write_bw -d mlx5_0 --iters 5000 --size 65536 192.168.100.2

-------------------------------------------------------------------
 #bytes     #iterations    BW peak[GB/sec]    BW average[GB/sec]
 65536      5000             46.92              46.81
-------------------------------------------------------------------

The rest of the perftest suite, and when to reach for each

ib_write_bw and ib_write_lat are two of six. The others isolate different things, and running the right one first saves a lot of guessing:

All six take the same shape: run the server bare, give the client the server's address.

# Server
ib_write_lat -d mlx5_0

# Client
ib_write_lat -d mlx5_0 192.168.100.2

A healthy 400G link answers roughly:

 #bytes  #iterations  t_min[usec]  t_max[usec]  t_typical[usec]  t_avg[usec]
 2       1000         0.94         4.12         0.96             0.97

Take t_typical, not t_avg -- the average is dragged by the tail, and the tail is what you are trying to detect separately.

Two things that will waste your afternoon

Match the CPU frequency on both ends. perftest is sensitive to it, and a server and client running different governors produce a result that looks like a fabric problem and is not. If the two hosts are not configured identically, fix that before you believe any number. This is the same class of error as unpinned GPU clocks in Chapter 8.

Always pass -d. With eight rails in a DGX, perftest with no device flag benchmarks whichever HCA it enumerates first. A "slow fabric" result from an unflagged run is frequently a perfectly healthy fabric measured on the wrong rail. Name the device, and name the same device on both ends.

What perftest prints in the clear

Every perftest run exchanges connection parameters over TCP and prints them:

local address:  LID 0x0d QPN 0x091a PSN 0xacc8f5 RKey 0x085010 VAddr 0x00000000ab5000
remote address: LID 0x1a QPN 0x0a03 PSN 0x11f4c2 RKey 0x0a4021 VAddr 0x00000000cd7000

Worth pausing on, because it is Chapter 23's RKEY exposure threat in a tool you already run. That line contains the queue pair number, the packet sequence number, the remote key, and the remote virtual address -- which is, between them, everything needed to write into that memory region. perftest prints it because it is a benchmark and both ends are you. A production application that leaked the same tuple would have handed over write access to GPU memory.

QP state machine ”” what breaks each transition:

RESET → INIT:   ibv_modify_qp() fails if protection domain wrong
INIT → RTR:     MTU negotiation failure (path MTU < message size) ”” silent BW loss
RTR → RTS:      timeout if remote QP not in RTR state (ordering issue)

Basic connectivity test:

dgx-01:~$ ibv_rc_pingpong -d mlx5_0 -g 0 192.168.100.2
  8192000 bytes in 0.003 seconds = 24530.27 Mbit/sec

dgx-01:~$ ibv_devinfo -d mlx5_0
hca_id: mlx5_0
  state:              PORT_ACTIVE (4)
  max_mtu:            4096 (5)
  active_mtu:         4096 (5)
  link_layer:         Ethernet
  active_speed:       400 Gb/sec

hca_id: mlx5_0
  state:       PORT_ACTIVE (4)
  active_mtu:  4096 (5)
  link_layer:  Ethernet
  active_speed: 400 Gb/sec