After applying the full RoCE configuration, verification requires two InfiniBand performance benchmarks: ib_write_bw for throughput and ib_write_lat for latency.
Expected performance at 400GbE:
| Metric | Expected value | Fail threshold |
|---|---|---|
| ib_write_bw | ≥ 46.8 GB/s (99.5% line rate) | < 40 GB/s: MTU issue |
| ib_write_lat median | ≤ 0.9µs | > 2µs: QoS misconfiguration |
| ib_write_lat p99 | ≤ 1.2µs | > 5µs: PFC storm or retransmit |
Running ib_write_bw:
# On dgx-node-02 (server):
dgx-02:~$ ib_write_bw -d mlx5_0 --iters 5000 --size 65536
# On dgx-node-01 (client):
dgx-01:~$ ib_write_bw -d mlx5_0 --iters 5000 --size 65536 192.168.100.2
-------------------------------------------------------------------
#bytes #iterations BW peak[GB/sec] BW average[GB/sec]
65536 5000 46.92 46.81
-------------------------------------------------------------------
QP state machine ”” what breaks each transition:
RESET → INIT: ibv_modify_qp() fails if protection domain wrong
INIT → RTR: MTU negotiation failure (path MTU < message size) ”” silent BW loss
RTR → RTS: timeout if remote QP not in RTR state (ordering issue)
Basic connectivity test:
dgx-01:~$ ibv_rc_pingpong -d mlx5_0 -g 0 192.168.100.2
8192000 bytes in 0.003 seconds = 24530.27 Mbit/sec
dgx-01:~$ ibv_devinfo -d mlx5_0
hca_id: mlx5_0
state: PORT_ACTIVE (4)
max_mtu: 4096 (5)
active_mtu: 4096 (5)
link_layer: Ethernet
active_speed: 400 Gb/sec
INTERACTIVE
RoCE Verification Test Sequence
COMMAND
dgx-01:~$ ibv_devinfo -d mlx5_0
hca_id: mlx5_0 state: PORT_ACTIVE (4) active_mtu: 4096 (5) link_layer: Ethernet active_speed: 400 Gb/sec
✓ INTERPRETATION
PORT_ACTIVE + active_mtu=4096 confirms jumbo MTU negotiated. MTU 5 = 4096 bytes in IB encoding.
Metric: Port state + MTU
Pass: PORT_ACTIVE, mtu=4096