Skip to content

Chapter 0: How We Got Here -- The Hardware Story

RoCEv2 + Lossless EthernetBeginner40 min read
Building the model…

An HGX B300 node taken apart: eight GPUs on the baseboard, two NVLink switch chips between them, eight ConnectX-8 rail ports at the rear, BlueField-3 for frontend and storage, and the BMC for out-of-band. Click any part to inspect it.

You have spent years managing networks built around servers that look roughly the same. A 1U or 2U chassis. Two CPUs. Some RAM. A handful of drives. One or two NICs running at 10GbE or 25GbE. The NIC is an afterthought -- a commodity part you order in bulk and never think about again.

The switch it connects to is familiar too. You understand spanning tree, ECMP, QoS queues. You can troubleshoot a flapping link without looking at the documentation.

This chapter is about how that familiar world collided with a fundamentally different problem -- and what the industry had to build from scratch to solve it.

A note on the reference platform. The story below is told through the DGX H100, because it is the generation where every one of these design decisions was made explicit and is easiest to see. The build this course targets is a generation newer: HGX B300, eight Blackwell Ultra GPUs per node with 288 GB of HBM3e each, 1.8 TB/s of NVLink per GPU, and one ConnectX-8 SuperNIC per GPU at 800 Gb/s. Every principle in this chapter carries forward unchanged; the numbers double. Where a B300 detail changes an operational conclusion, the text says so.


Read aloudPage 1