Skip to content

Chapter 0: How We Got Here -- The Hardware Story · Part 12 of 15

Act 9 -- How it all wires together

Jump to section

1. Start from what you know2. The normal server3. What AI training actually requires4. Inside one server first -- NVLink5. Act 3B -- What AMD does instead: Infinity Fabric and xGMI6. Why standard Ethernet breaks down7. What the industry tried, and what won8. Why InfiniBand is still chosen9. The NIC that stopped being a NIC10. The server that emerged: DGX H100 anatomy11. Checkpointing and the cost of failure12. How it all wires together13. Why the wiring is rigid by design14. One card, two worlds15. What you now know

Now you can see how a full deployment connects. This is the complete picture:

Three-network architecture — click each network

DGX H100 NODE8× H100 GPUConnectX-7 × 8Slot1/2 CX7BMC (iDRAC)LEAF SWITCHESQM9700 × 8(1 per GPU rail)SPINEQ3400InfiniBandCompute fabricSTORAGESN5600400GbE EthernetStorage fabricMGMT10/25GbE + BMCManagement
Click a network in the diagram or the labels below it to learn more
Official NVIDIA DGX SuperPOD H100 scalable unit rack layout diagram.

Figure: Official NVIDIA DGX SuperPOD H100 scalable unit rack layout. This makes the physical density and repeated rack pattern behind the wiring discussion concrete.

80% complete

Previous

Checkpointing and the cost of failure

Next

Why the wiring is rigid by design