Skip to content

Chapter 17: Storage Network Packet Path

HPC Networking FoundationsAdvanced55 min read

The GPU just finished a training step. Before the next one begins, the framework writes a checkpoint -- gigabytes of model weights flushed from HBM to persistent storage. That write crosses a network fabric you have not traced at packet level yet. Chapter 10 showed you the architecture. This chapter shows you the bytes.

You already know how a compute RoCEv2 packet travels from one GPU to another (Ch16). The storage path reuses the same NIC silicon and many of the same protocol layers, but the software above it is completely different. The command that arrives at a storage appliance is not a NCCL message -- it is an NVMe command capsule wrapped in an RDMA transfer. The fabric it crosses does not need PFC in most designs. The congestion thresholds are set for a different traffic profile.

By the end of this chapter you will be able to trace a checkpoint write from HBM through PCIe through the CX7 NIC through the storage switch to the NVMe-oF target, name every header along the way, and diagnose failures with the right commands on the right devices.