The NVMe SSDs and the separate storage fabric exist for a reason that is easy to overlook: at cluster scale, hardware failure is not an edge case. It is a scheduled event.
Consider the math. A DGX SuperPOD has 32 nodes x 8 GPUs = 256 GPUs. A large training cluster might have 4,000 nodes x 8 GPUs = 32,000 GPUs. If each GPU has a mean time between failure (MTBF) of 10,000 hours -- optimistic for a system under constant maximum load -- then a cluster of 32,000 GPUs experiences a GPU failure every 18 minutes on average.
When a GPU fails, the training job crashes. Every other GPU in the job stops. The cluster rolls back to the last saved checkpoint and resumes from there. The time between the last checkpoint and the failure is wasted compute.
Checkpoint time = (checkpoint size) / (storage fabric bandwidth)
For a 70B parameter model, a full checkpoint is roughly 140 GB. At 800 Gb/s storage fabric bandwidth per node (2x dual-port ConnectX-7 cards at 400G each, used for the storage and in-band management fabric), writing a checkpoint takes approximately 1.4 seconds. Teams typically checkpoint every 10-30 minutes. When a failure occurs, you lose at most 30 minutes of GPU time -- on a cluster that costs $500 per GPU-hour, that is $250,000 of wasted compute per failure on a 32,000-GPU job.
This is why the storage fabric is fast, why it is separate from the compute fabric, and why GPUDirect Storage (which allows GPUs to write directly to NVMe-oF targets without CPU involvement) matters. Every second of checkpoint time is multiplied by thousands of GPUs sitting idle.
The storage bandwidth available to the cluster determines how quickly those seconds add up:
| Storage bandwidth per node | Time to write 140 GB checkpoint | Notes |
|---|---|---|
| 100 Gbps (12.5 GB/s) | ~11.2 seconds | Single-NIC storage -- common in legacy HPC |
| 200 Gbps (25 GB/s) | ~5.6 seconds | Single dual-port CX7 storage card |
| 800 Gbps (100 GB/s) | ~1.4 seconds | DGX H100 target -- 2x dual-port ConnectX-7 storage cards |
| 3.2 Tbps (400 GB/s) | ~0.35 seconds | Full DGX node aggregate (theoretical) |
At 32,000 GPUs and a 30-minute checkpoint interval, reducing checkpoint time from 11 seconds to 1.4 seconds saves approximately 3.1 GPU-hours per checkpoint -- roughly $1,500 at $500/GPU-hour. Across 48 checkpoints per day, the storage fabric investment pays for itself in wasted-compute terms within weeks.
Why DGX and not something else?
There are alternatives. HPE Cray supercomputer nodes, AMD Instinct MI300X servers, Google TPU pods. Each has real strengths. But for the purposes of this course, DGX is the right reference platform because:
- It is the most widely deployed AI training platform in enterprise data centers
- NVIDIA's vertical integration (GPU + NIC + switch + software) means the reference architectures are well-documented
- The InfiniBand and RoCEv2 concepts you learn here apply regardless of which hardware you eventually work with