Storage in an AI factory is not a single tier — it is three separate tiers with different hardware, protocols, and performance characteristics, each serving a different phase of the training lifecycle. Treating storage as an afterthought until the cluster is built is a common operational mistake that manifests as GPU idle time during checkpoint and dataset load.

Local NVMe in each DGX node is primarily the OS boot drive. DGX H100 ships with approximately 3.84 TB of local NVMe in a RAID-1 configuration — sufficient for the OS, Docker images, and application binaries, but not for training datasets or model checkpoints. Local NVMe is invisible to the network fabric. Never size a training dataset to fit on local NVMe; it will not scale past single-node training.

NVMe-oF over RDMA (GPUDirect Storage) eliminates the CPU data copy on the read path between storage and GPU HBM. The traditional path is: storage → PCIe → CPU DRAM → PCIe → GPU HBM, with two PCIe traversals and one CPU copy. GPUDirect Storage directly maps a storage NVMe namespace into GPU HBM address space via RDMA, bypassing CPU DRAM entirely. The DGX ConnectX-7 NIC terminates both compute RDMA (to other DGX nodes) and storage RDMA (to the storage system) — port selection is via DSCP marking and the dedicated storage leaf switch. DSCP 10 (0x0A) identifies checkpoint-to-storage traffic (see PRODEPLOY_STRATEGY_COMPACT.md DSCP table).

Parallel file systems are the workhorse storage tier for training datasets and distributed checkpointing. Lustre is the most widely deployed in AI clusters, providing POSIX semantics, stripe-based parallelism across object storage targets (OSTs), and mature RDMA transport (Lustre over LNet). IBM Spectrum Scale (GPFS) offers strong metadata performance and is common in enterprise deployments. VAST Data and WekaIO are newer entrants with NVMe-based all-flash backends and POSIX+S3 dual-protocol support — WekaIO in particular is popular for clusters that also serve model inference directly from the training storage tier.

The bandwidth mismatch between compute and storage is fundamental: 8 DGX H100 nodes at full utilisation present 8 × 8 NICs × 50 GB/s NIC throughput = 3.2 TB/s aggregate compute bandwidth. No cost-effective storage system delivers 3.2 TB/s of write bandwidth continuously. Realistic checkpoint bandwidth for a 70B parameter model at bf16 precision is 140 GB of gradient state per checkpoint event. At a checkpoint frequency of once per 500 iterations (roughly every 5 minutes at typical batch sizes), effective checkpoint bandwidth demand is approximately 470 MB/s sustained — well within the capability of a Lustre cluster with 4–8 OSTs backed by NVMe drives.

Storage leaf design: each Scalable Unit has one dedicated SN4600C storage leaf switch. The SN4600C connects downward to the DGX storage NICs (typically CX7 port 7, dedicated to storage) and upward to the storage array network. Separating storage traffic onto a dedicated switch prevents a checkpoint storm from impacting the training fabric: storage ECN events cannot propagate pause frames onto the training fabric because the two switch domains are physically separate.

# Verify GPUDirect Storage is active and using RDMA path
dgx-01$ gds_stat -p
GDS version: 1.7.2
GDS driver version: 515.105
Detected RDMA capable network: yes
Active network interface: mlx5_7 (storage NIC, ConnectX-7)
Current GDS configuration: enabled

# Check storage NIC RDMA stats (mlx5_7 = NIC 7, dedicated to storage)
dgx-01$ ethtool -S mlx5_7 | grep -E "rx_bytes|tx_bytes|rdma"
     rx_bytes: 142832941568
     tx_bytes:   8429184000
     rdma_read_requests:    1,247,832
     rdma_write_requests:     892,441

# Measure checkpoint write bandwidth to Lustre
dgx-01$ ior -a POSIX -w -r -o /lustre/checkpoints/test.ckpt \
  -b 4g -t 1m -g -F -e -k
Results:
  write: 423.85 MiB/s
  read:  891.22 MiB/s