Chapter 0 mentions BeeGFS and WEKA in a single line. They are not interchangeable with a standard NAS. Understanding what a parallel file system actually does explains both why it exists and how to size the storage fabric.
Parallel file system striping — 140 GB checkpoint
Drag the slider to see how adding storage nodes increases aggregate write throughput
Why a single storage server fails immediately
A high-end NVMe SSD achieves approximately 7 GB/s sequential read bandwidth. A single enterprise storage server with 24 NVMe SSDs achieves roughly 80-100 GB/s of aggregate read bandwidth.
Now consider the read pattern during distributed training. At the start of each training step, 32 DGX nodes each need to pre-fetch their next batch of training data from shared storage. If those batches are 10 GB each -- typical for large language model training -- 32 nodes x 10 GB = 320 GB must be delivered from storage within the training step window.
With a single storage server at 100 GB/s, delivering 320 GB takes 3.2 seconds. But the compute step that follows might take only 0.8 seconds. The cluster is idle for 3.2 seconds waiting for data, then busy for 0.8 seconds computing. Storage utilisation is 80% of the critical path.
This is the I/O bottleneck that stalled the first generation of large model training. The solution is parallel file systems.
What striping actually does
A parallel file system like BeeGFS or WEKA distributes file data across many storage nodes. When a 140 GB checkpoint is written, the file system divides it into chunks -- typically 1 MB to 256 MB -- and writes each chunk to a different storage node. If there are 16 storage nodes, the 140 GB checkpoint is split into 8.75 GB per node, written in parallel.
From the cluster's perspective, the write bandwidth of the storage tier is the aggregate bandwidth of all storage nodes: 16 nodes x 100 GB/s = 1.6 TB/s theoretical peak. The checkpoint write that takes 1.4 seconds at 800 Gbps storage CX7 bandwidth is now constrained by the storage fabric bandwidth and the storage nodes' aggregate write speed -- not by any single node.
For reads, the same striping applies. Each DGX node's batch read request is served by multiple storage nodes simultaneously. 32 DGX nodes reading 10 GB batches do not all read from the same storage server -- each node's data is striped across many servers, so the reads naturally parallelise.
BeeGFS vs WEKA -- the practical difference
BeeGFS is open-source, originally developed at Fraunhofer Gesellschaft for HPC clusters. It runs on commodity x86 servers. The storage fabric administrators manage the metadata and storage servers directly using standard Linux tooling. BeeGFS is widely deployed in academic and government HPC clusters and is increasingly used in enterprise AI. The operational model is hands-on -- you manage the servers, the disks, the network paths.
WEKA is a commercial parallel file system designed specifically for GPU-intensive AI workloads. It runs on NVMe-based servers and is optimised for the sustained write patterns of checkpoint operations and the random read patterns of dataset pre-fetching. WEKA's storage protocol is purpose-built for low-latency, high-throughput flash access. The operational model is appliance-like -- WEKA manages its own cluster, exposes a POSIX file system interface, and provides dashboards and alerts.
The choice between them is primarily operational: BeeGFS if you have a team comfortable managing Linux storage servers and want open-source flexibility; WEKA if you want a supported appliance model optimised for AI workloads.
The in-cast problem on the storage fabric
One failure mode that parallel file systems introduce on the storage fabric: all 32 DGX nodes starting a batch read simultaneously after an AllReduce barrier creates an in-cast event. All 32 nodes send read requests to the storage fabric at the same moment, causing simultaneous traffic from 32 sources converging toward the storage tier.
This is the same in-cast congestion covered in Chapter 6 for compute traffic -- but on the storage fabric switches instead of the compute fabric switches. Standard ECN and PFC configuration on the storage fabric switches handles this, same as on the compute fabric. The key difference: storage TCP traffic tolerates brief congestion and retransmits without stalling the entire cluster -- unlike compute RDMA traffic, which stalls all GPUs at the barrier. In-cast congestion on the storage fabric slows checkpoint writes or batch reads; it does not immediately stop training.