ProDeploy — Learn the fabric that keeps AI clusters alive

PCIe bandwidth ceiling

Select PCIe generation and NIC speed to see how the bandwidth ceiling fills up (without GDS bypass)

PCIe generation

NIC speed

PCIe Gen 4 (x16) — 31.5 GB/s per direction

NIC inbound (400G NIC)100% used

50.0 GB/s used of 31.5 GB/s ceiling — SATURATED

GPU control traffic13% used

4.0 GB/s used of 31.5 GB/s ceiling

Total (without GDS)100% used

104.0 GB/s used of 31.5 GB/s ceiling — SATURATED

Summary

PCIe ceiling (1-dir)31.5 GB/s

NIC demand50 GB/s

Without GDS: total104 GB/s

GDS bypasses cudaMemcpyRequired

PCIe Gen 4 is saturated by a 400G NIC without GDS. Enable GPUDirect RDMA to bypass host RAM copies.

SXM GPUs avoid the bottleneck differently: GPUDirect Storage uses the Slot1/Slot2 ConnectX-7 NIC over PCIe Gen5 to DMA directly into GPU HBM, eliminating system RAM copies. PCIe Gen5 x16 (64 GB/s) is wide enough that the NIC, not PCIe, is the limit. Control traffic is the only non-data PCIe consumer.

PCIe bandwidth is the hidden bottleneck in non-SXM GPU deployments. Understanding it requires knowing the actual numbers at each generation.

PCIe Gen 4 (x16 slot): 31.5 GB/s per direction (63 GB/s bidirectional). This is the standard for servers shipped with A100 PCIe and H100 PCIe GPUs. A ConnectX-7 running 400 Gbps Ethernet requires 50 GB/s of host memory bandwidth. That exceeds the one-directional PCIe Gen 4 ceiling. In practice, DMA operations from NIC to GPU memory do not go through host RAM (GDS bypasses this), but any copy-path traffic competes for PCIe bandwidth with GPU-CPU command traffic.

PCIe Gen 5 (x16 slot): 63 GB/s per direction (126 GB/s bidirectional). Doubles Gen 4. A 400 Gbps NIC now fits comfortably. Two 400 Gbps NICs (800 Gbps = 100 GB/s combined) nearly hits the ceiling in one direction. This is the target for server platforms co-designed with B200 PCIe GPUs and 800G ConnectX-8 NICs.

PCIe Gen 6 (emerging, 2025+): 126 GB/s per direction. Targets 800 Gbps NICs at full line rate without PCIe bottleneck.

The ceiling manifests in production in three ways:

First, NIC-to-GPU DMA without GDS: if GPUDirect Storage or GPUDirect RDMA is not enabled, every packet received by the NIC must traverse PCIe from the NIC to host RAM and then again from host RAM to GPU HBM. A 400G link generates 50 GB/s of inbound data -- that is 100 GB/s of PCIe traffic (NIC->RAM + RAM->GPU), more than Gen 4 can handle. This is why GDS (covered in Chapter 10) is not optional for high-throughput storage or network workloads with PCIe GPUs.

Second, concurrent CPU and GPU memory access: in PCIe GPU servers, the CPU and GPU share the PCIe bandwidth to DRAM. If the CPU is running a data preprocessing pipeline while the GPU is performing DMA, they compete on the same PCIe lanes.

Third, p2p GPU communication within a server: NCCL's p2p transport between PCIe GPUs without NVLink must use the PCIe bus. For an AllReduce across 8 PCIe GPUs in one server, traffic bottlenecks at the PCIe root complex even if each GPU-to-GPU path is only x16.

None of these constraints exist in SXM deployments because the NVLink fabric is entirely off-PCIe for GPU-to-GPU traffic. The PCIe bus in a DGX H100 is used only for CPU-to-GPU control traffic and NVMe-oF storage operations via the Slot1/Slot2 dual-port ConnectX-7 cards -- neither of which is on the critical path for training throughput.