ProDeploy — Learn the fabric that keeps AI clusters alive

GPU form factors: SXM vs PCIe

Toggle to compare connectivity paths and bandwidth in each form factor

GPU-to-GPU bandwidth

SXM: 900 GB/s (NVLink 4)

PCIe: 31.5 GB/s (PCIe Gen 4)

SXM advantage: 28×

NIC-to-GPU (RDMA)

SXM: Full line rate (GDS)

PCIe: PCIe ceiling limited

SXM advantage: ~6×

NVLink Switch eligible

SXM: Yes — full port count

PCIe: No — NVLink absent

Primary workload

SXM: Training at scale

PCIe: Inference / cloud VM

SXM connectivity paths

GPU A (HBM)→ via NVSwitch →GPU B (HBM)

900 GB/s

GPU→ via NVLink → CPU PCIe →ConnectX-7

400 Gbps NIC

DPU→ via NVMe-oF / GDS →NVMe storage

800 Gbps

CPU→ via PCIe Gen 5 (control only) →GPU

63 GB/s

SXM: NVLink handles all GPU-to-GPU traffic. PCIe carries only CPU control traffic and ConnectX-7 storage NIC (Slot1/Slot2) operations — neither on the training critical path.

Every NVIDIA data centre GPU generation ships in at least two form factors: SXM and PCIe. The choice between them is not primarily about raw compute -- both ship the same GPU die -- it is about interconnect capability.

SXM (Server PCI Express Module): a proprietary NVIDIA board form factor that connects the GPU to the motherboard via a high-density Land Grid Array (LGA) socket on the DGX node's baseboard. The key property: the SXM socket exposes the full NVLink port count and high-power delivery (up to 700W for H100 SXM5). SXM GPUs in a DGX node connect to each other via the onboard NVSwitch fabric -- no PCIe bus involved in GPU-to-GPU communication. The CPU still communicates with each GPU via PCIe, but that path is only used for host-initiated operations (DMA, kernel launch, UVM faults), not for the high-bandwidth GPU-to-GPU traffic.

PCIe form factor: standard PCIe add-in card. The GPU connects to the system via PCIe x16. NVLink is either absent (PCIe H100 has no NVLink) or present in reduced form (some Ampere OEM cards have 2 NVLink ports for 2-GPU peer connections). The critical constraint: all GPU-to-GPU communication across any distance must traverse the PCIe bus. For AI training with multiple PCIe GPUs, the collective communication library (NCCL) uses the PCIe or NVLink-peer path within a node and the NIC for inter-node traffic -- but the NIC itself is PCIe-attached, so NIC-to-GPU DMA also goes over PCIe.

The practical consequences for fabric design:

A DGX H100 with 8 SXM5 GPUs: GPU-to-GPU AllReduce within the node uses NVLink 4 at 900 GB/s per GPU -- the ConnectX-7 NICs are not involved for intra-node traffic. The 8 ConnectX-7s (400G each, 3.2 Tbps total) carry only inter-node traffic.

A server with 8 PCIe H100 GPUs (OEM configuration): every GPU communicates via PCIe. A typical 2-socket server has two PCIe root complexes (one per CPU), each handling 4 GPUs. GPUs on different root complexes must communicate CPU-to-CPU via UPI or QPI before reaching the NIC. Effective GPU-to-GPU bandwidth is 31.5 GB/s (PCIe Gen 4 x16 one-directional) versus 900 GB/s for SXM. For large-scale training, PCIe GPU servers require more nodes and more inter-node bandwidth to achieve equivalent performance to a smaller SXM cluster.

When PCIe makes sense: inference serving at small batch size (bandwidth-limited less critical, latency to CPU matters more), cloud multi-tenant workloads where the host OS needs fine-grained PCIe device assignment per VM, and budget-constrained deployments where SXM platform cost is prohibitive. OEM PCIe GPU servers from Dell, Supermicro, and others serve these use cases. For distributed training at scale, SXM is always preferred.