The GH200 Grace Hopper Superchip is NVIDIA's most significant departure from the conventional CPU-GPU architecture in the DGX H100 generation. It is not a replacement for DGX H100 in scale-out training -- it is a different product targeting a different bottleneck.

The conventional DGX architecture (Ch0 Act 8) has a clear split: 2 Intel Xeon CPUs connected to 8 H100 GPUs via PCIe. The CPUs have access to ~2 TB of DDR5 DRAM; the GPUs each have 80 GB of HBM3. Memory cannot be directly shared -- data must be explicitly transferred via DMA. For workloads that keep large models in GPU HBM this works perfectly. For workloads that need to move data between CPU memory and GPU HBM repeatedly -- large batch inference with huge model state, recommendation systems, or models that exceed GPU HBM capacity -- the PCIe transfer bandwidth (31.5 GB/s per direction) becomes the bottleneck.

The GH200 superchip: a single package containing the Grace CPU (ARM Neoverse V2, 72 cores) and an H100 GPU, connected by NVLink-C2C (Chip-to-Chip) running at 900 GB/s -- the same bandwidth as a full GPU-to-GPU NVLink 4 link, but over a die-to-die connection that is 30x shorter than a PCIe link and supports cache-coherent memory access.

The coherence property is the key difference: the Grace CPU and the H100 GPU share a unified memory address space. The GPU can read CPU DRAM directly at 900 GB/s without an explicit DMA. The CPU can read GPU HBM without an explicit DMA. No cudaMemcpy required for host-device transfers. This is called Unified Memory with cache coherence, enabled by the NVLink-C2C interconnect.

Memory configuration: Grace has 480 GB of LPDDR5X memory (the CPU's DRAM) connected at 500 GB/s aggregate, plus the H100's 96 GB of HBM3. The combined address space is ~576 GB, accessible from either die.

DGX GH200: NVIDIA's rack-scale GH200 system -- 256 GH200 superchips connected by a 3rd-gen NVSwitch fabric and NVLink Switch modules. Total: 256 H100 GPUs + 256 Grace CPUs, 144 TB of combined LPDDR5X + HBM, connected by an NVLink-based all-to-all fabric. This targets very large model inference (models too large for HBM alone) and recommendation systems where CPU memory access patterns matter.

Network implications of GH200 for the fabric engineer:

The GH200 does not change the scale-out fabric fundamentally. Each DGX GH200 node still connects to the spine via ConnectX-7 or equivalent NICs. What changes is the CPU-GPU interface: the 900 GB/s NVLink-C2C replaces the PCIe bus for CPU-to-GPU traffic. The host-side bottleneck (PCIe Gen 4 at 31.5 GB/s) essentially disappears. GPUDirect Storage can now move data from NVMe into HBM without the PCIe ceiling on the GPU side -- the storage ConnectX-7 DMA engine writes directly to the Grace CPU's unified memory space.

The practical impact: for inference serving where the bottleneck was CPU-to-GPU model weight loading, GH200 dramatically reduces that bottleneck. For AllReduce in distributed training, the bottleneck remains the scale-out fabric bandwidth -- GH200's NVLink-C2C does not affect inter-node communication speed.