The second question. The fabric is proven; now measure what a real training workload extracts from it.
Use a benchmarking framework with published reference figures, because "fast" is meaningless without a comparison. NVIDIA's DGX Cloud Benchmarking recipes are the common choice: an installer stages container images, datasets and virtual environments; a runner submits jobs to Slurm and parses metrics; and the repository publishes baseline tables collected on the reference architecture for each workload, scale and precision.
# One-time install: images, datasets, venvs, cluster config
$ git clone https://github.com/NVIDIA/dgxc-benchmarking.git
$ cd dgxc-benchmarking && export HF_TOKEN=hf_xxxxxxxx
$ ./install.sh # run inside tmux; this takes hours
# Prescreen: collects CPU/NUMA/container diagnostics before any benchmark
$ llmb-run submit -w microbenchmark_system_info --scale 8
# Unit-scale stress, then the ladder
$ llmb-run submit -w pretrain_nemotron3 -s 30b --dtype fp8 --scale 64
$ llmb-run submit -w pretrain_nemotron3 -s 120b --dtype bf16 --scale 128
$ llmb-run submit -w pretrain_nemotron3 -s 120b --dtype bf16 --scale 512
# One profiled run per scale point for the evidence pack
$ llmb-run submit -w pretrain_nemotron3 -s 120b --dtype bf16 --scale 512 -p
$ llmb-run jobs # find the job id
$ llmb-run jobs log <job_id> --tail 400
Understand what the recipe is measuring. These benchmarks use weak scaling: global batch size grows with GPU count, so per-GPU throughput should stay roughly flat as scale increases. That flatness is the fabric verdict — it is the direct measurement of how much of each GPU's time is being spent waiting for other GPUs. Each run executes a fixed number of steps and averages a window in the middle, excluding warmup, and reports seconds per iteration, tokens per second per GPU, and derived TFLOPs per GPU.
For model FLOPs utilisation you need the platform's peak throughput. For Blackwell Ultra the dense per-GPU figures used by the recipes are BF16 2,250 TFLOPS, FP8 4,500 TFLOPS, NVFP4 13,500 TFLOPS — which is the same silicon that an HGX B300 board presents as 18, 36 and 108 dense PFLOPS across its eight modules.
| Workload criterion | Target |
|---|---|
| Tokens/s/GPU vs published baseline (same recipe version, size, precision, scale) | ≥ 95 % |
| Step-time stability (CV of s/iter across the measurement window) | ≤ 1 % |
| Cross-unit consistency of tokens/s/GPU at unit scale | CV ≤ 2 %; no unit more than 4 % below the best |
| Weak-scaling efficiency (full-scale ÷ unit-scale tokens/s/GPU) | ≥ 95 %, and the fall-off must track the reference curve's shape |
| Sustained run (2 – 4 h at full scale) | Throughput drift ≤ 2 %, no interruption |
| Run hygiene | Zero collective timeouts or restarts; no throttle events in DCGM; fabric telemetry inside the Phase-1 envelope |
Run FP8 as the primary precision where the reference table publishes it. Lower precision moves the same tokens in less compute time, which raises the ratio of communication to computation and puts more pressure on the network per unit of wall clock. It is the harder test, and it is the one that surfaces fabric problems that BF16 hides behind longer compute phases.
Weak-scaling shape matters more than the absolute number. A cluster that lands at 96 % of baseline at every scale point is healthy. A cluster that lands at 99 % on one unit and 91 % at full scale has a spine or congestion problem, even though both numbers are individually respectable — the divergence from the reference curve's own shape is the finding.