Chapter 0 gives the basic formula. This act goes deeper into what practitioners actually have to decide.

The two competing costs

Checkpointing creates two costs that pull in opposite directions.

Checkpoint overhead cost: the time the cluster spends writing the checkpoint, multiplied by the number of GPUs sitting idle during that write. At 1.4 seconds per checkpoint on a 32,000-GPU cluster at $500/GPU-hour, each checkpoint costs:

32,000 GPUs x 1.4 seconds x ($500 / 3600 seconds) = $6,222 per checkpoint.

If you checkpoint every 10 minutes, that is 6 checkpoints per hour x $6,222 = $37,333 per hour in checkpoint overhead.

Recovery cost: when a failure occurs and the cluster rolls back to the last checkpoint, all GPU-hours since that checkpoint are wasted. If you checkpoint every 30 minutes and a failure occurs 29 minutes after the last checkpoint, you waste 29 minutes x 32,000 GPUs x $8.33 per GPU-minute = $7.7 million in wasted compute on that one failure.

The MTBF-driven optimisation

The optimal checkpoint frequency depends on the cluster's mean time between failures (MTBF). For a cluster of N GPUs each with individual MTBF of T hours:

Cluster MTBF = T / N

For a 32,000-GPU cluster with per-GPU MTBF of 10,000 hours: Cluster MTBF = 10,000 / 32,000 = 0.3125 hours = 18.75 minutes.

This means a failure is expected approximately every 19 minutes. With this MTBF, the expected wasted compute between failures at a given checkpoint interval F (in minutes) is F/2 minutes worth of GPU time (average failure occurs halfway between checkpoints).

Checkpoint frequency faster than MTBF is counterproductive: you spend more time checkpointing than you recover from failure avoidance. The practical target is checkpoint frequency between MTBF/4 and MTBF/2. For an 18-minute MTBF, that suggests checkpointing every 5-10 minutes.

What determines checkpoint size

For a model with P parameters, where each parameter is stored as a 16-bit (BF16) float:

Checkpoint size = P x 2 bytes (parameters)

• P x 2 bytes (gradients, if saved)
• P x 4 bytes (optimiser states -- Adam stores momentum and variance, both float32) = P x 8 bytes for a full Adam checkpoint.

For a 70B parameter model: 70 x 10^9 x 8 = 560 GB for a full Adam checkpoint. Teams often save partial checkpoints -- parameters only, not gradients or optimiser states -- for resumption from epoch boundaries, at 140 GB (70B x 2 bytes). The 140 GB number from Chapter 0 is the parameter-only checkpoint.

Mixed-precision training further complicates this: the model computes in BF16 but the master weights kept for the optimiser are FP32, doubling the master weight size. What goes to storage depends on which combination the framework saves.

Gradient accumulation and checkpoint timing

Many large model training runs use gradient accumulation: instead of running AllReduce after every forward/backward pass, the framework accumulates gradients across multiple micro-batches before synchronising. This reduces AllReduce frequency, which reduces compute fabric load. But it also means the model's in-progress gradient state is larger and more complex to checkpoint correctly. A checkpoint taken mid-accumulation must save the accumulated gradients in addition to the model weights -- or restart from the beginning of the current accumulation window after recovery. Frameworks handle this differently; it is a source of subtle correctness bugs after failure recovery.