Debug by bisection, downward first. Re-run the failing test at half the failing scale. If it passes, bisect upward to isolate the unit or spine group responsible. If it fails at half scale too, drop back to Phase-1 collectives at that scale before touching a single training configuration — a training config change made to chase a fabric fault is how a two-day investigation becomes a two-week one.
The usual suspects, in the order they actually occur:
- A straggler node. One GPU thermally throttling, one NIC at reduced rate, one node with a degraded PCIe link. Check DCGM thermal and clock fields and the node's link states before anything else.
- A unit that was marginal in Phase 1. Go back to the consistency table. An outlier unit that was "just inside" the criterion is the first place to look.
- Congestion-control configuration. ECN thresholds inconsistent across a tier, a switch that missed a configuration push, an MTU mismatch on one hop.
- Storage or dataloader stalls misattributed to the fabric. A profiled run settles this immediately — if the gap in the timeline is before the collective rather than inside it, the compute fabric is not the problem.
A profiled run per scale point is the tiebreaker for all of the above. When a number is disputed, the communication-versus-computation overlap in the trace is the ground truth, and it is much easier to capture during the campaign than to reconstruct afterwards.
The evidence pack is the deliverable, and it should be assembled as you go rather than reconstructed at the end. For each gate: the raw logs, the parsed metric tables, the switch telemetry exports for the same windows, the configuration and firmware manifest snapshot, and a comparison table per scale point showing measured value, reference value, delta percentage, and pass or fail. Any point below target gets a root-cause note referencing the Phase-1 data and the profile trace. Assembled once, that same archive serves both customer acceptance and any vendor validation programme you are pursuing — there is no reason to do the work twice.