ProDeploy — Learn the fabric that keeps AI clusters alive

← Labs ← Chapter 3: The CLIIdentify the failed rail

Desktop recommendedChecking session

Identify the failed rail

beginner

Elapsed

00:00

Scenario

A 16-node DGX H100 cluster is running distributed training across all 128 GPUs. Throughput has dropped to 87.5% of baseline - exactly 7/8 of expected. GPU 3 (mlx5_3 / eth3) on DGX Node A is no longer contributing to AllReduce. The other 15 nodes are unaffected. Cluster topology: 16 DGX nodes x 8 GPUs = 128 GPUs 8 leaf switches (one per GPU rail, Rail 0-7) Each leaf (SN5600 64-port): 16 active DGX downlinks + 16 active spine uplinks (32 ports unused - cluster is sized for 16 nodes today) Your task: 1. Identify which rail failed and map Rail 3 to the switch port in UFM 2. Confirm DGX-side RDMA and NIC state with native host commands 3. Determine NIC side vs switch side fault 4. Find the exact fault type on the Cumulus leaf interface

Conditions

Rail identified
Link state confirmed
Fault isolated to switch port

Need Help?

Open the step-by-step command guide for this lab.

Topology

Filled green circle = Active

Unfilled red circle = Down

Solid line = 400G link

Dashed line = degraded

Click a device node to open its terminal session

Preparing lab environment...