Lab 4 presents a topology decision scenario. A team is deploying 64 DGX H100 nodes and has received two proposals from different integrators. The learner must evaluate which supports full AllReduce bandwidth and which does not -- using the calculations from this chapter.

Proposal A: 32-port switches, 3 stages

Leaf switches:    32 switches
  Downlinks:       16 ports x 400G to DGX nodes
  Uplinks:         16 ports x 400G to spine

Spine switches:  16 switches
  Downlinks:       32 ports x 400G to leaf
  Uplinks:         0 (2-stage only)

Wait -- this requires 16 uplinks per leaf, but only 16 spine switches.
Each leaf sends 1 uplink to each spine. 32 leafs x 16 uplinks = 512 uplink connections.
16 spine switches x 32 ports = 512 connections. Matches. [ok]

Oversubscription check:
  Leaf downlinks:   32 switches x 16 ports x 400G = 204.8 Tb/s
  Leaf uplinks:     32 switches x 16 ports x 400G = 204.8 Tb/s
  Ratio: 1:1 -> non-blocking [ok]

Total switches: 32 leaf + 16 spine = 48 switches
AllReduce hops: 2 (non-blocking, direct path)

Proposal B: 64-port switches, 2 stages

Leaf switches:    8 switches (one per GPU rail)
  Downlinks:       32 ports x 400G to DGX nodes
  Uplinks:         32 ports x 400G to spine

Spine switches:  32 switches
  Downlinks:       8 ports x 400G to leaf

Wait -- 64-node cluster, 8 NICs per node = 512 NICs.
8 leaf switches with 32 downlinks each = 256 NIC connections. Only 256 of 512 are connected.
This configuration can only support 32 nodes, not 64. [x]

Proposal B as stated has an error -- 8 leaf switches with 32 downlinks each gives only 256 NIC connections, insufficient for 64 nodes x 8 NICs = 512 connections. The vendor specified too few leaf switches for the stated node count. Correcting this to 16 leaf switches would provide the 512 connections needed. But then 16 leaf x 32 uplinks = 512 uplinks to spine, requiring 32 spine switches with 16 active ports each. With 64-port spine switches, 48 of 64 ports on every spine switch would be dark -- 75% port waste. This is technically functional but economically irrational: you are paying for 64-port switch hardware while using fewer than 16 ports. The actual root failure of Proposal B is not switch radix -- it is that the vendor quoted the wrong leaf count for the node target.

The lab exercises this calculation: given a node count and switch radix, determine how many switches are needed, whether the design is non-blocking, and what the AllReduce hop count will be. This is the core skill a network engineer needs before committing to a hardware order for a new AI cluster.