Failure domain analysis in a Spectrum-X AI fabric requires thinking at four levels simultaneously: individual link, switch, rail, and collective-participation. These levels have different blast radius and different recovery mechanisms.

A failure domain in this context is the set of compute resources that are affected when a single fault occurs. In a conventional three-tier data centre network, a single top-of-rack switch failure takes all servers in that rack offline. In a rail-optimised Spectrum-X SU, a single leaf switch failure affects 4 NICs (out of 8) on every DGX node in the SU simultaneously — no node loses all connectivity, but all nodes lose half their bandwidth. The training job can continue at reduced efficiency rather than stopping entirely. This is an intentional resiliency trade-off: graceful degradation over hard failure.

Individual link failure is the most common fault in an AI cluster (fibre optic failures, SFP/QSFP transceiver failures, mechanical disturbance of cable trays). The Spectrum-4 adaptive routing engine detects queue depth collapse on a failed link within microseconds — the link goes down, queue depth drops to zero, AR stops routing packets to that next-hop. BGP reconverges to remove the failed link's ECMP next-hop entry. With BFD (Bidirectional Forwarding Detection) configured at 50 ms detection timers on all spine-facing links, the maximum time to detect a silent link failure (one where the physical layer does not signal loss) is 3 × 50 ms = 150 ms. ECMP reconvergence after BFD triggers BGP withdraw takes an additional 100–200 ms with standard BGP timer tuning. Total link failure impact: one collective operation may fail (NCCL timeout), and the training framework (NEMO, Megatron-LM) handles this via elastic training restart.

Spine link failures are higher impact because spine links carry inter-SU traffic. A spine link failure reduces bisection bandwidth between affected leaf pairs. With 4 spine switches in a BasePOD and 4 uplinks per leaf per spine (16 uplinks per leaf total), losing one spine switch removes 25% of uplink capacity. ECMP reconverges across the remaining spines with no configuration change required.

MLAG (Multi-Chassis LAG) on the SN4600C storage switch provides active-active redundancy for the storage fabric. Two SN4600C switches are configured as an MLAG pair — they appear as a single logical switch to the DGX storage NICs and the storage array. Either switch can fail without storage access interruption. The MLAG peer link (a 400 GbE direct interconnect between the two SN4600C switches) carries synchronisation traffic and must be provisioned with adequate bandwidth to carry the full storage load during single-switch operation.

BF3 link-level heartbeats (in B200 deployments) add a 100 ms keepalive between the DPU OS and the network fabric. This heartbeat is distinct from BFD — BFD monitors forwarding plane paths, while BF3 heartbeats monitor the DPU-to-switch control plane channel. If the heartbeat fails, the BF3 DPU can declare an isolated state and flush its reorder buffer, preventing stale out-of-order packets from being delivered to QPs after a link recovery event.

The trade-off between fast failure detection and fast convergence is non-trivial. Aggressive BFD timers (50 ms intervals, 3-packet holddown = 150 ms detection) produce low MTTD but increase CPU load on the switch in proportion to the number of BFD sessions. A BasePOD with 8 leaf switches and 4 spines has 32 spine-facing BFD sessions. At 50 ms transmit interval, each switch generates 20 BFD packets per second per session = 640 BFD packets per second switch-wide. This is trivial overhead. But in a SuperPOD with 64 leaf switches (512 BFD sessions per spine), the BFD overhead becomes measurable.

# Configure BFD on spine-facing links (Cumulus FRR syntax)
leaf-01# nv set vrf default router bgp neighbor swp33 bfd
leaf-01# nv set vrf default router bgp neighbor swp33 bfd detect-multiplier 3
leaf-01# nv set vrf default router bgp neighbor swp33 bfd min-rx-interval 50
leaf-01# nv set vrf default router bgp neighbor swp33 bfd min-tx-interval 50
leaf-01# nv config apply

# Verify BFD session state on all spine uplinks
leaf-01# net show bfd peers
BFD Peers:
  peer 10.1.0.1 vrf default
    ID: 1, Interface: swp33, State: UP
    TX interval: 50ms, RX interval: 50ms, Multiplier: 3
    Last state change: 2024-03-15 14:00:00
  peer 10.1.0.2 vrf default
    ID: 2, Interface: swp34, State: UP
    TX interval: 50ms, RX interval: 50ms, Multiplier: 3

# Simulate spine link failure and measure convergence
leaf-01# ip link set swp33 down
leaf-01# date +%s%3N   # capture ms timestamp
1710511200123

# Watch BGP withdraw propagate
leaf-01# journalctl -fu frr | grep "withdrawn"
Mar 15 14:25:00.247 bgpd[1234]: 10.1.0.1 WITHDRAWN prefix 10.100.0.0/14
# Convergence time: 247ms - 123ms = 124ms ✓ (within target <200ms)