ProDeploy — Learn the fabric that keeps AI clusters alive

In Ch4 you ran ibdiagnet as a one-off diagnostic tool when investigating a specific fault. In production, ibdiagnet is most valuable as a scheduled sweep -- run nightly or weekly, output stored, and results compared against a baseline.

The scheduled sweep pattern

#!/bin/bash
# /opt/fabric/scripts/ibdiagnet-sweep.sh
# Run nightly via cron at 02:00

DATE=$(date +%Y%m%d_%H%M)
OUTDIR=/var/lib/fabric-health/ibdiagnet/${DATE}
mkdir -p ${OUTDIR}

# Full fabric sweep -- BER check, routing, SM, counters
ibdiagnet \
  --ber_threshold 1e-12 \
  --pm_pause_counters \
  --cable_info \
  -o ${OUTDIR}/ibdiagnet_${DATE}

# Extract summary line for trending
grep "FAILED\|MARGINAL\|PASSED" ${OUTDIR}/ibdiagnet_${DATE}.log \
  >> /var/lib/fabric-health/sweep-summary.log

# Alert if any FAILED checks
if grep -q "FAILED" ${OUTDIR}/ibdiagnet_${DATE}.log; then
  echo "ibdiagnet FAILED checks: $(date)" | mail -s "Fabric health alert" ops-team@company.com
fi

# crontab entry
0 2 * * * /opt/fabric/scripts/ibdiagnet-sweep.sh

Interpreting trends, not snapshots

A single ibdiagnet run tells you the state of the fabric right now. A series of runs tells you whether the fabric is healthy, improving, or degrading.

Key trend indicators:

BER (Bit Error Rate) marginal links: A link at 1e-13 today is fine. The same link at 1e-11 next week is degrading -- BER two orders of magnitude worse than baseline means the physical layer is deteriorating. Replace the cable or transceiver before the BER crosses 1e-12 (ibdiagnet's FAILED threshold).

Routing check warnings: Consistent routing warnings about specific GUIDs indicate a subnet manager issue or a persistent link flap that is causing SM reroutes. These do not resolve on their own.

Counter accumulation across sweeps: Compare SymbolErrors totals between runs. A link accumulating 500 errors/night vs 10 errors/night for the same link six weeks ago is telling you something about physical layer ageing.

What nightly sweeps catch that continuous polling misses

Continuous Prometheus polling catches rate-of-change anomalies during training jobs. ibdiagnet nightly sweeps catch:

Routing correctness -- a misconfigured route that reduces effective bisection bandwidth, invisible in counter telemetry
Cable BER at rest -- many marginal cables show BER issues only when the fabric is quiet and there is no burst traffic masking the retransmit patterns
SM configuration drift -- partition key mismatches, incorrect adaptive routing weights, QoS misconfigurations accumulate silently
Dead end paths -- a link that is Active but routing zero traffic due to a subtle SM bug

Treat ibdiagnet sweeps and Prometheus telemetry as complementary, not redundant. Neither replaces the other.