The most common mistake in AI fabric monitoring is copying threshold values from generic server monitoring playbooks. An HPC fabric is not a generic server environment. Thresholds that generate actionable signal in one context produce noise in the other.
Alert Threshold Calibration
Adjust the threshold and observe the false-positive / false-negative trade-off. Rates shown are illustrative events per day.
Good range
8–12%: catches sustained congestion (DCQCN misconfiguration, sender not backing off) while ignoring normal burst absorption.
PFC pause rate
Wrong threshold: alert on any non-zero PFC pause frames.
In a correctly configured RoCEv2 fabric under load, PFC pause frames are expected. PFC is the backpressure mechanism -- it is doing its job when it fires occasionally during burst absorption. Zero PFC in a 400G fabric under AI training load may actually indicate ECN is absorbing all congestion before PFC fires (good) or that your PFC is misconfigured and not actually enabled (check with show dcb pfc).
Right threshold: alert when PFC pause duration exceeds a rate that indicates the fabric is not draining. A reasonable starting rule:
# PFC: alert when pause duration > 10% of total link time, sustained for 2 minutes
expr: rate(pfc_pause_duration_total[2m]) / rate(link_up_time_total[2m]) > 0.10
for: 2m
Sustained PFC at >10% of link time means the pause propagation is not resolving -- likely a misconfigured DCQCN or a sender that is not responding to ECN signals.
Symbol error rate -- trend, not absolute
Wrong threshold: alert when SymbolErrors > 0.
Symbol errors are non-zero on almost every link in a large InfiniBand fabric. Cosmic rays, thermal noise, and marginal transceivers produce occasional errors that the FEC layer corrects transparently. The absolute count is meaningless on its own.
Right threshold: alert on rate of change. A link that has had 200 total symbol errors over 60 days is fine. The same link accumulating 200 symbol errors in 10 minutes is degrading.
# Symbol errors: alert on rate > 10/min sustained
expr: rate(ufm_port_symbol_errors_total[10m]) > 0.17 # 10/min = 0.17/s
for: 5m
Also: correlate with the link's error history baseline. A link that has always been at 2/min and is now at 8/min is more concerning than a link that has been stable at 5/min for months.
Buffer utilisation
Buffer occupancy is only meaningful during bursty traffic. A 30% buffer utilisation reading on a switch port that carries a 400G GPU-to-leaf link is normal during AllReduce scatter phases. The signal you want is sustained high buffer occupancy, which means traffic is not draining.
# Buffer: alert on sustained >60% occupancy for >1 min
expr: switch_buffer_utilisation_percent > 60
for: 1m
Link flap frequency
A link that flaps once is physical layer noise. A link that flaps 3 times in an hour is a degrading cable. A link that flaps 10 times in an hour is down.
# One flap: informational. Three in an hour: warning.
expr: increase(port_state_changes_total[1h]) >= 3
for: 0m # fire immediately
severity: warning
The false-positive cost
Every spurious alert trains your team to ignore alerts. In a 256-GPU cluster monitored by one or two fabric engineers, alert fatigue is a serious operational risk. Calibrate thresholds conservatively: it is better to miss a marginal link for a few hours than to desensitise your team with 40 false alarms per week.
A useful metric: track alert-to-action ratio. If fewer than 30% of your alerts result in a configuration change or hardware replacement within 48 hours, your thresholds are too sensitive.