Skip to content

AI Networking Security · Part 6 of 9

Act 6 — UFM Cyber-AI Platform

The UFM (Unified Fabric Manager) product family has three editions with distinct purposes. UFM Enterprise manages fabric routing, PKey assignments, and device configuration. UFM Telemetry streams per-port metrics to Prometheus and Grafana. UFM Cyber-AI is the anomaly detection engine — it is passive monitoring only and never modifies fabric state.

Understanding which edition does what prevents a common operational mistake: engineers who expect UFM Telemetry to alert on security anomalies (it does not — it exports raw counters), or who expect UFM Cyber-AI to automatically remediate issues (it does not — it detects and alerts, humans remediate).

How Cyber-AI learns the baseline

UFM Cyber-AI operates in two phases. During the learning phase (7–14 days by default), it collects port counters, traffic patterns, error rates, and timing distributions from every switch port and HCA in the fabric. It builds a statistical baseline: what is normal throughput for each port at each hour of the day, what is the normal PFC pause rate, what is the normal symbol error rate.

After the learning phase, the detection phase begins. Live telemetry is compared continuously against the baseline. NVIDIA's ML model flags deviations that exceed learned thresholds — accounting for daily and weekly periodicity (a higher traffic burst on Monday morning is normal; the same burst at 3am Saturday is anomalous).

Alert taxonomy

UFM Cyber-AI classifies alerts into three categories:

Hardware anomalies: port symbol error rate spikes, link flap patterns, or unusual BER (Bit Error Rate) distributions. These indicate cable, optic, or connector degradation. A symbol error rate spike on a single port crossing 3σ above baseline is typically a dirty QSFP connector; crossing 8σ and sustained means the optic is failing.

Traffic anomalies: unusual traffic flow patterns — all-to-one from a single source (potential data exfiltration), all-from-one to multiple destinations (bulk unauthorised data copy), or a sudden sustained throughput to destinations outside the normal job communication pattern.

Fabric health anomalies: coordinated changes across multiple ports — a synchronised PFC pause increase across 8 ports suggests incast or PFC misconfiguration; a pattern of SM communication failures suggests routing table corruption or a rogue SM.

INTERACTIVE
UFM Cyber-AI — Anomaly Detection Dashboard
Active events:1 Critical1 High1 Medium
0204060800h6h12h18h24hbaseline!error rate
CRITICALHardware — Degraded optic detected
Affected: leaf-02 swp7
Port symbol error rate has spiked 8σ above baseline over the past 4 minutes. This pattern is consistent with a dirty or failing QSFP28 optic. The link has not gone down yet but is at risk of err-disable.
Recommended action: Run ibdiagnet -r to confirm BER rate. Check physical connector and clean QSFP if accessible. Schedule optic replacement at next maintenance window. Reference: Lab 9 err-disable recovery.

SIEM integration

UFM Cyber-AI exports all alert events via syslog (RFC 5424 format) and a REST API. The syslog stream is compatible with Splunk, ELK Stack, and Graylog out of the box. The REST API follows the UFM REST conventions (same as UFM Enterprise).

# Configure syslog export from UFM Cyber-AI:
# In /opt/ufm/conf/cyber_ai.conf:
[syslog]
enabled = true
server = 10.0.0.100
port = 514
facility = LOCAL0
severity_filter = MEDIUM,HIGH,CRITICAL

# Example syslog output for a traffic anomaly:
# <133>1 2026-03-15T02:14:38Z ufm-server UFMCyberAI - - -
#   [ALERT] node=0x506b4b0300a1b202 port=1 type=TRAFFIC_ANOMALY
#   severity=HIGH detail="all-to-one pattern 380Gbps sustained 22min
#   dst=10.50.0.1 outside normal job communication matrix"

# Query via REST API:
curl -s -u admin:password https://ufm-server:8443/ufm/resources/cyber_ai/events \
  | python3 -m json.tool