Security without auditability is incomplete. An operator who knows GBP was misconfigured needs to know when it was changed, who changed it, and what the previous state was. UFM and Cumulus Linux both provide the necessary audit infrastructure.

UFM RBAC

UFM Enterprise implements role-based access control with three built-in roles and support for custom roles:

For AI factory operations, the recommended RBAC model is: fabric engineers hold Operator role, security teams hold Monitor role, only the fabric-owner holds Admin. PKey modifications require Admin — and because they can immediately disrupt running training jobs, they should require a second Admin to confirm (four-eyes principle).

# UFM: create a Monitor role user for the security team
curl -s -X POST https://ufm-server:8443/ufm/resources/users \
  -H "Content-Type: application/json" \
  -u admin:password \
  -d '{
    "username": "security-monitor",
    "password": "<strong-password>",
    "role": "Monitor",
    "email": "security@company.com"
  }'

# Verify role assignment:
curl -s https://ufm-server:8443/ufm/resources/users/security-monitor \
  -u admin:password | python3 -m json.tool

Audit trail and syslog

Every UFM configuration change is written to the audit log with user identity, source IP, timestamp, and the exact change applied. The audit log is also forwarded via syslog for archival and SIEM ingestion.

# View UFM audit log:
cat /opt/ufm/log/audit.log | grep -E "PKey|GBP|policy" | tail -20
# Example entries:
# 2026-03-15 09:14:22 [AUDIT] user=fabric-eng ip=10.0.0.50
#   action=PKey_MODIFY partition=TenantA node=0x506b4b0300a1b204
#   change="added to TenantA partition as full member"
# 2026-03-15 09:14:55 [AUDIT] user=fabric-eng ip=10.0.0.50
#   action=PKey_APPLY partition=TenantA status=SUCCESS

# On Cumulus Linux: all NVUE changes are logged
journalctl -u nv-config --since "24 hours ago" \
  | grep -E "apply|commit|user"

PKey change management workflow

Because a PKey misconfiguration can immediately partition a running training job — silently, with no error visible to the application — PKey changes require a careful workflow:

1. Test in NVIDIA Air first: push the proposed partitions.conf to an Air simulation of the fabric and verify all expected QP connections succeed and all cross-tenant QP connections fail.
2. Schedule during maintenance: PKey changes require an OpenSM sweep to propagate. During the sweep, some ports may briefly not be usable. Never apply PKey changes to a fabric running a production training job.
3. Verify after apply: after the SM sweep completes, run smpquery pkeys on representative nodes from each tenant and confirm the partition assignments are correct.
4. Retain rollback config: keep the previous partitions.conf with a timestamp so recovery is a single file replace + SM restart.