Skip to content

Chapter 1: Operating Systems, Management Platforms, and First Power-On · Part 4 of 9

Is it all CLI? Or all orchestrated?

The honest answer is: both, depending on what you are doing.

CLI vs GUI vs orchestrated — by operational phase

CLI
GUI
Orchestrated
API
GUI
Monitor GPU utilisation per job
BCM dashboard / DCGM
GUI
Check fabric health overview
UFM dashboard
Orchestrated
Submit training jobs
Slurm / Kubernetes
CLI
Check switch error counters
ONYX: show ib counters
CLI
Inspect PFC state on a specific port
Spectrum: show dcb pfc
GUI
View per-port bandwidth utilisation
UFM topology view
CLI
Check RDMA link state on a node
DGX: ibstat / rdma link show
Orchestrated
Upgrade switch firmware (all switches)
UFM or Ansible

Day 0 (initial setup): Mostly orchestrated. You use BCM to provision DGX OS. UFM discovers the fabric automatically. You configure Spectrum-X switches via NCLU templates or SONiC config push. Very little manual CLI work -- the platforms do the heavy lifting.

Day 1 (operations): Mixed. Monitoring happens through UFM GUI, BCM dashboard, and Grafana dashboards connected to DCGM. Diagnostics require CLI -- you SSH into switches to check port state, run show ib counters, use ibdiagnet from a management host. Job scheduling is entirely through Slurm or Kubernetes.

Day 2 (troubleshooting): Mostly CLI. When a GPU rail goes dark, a PFC storm starts, or AllReduce latency spikes, you are in SSH sessions on switches and DGX nodes running diagnostic commands. UFM helps you correlate what you find with the fabric topology. But the raw data comes from CLIs.

What you cannot do from CLI:

  • Change UFM's routing algorithm selection (requires UFM GUI or API)
  • Provision a new DGX OS image (requires BCM)
  • Monitor per-job GPU utilisation (requires DCGM or NGC dashboards)
  • Upgrade switch firmware at scale (requires UFM or Ansible)

What you can only do from CLI:

  • Real-time per-port error counter inspection
  • Fabric loop detection and diagnosis
  • Cable testing and qualification
  • Detailed PFC/ECN counters that UFM does not surface
  • Any urgent change when the management plane is unavailable