Chapter 4 spent a whole section on the Subnet Manager, because on InfiniBand the SM is the thing that turns a pile of cables into a fabric. It discovers topology, assigns LIDs, computes routing tables, and pushes them into every switch. When it is not running, nothing forwards.

The NVLink fabric has an exact counterpart, and almost nobody outside the DGX world can name it: NVSwitch Fabric Manager, the nvidia-fabricmanager service.

Fabric Manager does three jobs that should sound familiar by now:

• It configures routing among NVSwitch ports -- the NVLink equivalent of the SM computing forwarding tables.
• It coordinates with the GPU driver to initialise GPUs into the NVLink fabric, which is what makes a GPU addressable by its peers across the switch.
• It monitors the fabric for NVLink and NVSwitch errors, which is where the counters in Act 6 originate.

On first- and second-generation NVSwitch systems it does a fourth job as well: it trains the links. Newer systems support ALI -- Autonomous Link Initialization -- where the hardware brings NVLink up on its own. Without ALI, Fabric Manager has to explicitly train both the NVSwitch-to-NVSwitch and the NVSwitch-to-GPU interconnects, which is why older NVSwitch generations have a link-training failure mode that newer ones simply do not have.

The failure that does not look like a network failure

This is the reason a network engineer needs to know the service exists.

When Fabric Manager has not finished initialising, CUDA does not report a fabric problem. It reports:

cudaErrorSystemNotReady

An application launched before Fabric Manager completes will fail with that error and nothing else. There is no link-down message, no port counter, no SM log entry, because none of those layers is involved. A job that dies this way looks like a broken application to the ML team and like nothing at all to the fabric team -- which is exactly how a node ends up bounced between two groups for a day.

Checking it

The fabric state is exposed through nvidia-smi, not through any of the IB tooling:

# Is this GPU registered with the NVLink fabric?
nvidia-smi -q -i 0 | grep -i -A 2 Fabric

# Healthy output:
#     Fabric
#         State                     : Completed
#         Status                    : Success

# The service itself
sudo systemctl status nvidia-fabricmanager
sudo journalctl -u nvidia-fabricmanager

# Start / stop / persist across reboots
sudo systemctl start nvidia-fabricmanager
sudo systemctl enable nvidia-fabricmanager

A State other than Completed means the GPU is not yet part of the fabric. Treat it exactly as you would treat a port that has not come up: nothing above it will work, and there is no point testing anything above it until it does.

DGX versus HGX

The packaging difference matters operationally, because it decides whether this is ever your problem:

On an HGX platform the package is versioned against the driver branch:

sudo apt-get install cuda-drivers-fabricmanager-<driver-branch>

There are two flavours. The base nvidia-fabricmanager package is the one bare-metal clusters need -- the service, its unit file, and the topology files. The nvidia-fabricmanager-devel package adds the shared-NVSwitch and vGPU multi-tenancy interfaces, and is only required if you are partitioning the NVLink fabric between virtual machines.

The practical consequence: on a mixed estate, the DGX nodes come up clean and the HGX nodes fail with cudaErrorSystemNotReady until somebody installs a service nobody thought to check, because on every other node it was already there.