Confidential Computing (CC) on the H100 introduces a TEE (Trusted Execution Environment) inside the GPU. When CC mode is enabled, the GPU firmware establishes a secure enclave: all GPU memory is encrypted, the CPU cannot read GPU HBM contents even with physical DMA access, and the host driver cannot inspect model weights, gradients, or intermediate activations. The GPU attestation mechanism allows the tenant to verify that the correct GPU firmware is running before loading secrets into the enclave.

This solves a real problem for cloud providers deploying multi-tenant AI infrastructure: tenants reasonably don't want the cloud operator's host OS to be able to read their model weights or training data. CC mode provides hardware-enforced isolation.

The diagnostic opacity problem:

The same isolation that prevents weight leakage also prevents observability. When a GPU is running in CC mode:

• The host cannot read GPU memory contents. Counter values that are derived from memory (e.g. memory ECC error addresses) are not readable by the host driver.
• DCGM cannot collect certain telemetry counters from CC-mode GPUs. Specifically, counters that require reading GPU memory state (not just register-level counters) are blocked.
• PCIe DMA from host to GPU HBM is blocked. This means GDS bypass paths that rely on host-initiated DMA don't work in CC mode without the tenant explicitly authorising each transfer.
• The NIC cannot perform RDMA writes directly to GPU HBM without the tenant's encryption key being in the path. This affects GDS over NVMe-oF and GPUDirect RDMA in CC deployments.

What still works:

Network-level counters on the ConnectX-7 and switch are unaffected by GPU CC mode. PFC pause rates, ECN marks, symbol errors, and link state are visible to the fabric operator regardless of CC mode. The opacity is at the GPU-memory boundary, not at the network boundary.

DCGM can still collect power, temperature, and SM utilisation counters from CC-mode GPUs -- these are register-level reads that don't cross the memory encryption boundary.

The operational implication:

For a cluster operator running multi-tenant workloads where some tenants enable CC mode, your monitoring strategy must account for degraded visibility into GPU internals. Alert rules that depend on ECC error rate or NVLink BW counters may not fire for CC-enabled GPUs. You should monitor the network-side indicators (NVLink Switch counters via DCGM, scale-out fabric counters via UFM) and combine them with tenant-visible CC attestation logs to correlate GPU-internal events with network-observable symptoms.

For training cluster operators running their own workloads (single-tenant), CC mode is typically not enabled -- the observability loss outweighs the isolation benefit when the operator is also the tenant. CC mode is primarily a cloud provider feature for multi-tenant inference.