The Kubernetes side: two operators, not one
SR-IOV and MIG describe what the hardware can partition. On a Kubernetes cluster there is a second question -- what actually reaches the pod -- and it has an answer worth knowing, because getting it wrong produces a specific and much-misdiagnosed symptom.
The GPU Operator admits GPUs to pods. It does not admit RDMA devices. A separate Network Operator handles the host networking side: the OFED driver stack, the RDMA shared device plugin, and the GPUDirect RDMA path between the NIC and GPU memory. In a full deployment there is usually a third piece, Multus, attaching a secondary interface to the pod, plus an SR-IOV device plugin advertising the virtual functions as schedulable resources.
The failure mode when the Network Operator is missing or misconfigured is not an error. The pod starts. The job runs. NCCL initialises, finds no RDMA device inside the container's network namespace, and falls back to TCP -- so a 400G fabric delivers something closer to a tenth of that, and every switch counter is clean because the fabric is doing exactly what it was asked.
The tell is in the NCCL log, not the fabric: NCCL_DEBUG=INFO names the
transport it selected. If it says Socket where it should say IB, stop
looking at the network and start looking at what the pod was given.
Multi-Instance GPU (MIG) is an H100 feature (also on A100 to a lesser extent) that partitions a single GPU into up to 7 independent instances, each with its own slice of SM compute, HBM memory, and memory bandwidth. From the operating system's perspective, each MIG instance appears as a separate GPU device.
MIG is designed for inference serving in multi-tenant environments: a single H100 can serve 7 independent inference workloads, each isolated from the others in hardware, without scheduling overhead or memory interference.
The network implications are specific to how MIG instances consume NIC bandwidth:
Each MIG instance needs its own network identity if it is serving independent tenants. This is where SR-IOV (Single Root I/O Virtualization) comes in. SR-IOV on a ConnectX-7 allows a single physical NIC to present multiple virtual functions (VFs) to the hypervisor, each with its own MAC address, RoCEv2 queue pair space, and bandwidth allocation. A 7-instance MIG partition can be served by 7 SR-IOV VFs from a single ConnectX-7, each VF with rate limiting to prevent one tenant from saturating the physical NIC.
MIG + SR-IOV configuration in practice:
Each MIG instance runs its own CUDA context. GDS (GPUDirect Storage) from a MIG instance requires the NVMe-oF initiator to route storage traffic to that specific MIG instance's HBM slice -- this requires firmware and driver configuration that maps NVMe-oF queue pairs to MIG geometry. On DGX H100/H200 (where storage uses standard CX7 NICs), this is handled through the host NVMe-oF stack. On DGX B200 (where BlueField-3 DPUs handle storage), the DPU virtual functions must be mapped to MIG instances.
RDMA from a MIG instance to a remote host requires the SR-IOV VF assigned to that instance to have its own queue pairs configured in the firmware. ConnectX-7 supports this natively -- each VF can be allocated a subset of the NIC's QP space.
Bandwidth isolation: without rate limiting on SR-IOV VFs, a single MIG instance running a high-throughput workload can consume the entire physical NIC bandwidth, starving other MIG instances of network access. ConnectX-7 supports per-VF bandwidth limits via MLNX_OFED configuration, allowing operators to guarantee each MIG instance a minimum and maximum network bandwidth allocation.
Monitoring implications (connecting back to Ch11): when MIG is enabled, DCGM metrics are reported per MIG instance, not per physical GPU. The nvidia-smi output shows separate entries for each instance. UFM sees the NIC-level RoCEv2 counters on each SR-IOV VF separately. Alert thresholds calibrated for whole-GPU workloads need to be rescaled for per-instance reporting -- a 7-instance MIG partition where each instance is fully loaded will show 1/7 the GPU utilisation per instance compared to a single-tenant full-GPU job.
SR-IOV without MIG: you can also deploy SR-IOV on ConnectX-7 without MIG, for VM-based multi-tenant inference. Each VM gets an SR-IOV VF and sees a dedicated portion of NIC bandwidth and QP space. This is the standard vGPU + SR-IOV pattern for cloud GPU instances.