23 Jul, 2026
How to Install and Configure the NVIDIA GPU Operator on Kubernetes
VMware , , ,

TL;DR

The NVIDIA GPU Operator automates the software stack required to make GPUs usable by Kubernetes workloads. It can deploy and manage NVIDIA drivers, the NVIDIA Container Toolkit, the Kubernetes device plugin, GPU Feature Discovery, DCGM Exporter, MIG Manager, and validation components.

A successful installation requires more than running a Helm command. The GPU hardware must be visible to the operating system, the Kubernetes and Linux versions must be supported, the driver ownership model must be decided, the container runtime must be compatible, and the device plugin must successfully advertise nvidia.com/gpu resources to Kubernetes.

This guide installs a pinned GPU Operator release, validates the node software stack, runs a CUDA workload, and establishes a practical troubleshooting and upgrade workflow.

Introduction

Adding a GPU-equipped server to a Kubernetes cluster does not automatically make the GPU available to containers.

Kubernetes can schedule CPU and memory without vendor-specific software, but GPUs require several additional layers. The Linux operating system needs a compatible NVIDIA driver. The container runtime needs a supported device-injection path. Kubernetes needs a device plugin that can advertise GPU resources to the scheduler. Operators also need monitoring, validation, labeling, and a controlled driver lifecycle.

Installing each component separately creates an operational dependency chain that is difficult to maintain consistently across a cluster. A driver update can break the container runtime. A kernel update can prevent the driver module from loading. A missing device plugin can leave healthy GPUs invisible to the scheduler.

The NVIDIA GPU Operator addresses this problem by managing the GPU software stack through Kubernetes resources and DaemonSets.

This article focuses on a conventional Kubernetes cluster running Linux GPU nodes on bare metal or virtual machines with GPU passthrough. NVIDIA vGPU, OpenShift, managed cloud Kubernetes services, confidential containers, and KubeVirt introduce additional platform-specific requirements that should be validated separately.

What You Will Accomplish

By the end of this tutorial, you will be able to:

  • Check whether the Kubernetes platform, operating system, and runtime are suitable for the GPU Operator.
  • Prepare GPU worker nodes for driver installation.
  • Choose between Operator-managed, host-installed, and precompiled drivers.
  • Install a pinned GPU Operator release using Helm.
  • Verify the driver, toolkit, device plugin, GPU discovery, and telemetry components.
  • Confirm that Kubernetes advertises nvidia.com/gpu resources.
  • Run a CUDA validation workload.
  • Plan Operator and driver upgrades.
  • Troubleshoot pods that cannot access GPUs.

How the GPU Node Software Stack Works

The GPU Operator does not replace Kubernetes scheduling. It prepares the node so that Kubernetes can understand and allocate the GPU.

The important point in the following diagram is that each layer depends on the layer below it. When a workload cannot access a GPU, troubleshooting should start at the hardware and driver layers rather than at the application.

The GPU Operator manages most of these layers through a ClusterPolicy resource. Node Feature Discovery identifies NVIDIA hardware, the driver DaemonSet prepares the kernel, the toolkit configures device injection, and the device plugin exposes schedulable resources.

Check the Supported Platform Before Installation

Do not assume that every combination of Linux, Kubernetes, kernel, runtime, and driver is supported.

For the current 26.3 release line, NVIDIA validates several mainstream Linux distributions against Kubernetes 1.32 through 1.36. The exact range varies by operating system and Kubernetes distribution.

The current matrix also validates containerd 1.7 through 2.2 and CRI-O on supported operating systems. Additional features such as the Node Resource Interface plugin have stricter minimum runtime versions.

Use the following checklist before making changes.

Check What to Confirm
GPU hardware The GPU model is supported and visible through PCI or the virtualization platform
Kubernetes The cluster version appears in the current GPU Operator platform matrix
Operating system The GPU node distribution and release are supported
Kernel The running kernel has compatible headers, packages, or precompiled driver support
Container runtime containerd or CRI-O is running at a supported version
Deployment model Bare metal, PCI passthrough, or a supported vGPU design
Security configuration Secure Boot, SELinux, and Pod Security settings are understood
Image access Nodes can reach required container registries and operating system repositories
Existing software Driver and toolkit ownership is known before the Operator is installed

Run these basic cluster checks from an administrative workstation:

kubectl version
kubectl get nodes -o wide
kubectl get nodes -o custom-columns=NAME:.metadata.name,OS:.status.nodeInfo.osImage,KERNEL:.status.nodeInfo.kernelVersion,RUNTIME:.status.nodeInfo.containerRuntimeVersion

On each intended GPU node, verify that the hardware is visible:

lspci -nn | grep -i NVIDIA
uname -r

If lspci does not show the GPU, the GPU Operator cannot fix the problem. Investigate BIOS configuration, PCI passthrough, hypervisor assignment, cloud instance configuration, or physical hardware installation first.

Prepare the GPU Worker Nodes

Node preparation is where many GPU Operator installations succeed or fail.

Stabilize the Kernel Lifecycle

A driver container must build, load, or provide kernel modules that match the running Linux kernel. Uncontrolled kernel updates can therefore break a previously healthy GPU node.

Check the running kernel:

uname -r

For Ubuntu nodes, confirm that matching kernel headers are available:

dpkg -l | grep "linux-headers-$(uname -r)" || true

For RHEL-compatible nodes, check the matching development package:

rpm -q "kernel-devel-$(uname -r)" || true

If the packages are not installed, make sure the node can retrieve them from an approved operating system repository. In restricted environments, mirror the required packages or use a validated precompiled driver strategy.

Do not allow an automated kernel update policy to move GPU nodes to a kernel that has not been tested with the selected driver branch.

Check for Existing Drivers

Determine whether the host already has an NVIDIA driver:

lsmod | grep -E '^nvidia' || true
command -v nvidia-smi >/dev/null && nvidia-smi || true

This decision affects the Helm configuration:

  • No host driver: allow the GPU Operator to deploy the driver.
  • Host driver already installed: disable the Operator-managed driver.
  • Driver and NVIDIA Container Toolkit already installed: disable both components in the Operator.
  • Immutable or tightly controlled nodes: consider precompiled drivers or a baked node image.

Do not intentionally install two competing driver-management systems on the same node.

Check the Nouveau Module

The open-source nouveau driver can conflict with NVIDIA driver initialization, particularly in vGPU environments.

Check whether it is loaded:

lsmod | grep nouveau || true

If it is present, follow the operating system and NVIDIA guidance for the selected deployment model. Denylisting the module normally requires updating the initramfs and rebooting the node.

Check Secure Boot

Inspect the current state when mokutil is available:

mokutil --sb-state 2>/dev/null || true

The current GPU Operator troubleshooting guidance identifies EFI Secure Boot as unsupported for the normal Operator-managed driver path and instructs administrators to disable it. Treat this as a platform design gate rather than discovering it during installation.

Verify Container Runtime Health

For containerd:

containerd --version
sudo systemctl status containerd --no-pager

For CRI-O:

crio --version
sudo systemctl status crio --no-pager

Also confirm that kubelet is healthy:

sudo systemctl status kubelet --no-pager

A GPU Operator installation cannot compensate for an unhealthy runtime, missing CRI socket, or kubelet configuration problem.

Confirm Registry and Package Access

A default installation pulls several container images. The driver container might also need operating system packages while preparing the node.

Validate that the GPU nodes have:

  • DNS resolution.
  • Outbound access to the required NVIDIA registry endpoints.
  • Access to operating system package repositories.
  • Access to any internal registry mirror.
  • Trusted certificate chains for private repositories.
  • Working image pull secrets when private images are used.

Air-gapped clusters require a separate mirroring and repository design. Do not begin with the default connected installation and assume it can later be converted without additional work.

Choose the Driver Installation Model

The driver model is the most important configuration decision in the installation.

Model Best Fit Operator Configuration Operational Ownership
Operator-managed driver Standard Kubernetes GPU node pools driver.enabled=true GPU Operator manages the containerized driver
Preinstalled host driver Golden images, managed cloud nodes, DGX-style images driver.enabled=false Node image or host-management process
Preinstalled driver and toolkit Fully prepared GPU node image Disable driver and toolkit Node image and operating system team
Precompiled driver container Controlled kernels and supported precompiled combinations Enable precompiled mode Operator plus kernel compatibility process
Custom driver image Specialized or isolated environments Custom repository and version Platform team, with limited vendor support

Operator-Managed Containerized Drivers

This is the default installation model.

The Operator deploys a driver DaemonSet to each detected GPU worker node. It installs and loads the appropriate NVIDIA kernel modules and makes the driver filesystem available to the other GPU components.

This model provides the most integrated lifecycle, but it also means the Operator needs privileged access and close interaction with the host kernel.

Preinstalled Host Drivers

Use host-installed drivers when the node image or infrastructure platform already manages the driver.

The installation override is:

--set driver.enabled=false

The GPU Operator can still manage the toolkit, device plugin, monitoring, GPU discovery, and other operands.

The Operator does not manage the lifecycle of a driver that was installed directly on the host. Driver upgrades must remain part of the node image, operating system, or infrastructure lifecycle.

Preinstalled Driver and Container Toolkit

When both components are already installed, use:

--set driver.enabled=false 
--set toolkit.enabled=false

NVIDIA documentation also requires the existing runtime configuration to be prepared correctly before using this model. Validate that the host toolkit and container runtime configuration match the Kubernetes workload pattern.

Precompiled Drivers

Precompiled driver containers avoid building kernel modules during node initialization. They can reduce installation dependencies and improve predictability, but only for supported operating system, architecture, kernel, and driver-branch combinations.

The relevant Helm settings are:

--set driver.usePrecompiled=true 
--set driver.version="<supported-driver-branch>"

Do not use an arbitrary branch number. Select one from the component and precompiled-driver matrices for the exact Operator release and kernel.

Open or Proprietary Kernel Modules

Current charts use driver.kernelModuleType=auto by default. With supported modern driver branches, the Operator selects the recommended open or proprietary kernel module based on the GPU and driver.

The available values are:

  • auto
  • open
  • proprietary

The older driver.useOpenKernelModules option is deprecated. Do not copy that setting from an outdated installation guide.

Understand the Current Container Runtime Model

Beginning with GPU Operator 25.10, Container Device Interface support is enabled by default.

CDI provides a standardized method for describing and injecting devices into containers. For normal Kubernetes workloads that request GPUs through the NVIDIA device plugin, CDI is largely transparent.

A standard GPU pod normally needs only:

resources:
  limits:
    nvidia.com/gpu: 1

It does not need runtimeClassName: nvidia simply to consume a GPU.

This is an important difference from many older GPU Operator tutorials.

CDI and Standard GPU Workloads

The device plugin allocates the GPU, and the container runtime uses CDI to inject the device, libraries, and required mounts.

With the default configuration:

cdi:
  enabled: true

Application teams should request GPUs through Kubernetes resources rather than manually setting NVIDIA_VISIBLE_DEVICES.

Optional NRI Integration

The Node Resource Interface plugin can handle device injection without modifying the container runtime configuration.

Enable it with:

cdi:
  enabled: true
  nriPluginEnabled: true

NRI requires a compatible containerd or CRI-O release. Validate the current minimum versions before enabling it.

For a first installation, leave NRI disabled unless you have a specific runtime-management requirement and have tested it with the Kubernetes distribution.

Custom containerd Paths

The default toolkit configuration expects the conventional containerd configuration and socket paths:

/etc/containerd/config.toml
/run/containerd/containerd.sock

K3s, RKE2, custom packages, and some managed distributions use different paths. In those environments, set the toolkit environment values or use the platform-specific installation instructions.

Do not overwrite a managed runtime configuration without understanding which service regenerates it.

Install the GPU Operator with Helm

The commands below use GPU Operator v26.3.3, which is the current patch release at the time of writing.

Pinning the version makes the installation reproducible. Recheck the release notes and support matrix before using a newer chart.

Set the Installation Variables

Set the chart version and the NVIDIA Helm repository endpoint. The repository endpoint is listed in External References.

export GPU_OPERATOR_VERSION="v26.3.3"
export NVIDIA_HELM_REPOSITORY="<NVIDIA Helm repository endpoint>"

Create the Namespace

The GPU Operator requires privileged workloads because it installs drivers, mounts host paths, and modifies runtime configuration.

Create the namespace:

kubectl create namespace gpu-operator 
  --dry-run=client 
  -o yaml | kubectl apply -f -

If Pod Security Admission is enforced, label the namespace:

kubectl label --overwrite namespace gpu-operator 
  pod-security.kubernetes.io/enforce=privileged

Review this exception with the Kubernetes security team. The privileged namespace should be restricted to the Operator and its managed components.

Add the Helm Repository

helm repo add nvidia "${NVIDIA_HELM_REPOSITORY}"
helm repo update nvidia

Confirm the chart is available:

helm search repo nvidia/gpu-operator --versions | head

Create a Baseline Values File

Create gpu-operator-values.yaml:

driver:
  enabled: true
  kernelModuleType: auto

toolkit:
  enabled: true

cdi:
  enabled: true
  nriPluginEnabled: false

nfd:
  enabled: true

dcgmExporter:
  enabled: true

This values file makes the key decisions visible without unnecessarily overriding every chart default.

Change the following only when required:

  • Set driver.enabled to false for host-installed drivers.
  • Set toolkit.enabled to false for a host-installed toolkit.
  • Set nfd.enabled to false if the cluster already runs a compatible Node Feature Discovery deployment.
  • Enable NRI only after runtime compatibility has been verified.
  • Pin driver.version when your support or validation process requires a specific driver.
  • Configure image pull secrets when using private repositories.

Inspect all available settings before installation:

helm show values nvidia/gpu-operator 
  --version "${GPU_OPERATOR_VERSION}" 
  > gpu-operator-default-values.yaml

Install the Chart

helm install gpu-operator nvidia/gpu-operator 
  --namespace gpu-operator 
  --version "${GPU_OPERATOR_VERSION}" 
  --values gpu-operator-values.yaml 
  --wait 
  --timeout 20m

The --wait flag causes Helm to wait for the release resources, but it does not replace the component-level validation that follows.

Check the Helm release:

helm list -n gpu-operator
helm status gpu-operator -n gpu-operator

Verify the Operator Components

A default installation normally deploys components for:

  • NVIDIA GPU driver.
  • NVIDIA Container Toolkit.
  • NVIDIA Kubernetes Device Plugin.
  • GPU Feature Discovery.
  • Node Feature Discovery.
  • DCGM Exporter.
  • MIG Manager on compatible systems.
  • Operator validation.

List the pods:

kubectl get pods -n gpu-operator -o wide

Healthy long-running pods should show Running. One-time CUDA validation pods can show Completed.

Check the ClusterPolicy:

kubectl get clusterpolicy

The expected state is ready.

Inspect the full resource if it is not ready:

kubectl describe clusterpolicy cluster-policy

Check the managed DaemonSets:

kubectl get daemonsets -n gpu-operator

For each GPU node, the desired and ready counts should align for the driver, toolkit, device plugin, feature discovery, and telemetry DaemonSets that apply to that node.

Confirm Device Plugin Registration and GPU Discovery

The device plugin registers GPUs with kubelet as an extended resource named nvidia.com/gpu.

Check node capacity and allocatable resources:

kubectl get nodes 
  -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia.com/gpu

A healthy GPU node should show the expected number of allocatable GPUs.

Inspect a specific node:

kubectl describe node <gpu-node-name>

Look for a capacity section similar to:

Capacity:
  nvidia.com/gpu: 1

Allocatable:
  nvidia.com/gpu: 1

GPU Feature Discovery also adds useful labels that can support scheduling and inventory.

Review them with:

kubectl get node <gpu-node-name> --show-labels

For a cleaner view:

kubectl get node <gpu-node-name> -o json 
  | jq '.metadata.labels | with_entries(select(.key | startswith("nvidia.com/")))'

Common labels describe properties such as:

  • GPU presence.
  • GPU product.
  • Driver version.
  • CUDA capability.
  • GPU count.
  • MIG capability or configuration.
  • GPU memory.

These labels can later support node affinity, admission policies, chargeback, or workload-placement rules.

Run a CUDA Validation Workload

Resource advertisement proves that the scheduler can see the GPU. It does not prove that a container can execute a CUDA kernel.

Create cuda-vectoradd.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: cuda-vectoradd
spec:
  restartPolicy: Never
  containers:
    - name: cuda-vectoradd
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda12.5.0-ubuntu22.04
      resources:
        limits:
          nvidia.com/gpu: 1

Notice that the pod requests one GPU but does not specify runtimeClassName. That is intentional for the current CDI-based default configuration.

Apply the workload:

kubectl apply -f cuda-vectoradd.yaml

Watch its status:

kubectl get pod cuda-vectoradd -w

When the pod completes, inspect the logs:

kubectl logs cuda-vectoradd

A successful test includes:

Test PASSED
Done

Delete the validation pod:

kubectl delete -f cuda-vectoradd.yaml

If your environment cannot pull the public validation image, mirror it into the approved internal registry and update the YAML.

Validate GPU Access Inside a Running Container

A second validation pattern is useful for confirming nvidia-smi output from inside a container.

Create nvidia-smi-test.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: nvidia-smi-test
spec:
  restartPolicy: Never
  containers:
    - name: nvidia-smi
      image: nvcr.io/nvidia/cuda:12.8.1-base-ubuntu24.04
      command:
        - nvidia-smi
      resources:
        limits:
          nvidia.com/gpu: 1

Apply it and inspect the logs:

kubectl apply -f nvidia-smi-test.yaml
kubectl wait --for=condition=Ready pod/nvidia-smi-test --timeout=120s || true
kubectl logs nvidia-smi-test

The output should identify the allocated GPU and driver.

Remove the pod:

kubectl delete -f nvidia-smi-test.yaml

Always verify that the CUDA container image is compatible with the selected driver branch before standardizing it for production.

Establish an Operational Baseline

Before onboarding application teams, capture a known-good state.

helm get values gpu-operator 
  -n gpu-operator 
  -o yaml > gpu-operator-installed-values.yaml

kubectl get clusterpolicy cluster-policy 
  -o yaml > cluster-policy-baseline.yaml

kubectl get nodes 
  -o custom-columns=NAME:.metadata.name,GPUS:.status.allocatable.nvidia.com/gpu 
  > gpu-capacity-baseline.txt

kubectl get pods -n gpu-operator -o wide 
  > gpu-operator-pods-baseline.txt

Also record:

  • GPU Operator chart version.
  • Driver version.
  • Container Toolkit version.
  • Device plugin version.
  • GPU model and firmware.
  • Operating system and kernel version.
  • Kubernetes version.
  • Container runtime version.
  • Validation workload image.
  • Validation result and date.

This baseline becomes important after kernel updates, driver upgrades, node replacements, or cluster upgrades.

Plan Operator Upgrades Carefully

The GPU Operator is a cluster infrastructure component. Treat upgrades as platform changes rather than routine application releases.

The current lifecycle policy supports upgrades within the current major release or to the next major release. If a cluster is several release families behind, do not assume that a direct jump is supported.

Prepare the Upgrade

Before upgrading:

  1. Read the target release notes and component matrix.
  2. Verify the Kubernetes, Linux, kernel, runtime, and GPU combination.
  3. Export the existing Helm values and ClusterPolicy.
  4. Test the change on a representative non-production GPU node pool.
  5. Identify workloads that must be stopped or drained during driver changes.
  6. Confirm rollback boundaries.
  7. Verify image and package availability.
  8. Capture current GPU health and DCGM metrics.

Export the existing configuration:

helm get values gpu-operator 
  -n gpu-operator 
  -o yaml > gpu-operator-values-before-upgrade.yaml

kubectl get clusterpolicy cluster-policy 
  -o yaml > cluster-policy-before-upgrade.yaml

helm history gpu-operator -n gpu-operator

Upgrade the Helm Release

Set the target release:

export TARGET_GPU_OPERATOR_VERSION="<validated-target-version>"

Refresh the repository:

helm repo update nvidia

Fetch and review the target values:

helm show values nvidia/gpu-operator 
  --version "${TARGET_GPU_OPERATOR_VERSION}" 
  > gpu-operator-values-target.yaml

Compare the current and target values instead of blindly reusing an old file:

diff -u gpu-operator-values-before-upgrade.yaml 
  gpu-operator-values-target.yaml || true

Perform the upgrade:

helm upgrade gpu-operator nvidia/gpu-operator 
  --namespace gpu-operator 
  --version "${TARGET_GPU_OPERATOR_VERSION}" 
  --values gpu-operator-values-target.yaml 
  --disable-openapi-validation 
  --wait 
  --timeout 30m

Current GPU Operator releases use a Helm pre-upgrade hook for CRD upgrades. The hook has been enabled by default since the 24.9 release line.

After the upgrade, repeat the full component, allocatable-resource, CUDA, and telemetry validation.

Understand Driver Upgrades

The upgrade controller is enabled by default and is NVIDIA’s recommended method for upgrading Operator-managed containerized drivers.

A driver upgrade must:

  1. Stop clients that are using the driver.
  2. Unload the current kernel modules.
  3. replace the driver pod.
  4. Install and load the new modules.
  5. Restore GPU clients.

Monitor driver upgrade state:

kubectl get nodes -l nvidia.com/gpu.present 
  -o jsonpath='{range .items[*]}{.metadata.name}{"t"}{.metadata.labels.nvidia.com/gpu-driver-upgrade-state}{"n"}{end}'

A completed node should eventually report upgrade-done.

Drivers that were installed directly on the host are outside the Operator’s lifecycle. Those nodes must use the operating system or image-management process for upgrades.

Know the Rollback Boundary

A Helm rollback can restore chart-managed Kubernetes resources:

helm history gpu-operator -n gpu-operator

helm rollback gpu-operator <revision> 
  -n gpu-operator 
  --wait 
  --timeout 30m

However, a Helm rollback might not fully reverse:

  • Updated CRDs.
  • A host kernel change.
  • A driver module already installed or loaded.
  • Runtime configuration changes.
  • A MIG geometry change.
  • A node image replacement.

Document the rollback plan at the chart, driver, kernel, runtime, and node-image layers.

Troubleshoot Pods That Cannot See GPUs

Do not start by changing the application image or reinstalling the entire Operator.

Use a bottom-up workflow.

Symptom and Cause Matrix

Symptom Likely Layer First Checks
No NVIDIA PCI device Hardware or virtualization lspci, passthrough, instance type
Driver pod stuck in init Kernel or driver Driver logs, kernel headers, repositories, Secure Boot
nvidia-smi fails Driver or hardware Driver pod, dmesg, NVRM and Xid messages
No runtime for nvidia Toolkit or RuntimeClass Toolkit logs and runtime configuration
Node shows no nvidia.com/gpu Device plugin or kubelet Device plugin pod and node capacity
Pod remains Pending Scheduler or capacity Pod events, GPU requests, taints, affinity
Wrong number of GPUs Driver, device plugin, MIG, hardware nvidia-smi -L, allocatable resources, MIG config
CUDA test fails Image and driver compatibility Driver version, CUDA image requirements
Validator repeatedly fails Lower stack dependency Driver, toolkit, Fabric Manager, or hardware health

Check Operator State First

kubectl get clusterpolicy
kubectl get pods -n gpu-operator -o wide
kubectl get events -n gpu-operator --sort-by=.lastTimestamp

Find components that are not Running or Completed.

Describe the failed pod:

kubectl describe pod <pod-name> -n gpu-operator

Review all containers in the pod:

kubectl logs <pod-name> 
  -n gpu-operator 
  --all-containers 
  --tail=200

Check the Driver Layer

List driver pods and their assigned nodes:

kubectl get pods 
  -n gpu-operator 
  -l app=nvidia-driver-daemonset 
  -o wide

Select the driver pod on the affected node:

kubectl logs <driver-pod> 
  -n gpu-operator 
  -c nvidia-driver-ctr 
  --tail=200

Run nvidia-smi inside the driver container:

kubectl exec 
  -n gpu-operator 
  <driver-pod> 
  -c nvidia-driver-ctr 
  -- nvidia-smi

Check kernel messages:

kubectl exec 
  -n gpu-operator 
  <driver-pod> 
  -c nvidia-driver-ctr 
  -- dmesg | grep -Ei 'NVRM|Xid|nvidia'

Common driver-layer problems include:

  • Unsupported kernel.
  • Missing matching kernel packages.
  • Secure Boot.
  • Loaded nouveau module.
  • Inaccessible operating system repositories.
  • Incorrect vGPU driver or licensing path.
  • Hardware errors or GPUs that have fallen off the PCI bus.
  • Fabric Manager problems on NVSwitch-based systems.

Check the Toolkit and Runtime Layer

List toolkit pods:

kubectl get pods 
  -n gpu-operator 
  -l app=nvidia-container-toolkit-daemonset 
  -o wide

Review the pod on the affected node:

kubectl logs <toolkit-pod> 
  -n gpu-operator 
  --all-containers 
  --tail=200

Inspect the active containerd configuration on the node:

sudo containerd config dump | grep -i -A8 -B4 nvidia || true

For CRI-O:

sudo crio status config | grep -i -A8 -B4 nvidia || true

An error stating that no runtime named nvidia exists usually means a pod explicitly requested the NVIDIA RuntimeClass but the toolkit did not create the corresponding handler.

For ordinary device-plugin workloads on current releases, remove an unnecessary runtimeClassName: nvidia and use a normal nvidia.com/gpu resource limit.

Check the Device Plugin

List the device plugin pods:

kubectl get pods 
  -n gpu-operator 
  -l app=nvidia-device-plugin-daemonset 
  -o wide

Review the affected pod:

kubectl logs <device-plugin-pod> 
  -n gpu-operator 
  --all-containers 
  --tail=200

Check node allocation:

kubectl get node <gpu-node-name> 
  -o jsonpath='{.status.capacity.nvidia.com/gpu}{" capacity, "}{.status.allocatable.nvidia.com/gpu}{" allocatablen"}'

If the driver works but the resource is missing, investigate:

  • Device plugin pod failure.
  • Kubelet device plugin registration.
  • Node labels.
  • MIG configuration.
  • Operator deployment labels.
  • GPU health.
  • Runtime or mount failures affecting the device plugin.

Check the Workload Specification

Describe the workload:

kubectl describe pod <workload-pod>

Look at the Events section.

A GPU must normally be requested as a limit:

resources:
  limits:
    nvidia.com/gpu: 1

Also check:

  • The requested number of GPUs does not exceed allocatable capacity.
  • Node affinity allows a GPU node.
  • Taints have matching tolerations.
  • The pod is not requesting a MIG resource that the node does not expose.
  • Namespace quotas do not block the request.
  • Admission policies have not modified or rejected the pod.
  • The selected CUDA image supports the installed driver.

Collect a Must-Gather Archive

When component logs do not reveal the cause, use NVIDIA’s GPU Operator must-gather utility according to the current troubleshooting documentation.

The archive collects Kubernetes resources and logs for the Operator and managed operands. Review the contents for sensitive infrastructure information before transferring it to a vendor or external support system.

Practical Production Recommendations

A lab installation can use defaults. A production platform should establish stronger controls.

  • Pin the GPU Operator chart version.
  • Record and test the selected driver branch.
  • Use dedicated GPU node pools.
  • Control kernel upgrades.
  • Mirror images for restricted or production environments.
  • Monitor DCGM metrics and GPU Xid events.
  • Maintain a representative non-production GPU node.
  • Run CUDA validation after every platform change.
  • Define who owns the Operator, node image, kernel, driver, runtime, and workload.
  • Treat MIG and time slicing as separate capacity-design decisions.
  • Preserve Helm values, ClusterPolicy, logs, and validation evidence.
  • Align Kubernetes upgrades with the GPU Operator support matrix.
  • Do not let application teams bypass Kubernetes allocation by manually injecting devices.

The GPU Operator simplifies installation, but it does not remove the need for platform ownership and lifecycle discipline.

Conclusion

The NVIDIA GPU Operator provides the cleanest general-purpose path for turning Kubernetes GPU nodes into a repeatable platform service.

The installation itself is straightforward once the underlying decisions are made. The hardware must be visible. The operating system, kernel, Kubernetes release, and runtime must be supported. The platform team must choose whether the driver belongs to the Operator or the host image. The toolkit and CDI path must be healthy. Finally, the device plugin must advertise nvidia.com/gpu before Kubernetes can schedule a workload.

The most important operational lesson is to troubleshoot the stack from the bottom up. A missing GPU resource is usually not an application problem. It is evidence that one of the hardware, driver, runtime, toolkit, device plugin, or kubelet layers has not completed its role.

Once this foundation is stable, the cluster is ready for more advanced GPU platform capabilities such as NVIDIA NIM microservices, MIG partitioning, GPU time slicing, DCGM-based monitoring, Kubernetes autoscaling, and governed AI workload deployment.

External References

  • NVIDIA: Installing the NVIDIA GPU Operator
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/getting-started.html
  • NVIDIA: GPU Operator Platform Support
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/platform-support.html
  • NVIDIA: GPU Operator Release Notes
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/release-notes.html
  • NVIDIA: Container Device Interface and Node Resource Interface Plugin Support
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/cdi.html
  • NVIDIA: Upgrading the NVIDIA GPU Operator
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/upgrade.html
  • NVIDIA: GPU Driver Upgrades
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/gpu-driver-upgrades.html
  • NVIDIA: Troubleshooting the NVIDIA GPU Operator
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/troubleshooting.html
  • NVIDIA: Precompiled Driver Containers
    Canonical URL: https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/latest/precompiled-drivers.html
  • NVIDIA: GPU Operator Helm Repository
    Canonical URL: https://helm.ngc.nvidia.com/nvidia
  • Kubernetes: Schedule GPUs
    Canonical URL: https://kubernetes.io/docs/tasks/manage-gpus/scheduling-gpus/
  • Kubernetes: Device Plugins
    Canonical URL: https://kubernetes.io/docs/concepts/extend-kubernetes/compute-storage-net/device-plugins/

The post How to Install and Configure the NVIDIA GPU Operator on Kubernetes appeared first on Digital Thought Disruption.