Skip to content
Field Guide Kubernetes NVIDIA Architecture

NVKE, NVIDIA Kubernetes Engine:
Managed Kubernetes on Your Own GPUs

NVIDIA has open-sourced the layer that turns a rack of GPU servers into a managed Kubernetes service. Here is how NVKE works, drawn as diagrams: the architecture, how a cluster request becomes work, how nodes prove who they are, and the open source projects it runs on.

Updated: November 9, 2026  ·  Source: the NVKE repository and docs  ·  Read: 10 minutes

Share

At a glance

  • What it is: NVIDIA Kubernetes Engine (NVKE) is an open source management plane for running a managed Kubernetes service on your own NVIDIA GPU machines, on bare metal. Apache 2.0, written in Go, on GitHub at dsx-ai-factory/nvke.
  • Who it is for: platform teams and GPU clouds that want to hand customers their own Kubernetes clusters, each with its own control plane, etcd and node pools, the way a hyperscaler's managed Kubernetes does.
  • What it is not: a Kubernetes distribution or a Cluster API flavour. The project calls it a self-hosting factory for clusters: a throwaway seed cluster builds a management cluster, and that cluster then builds many isolated workload clusters on demand.
  • How it is built: one controller process runs a REST API backed by Postgres, a Temporal worker and a set of Kubernetes reconcilers. A gRPC agent on every node does the node-side work, and nodes prove who they are with TPM attestation through SPIFFE and SPIRE.
  • Status: a preview. APIs can change, API authentication is off after install until you configure it, and the project does not recommend it for production yet.

The architecture, at a high level

An NVKE installation has two kinds of Kubernetes cluster. The management cluster runs the NVKE management plane and the infrastructure controllers. Workload clusters run customer applications: the management plane creates them, and customers only ever see their own cluster's Kubernetes API.

Machines come from NVIDIA Infra Controller (NICo), which handles bare-metal lifecycle: BMC access, firmware, networking and DPUs. NVKE asks for machines through CAPNICo, a Cluster API infrastructure provider. Node images and component releases come from a companion project, NVIDIA Kubernetes Recipes (NVKR), which builds and signs them from pinned inputs.

NVKE high-level architecture The operator builds artifacts with NVKR, which publishes them to a registry. The operator manages clusters through the NVKE management plane on the management cluster, which also runs CAPNICo and NVKE observability. CAPNICo requests machines from NICo, which provisions NVIDIA GPU machines from the node OS image in the registry. The management plane drives each workload cluster, and dials nvke-agent on each node over gRPC. Customers use kubectl against their workload cluster's control plane. Operator nvkectl or NVKE API NVKR builds and signs Registry charts, images, node OS builds publishes Management cluster NVKE management plane API, workflows, reconcilers CAPNICo Cluster API provider NVKE observability telemetry pipeline (NKTP) manages clusters charts, images NICo bare-metal lifecycle requests node OS Workload cluster (one per customer, many per site) Kubernetes control plane API server, etcd NVIDIA GPU nodes nvke-agent on every node Accelerated runtime via AICR (optional) Customer, kubectl drives clusters gRPC to nvke-agent provisions
High-level architecture, redrawn from the product family diagram in the NVKE README. Green is NVKE, purple is NVKR, dark is hardware, dashed is optional. Arrows point from the side that starts the interaction.

Three NVKE pieces run in production. The controller ships as a Helm chart onto the management cluster. The node agent ships as a Debian package and runs under systemd on each node. The Cluster API infrastructure provider runs beside the controller and reconciles hosts NVKE has already provisioned, so Cluster API can bootstrap them. Everything else in the thirteen-module Go workspace is the CLI, generated API types, the auth library, test tooling and release qualification.

How a cluster request becomes work

The API never calls the workflow engine directly. It writes a row, and something else picks it up. This is the transactional outbox pattern, and it is the part of the codebase the project itself says people most often get wrong on first read.

  1. 1
    Client calls the API. POST /api/clusters, from nvkectl or any HTTP client.
  2. 2
    One transaction, two rows. The handler writes the cluster record and a workflow_outbox row together in Postgres. Either both land or neither does, so a committed cluster always has its work queued.
  3. 3
    The outbox processor picks it up. It polls the table, starts the matching Temporal workflow and marks the row processed.
  4. 4
    Temporal drives Cluster API. The workflow creates Cluster API resources. CAPNICo asks NICo for machines, which provisions or adopts the nodes.
  5. 5
    The controller dials each node. Over gRPC, it tells nvke-agent to run kubeadm init or kubeadm join, install the CNI and start the kubelet.
  6. 6
    Ready. The cluster reports Ready and its kubeconfig becomes available to the customer.
The cluster create path, from the NVKE architecture overview. Later workflow steps queue more work the same way, through the outbox, rather than starting workflows directly.

Beside the workflow engine, the same controller process runs controller-runtime reconcilers for node pools, control-plane scheduling, certificate renewal, tenant access and secrets, and TPM and SPIFFE identity for machines. One binary is a REST service, a Temporal worker and a Kubernetes operator at once. Read only the workflows, or only the reconcilers, and you miss half of what reacts to a change.

The node agent and node identity

nvke-agent is a gRPC server. It listens, and the controller dials it. It never calls home, and the controller has no node-registration endpoint for it to call, which shrinks what an attacker on the network can reach. The agent runs kubeadm, installs the CNI, bootstraps the kubelet, and handles drain, reboot and teardown.

Before a node gets credentials to join a cluster, it attests through SPIFFE and SPIRE using its TPM (a join token is the fallback). The node's transport mode, from plaintext through SPIFFE to external mTLS, is pinned on the Machine object, and operator-supplied metadata cannot override it. Encryption of etcd data is available per cluster.

Day 0, Day 1, Day 2

NVKE splits operations into building the site, configuring it and running it.

NVKE Day 0, Day 1 and Day 2 Day 0: a temporary kind seed cluster on the init node runs kubeadm through nvke-agent to form the management cluster, then is torn down. Day 1: the NVKE stack is installed on the management cluster: controller, Temporal, PostgreSQL, SPIRE, the telemetry pipeline and CAPNICo. Day 2: the management plane creates, scales, upgrades and deletes many workload clusters. DAY 0 · BOOTSTRAP DAY 1 · CONFIGURE DAY 2 · OPERATE Seed kind cluster temporary, on the init node kubeadm via agent Management cluster plain kubeadm, no NVKE yet nvkectl bootstrap site NVKE site stack controller · Temporal PostgreSQL · SPIRE telemetry (NKTP) CAPNICo, connected to NICo site ingested into NVKE deploy/nvke-deploy.sh Workload cluster A Workload cluster B Workload cluster C nvkectl create cluster
The site lifecycle, from the NVKE lifecycle guide. Day 0 leaves you an etcd encryption key file: move it off the bootstrap host straight away.

Once a site is running, day-to-day work is a short list of nvkectl commands:

TaskCommand
Create a workload clusternvkectl create cluster -f spec.yaml
Get a customer's kubeconfignvkectl get kubeconfig <name>
Add a node poolnvkectl create nodepool -f spec.yaml
Scale a node poolnvkectl patch nodepool <name> --patch 'desiredNodeCount: N'
Upgrade Kubernetesnvkectl patch cluster <name> --patch 'controlPlaneKubernetesVersion: "1.34.0"'
Check the infrastructure providernvkectl get providers

Where NVKE sits in the stack

NVKE is one layer of NVIDIA's open source DSX stack for AI factories. It provisions clusters with OVN-Kubernetes networking and then hands off: GPU drivers, workload recipes and readiness testing belong to the layers above it.

Nodewright GPU drivers and host packages AICR validated AI runtime recipes CRE readiness checks with real GPU jobs
NVKE creates and lifecycle-manages Kubernetes clusters
NICo bare-metal lifecycle: discovery, firmware, networking, DPUs
Hardware GPUs, DPUs, switches, BMCs

Beside the stack: NVSentinel handles workload-aware cordon, drain and node replacement, through NVKE's node lifecycle APIs.

The DSX stack as NVKE's documentation describes it. Tenant AI workloads run on top.

The open source underneath

When the NVKE team announced the project, they did something worth copying: they reviewed their dependencies across Go modules, Helm charts, node images and CI, and thanked the projects by name. In their words, every time NVKE brings up a GPU cluster, much of the code that runs was written by people who will never see their logs or their pager. Here is that list, grouped the way they grouped it. It doubles as a map of what a modern bare-metal Kubernetes platform is made of.

On every node

Kubernetes (kubeadm, kubelet, kubectl) · containerd · CNI plugins · cri-tools · SPIFFE and SPIRE · Helm

In tenant clusters

etcd · CoreDNS · node-local-dns · OVN-Kubernetes · kube-vip

In the management plane

Cluster API · controller-runtime · gRPC · Cedar · cert-manager

At sites that run them

CloudNativePG · Argo CD

Artifact distribution

ORAS (CLI and Go library)

In telemetry

OpenTelemetry · Prometheus · kube-state-metrics · metrics-server (optional)

In the dev loop

kind · kubebuilder · kustomize · ko · nerdctl · Lima · Distribution · KWOK · kube-burner

Not CNCF, just as essential

Temporal · PostgreSQL · Calico · Grafana · runc · Packer

Their closing note is the right one: if you build on these projects too, thank a maintainer today, and if you can, review a pull request or triage an issue.

Before you try it

  • No GPUs needed to evaluate. The local quickstart runs a development management plane and a workload cluster on simulated machines. It needs a Linux host or VM (amd64 or arm64) with 6 CPUs, 12 GiB of free memory, 100 GiB of disk, Docker and virtual TPM support. On macOS there is a guide that runs it inside a Colima VM.
  • For a real install, you bring the machines, networking, node images and registries, and access to NICo, which is the only supported infrastructure provider during the preview.
  • Lock it down first. After installation the API has authentication off and speaks plain HTTP. Configure API authentication before you let anyone call it.
  • Know the gaps. The management cluster does not yet upgrade, scale or repair itself, there is no management-plane backup and restore, and the full-disk-encryption node images are for evaluation only.
  • Building from source takes Go 1.27.3 and GNU Make 4: make build writes nine binaries to bin/, including nvkectl, controller and agent.

What is coming

The public roadmap has no dates, but the direction is clear. Coming soon: per-customer authorization on every API call, managing the management cluster's own nodes through NVKE, cluster certificates from your own PKI, and Kubernetes 1.37. In progress: GB300 and Vera Rubin worker nodes, separate control-plane and worker networks, automatic scale-down that respects PodDisruptionBudgets, and add-on management through the API. Being explored: an MCP server and agent skills so AI assistants can operate clusters under the same per-customer authorization, in-place CVE patching, backup and restore, and validation at 5,000 nodes per cluster.

Our take

The interesting thing about NVKE is not that NVIDIA wrote a Kubernetes manager. It is that the "managed Kubernetes on your own GPUs" layer, the part that separates a GPU cloud from a rack of servers, is now open source and Apache-licensed. For neoclouds, sovereign AI builders and enterprises with their own GPU fleets, that removes a large piece of undifferentiated platform work, and moves the competition up the stack to scheduling, utilization, price and service.

It also fits a theme we keep returning to: idle or stranded GPUs are a software problem as much as a power problem. See Stranded Compute for the infrastructure side and Physical AI Needs the Edge for where else bare-metal GPU fleets are heading. And the design choices are worth stealing even if you never run NVKE: a transactional outbox in front of durable workflows, an agent that listens instead of calling home, and hardware-rooted node identity before any join credential.

Sources: the NVKE repository, including its README, architecture overview, lifecycle guide, DSX stack page and roadmap, read on November 9, 2026; and the NVKE team's thank-you post to the open source projects it builds on. The diagrams are our own drawings of the project's documentation. NVKE is a preview and details will change. This guide is independent and not affiliated with NVIDIA. Kubernetes is a registered trademark of The Linux Foundation; NVIDIA is a trademark of NVIDIA Corporation. Spot something out of date? Tell us.