Building a Kubernetes Operator in .NET

The previous lesson added a new resource type to the API server. A FeatureFlag was validated and stored, and a .NET API read it, but nothing in the cluster changed because of it. An operator closes that loop. It watches a custom resource, compares what the resource asks for with what exists, and creates, updates or deletes ordinary Kubernetes objects until the two match. That is exactly how the Deployment controller turns a Deployment into ReplicaSets and Pods, and how Argo CD, KEDA, cert-manager and the Vault Secrets Operator from earlier lessons work.

In this lesson you build one in C#. A DotnetApp resource describes a .NET service in a few lines: image, replica count, port, health path and environment variables. The operator turns it into a Deployment and a Service, reports progress in the resource's status, keeps a shared catalog of all apps up to date, and cleans that catalog up through a finalizer when an app is deleted. It runs as two replicas with leader election, and it is built on nothing but the official KubernetesClient package and .NET 10's generic host.

You will break it on purpose: delete its Deployment, scale it by hand, kill the leader, and delete an app while the operator is down. Each time it converges back to what the DotnetApp says.

We will cover the following topics:

1. The operator pattern: custom resource plus controller, and why controllers are level-triggered
2. The reconcile loop: observe, compare, act, report, and why every step must be idempotent
3. Watches, a de-duplicating work queue, periodic resync and exponential backoff
4. Owner references and the garbage collector, and server-side apply for the children
5. Status, conditions and observedGeneration, and the scale subresource
6. Finalizers for cleanup that owner references cannot do
7. RBAC for an operator, and leader election with a Lease

Technical Requirements

1. A Kubernetes cluster and kubectl. The examples run on Docker Desktop's built-in Kubernetes with the kind provisioner (1 control-plane node and 3 workers, Kubernetes v1.36). A cluster created with kind create cluster and the same layout behaves the same; there, load the two images you build with kind load docker-image.
2. Docker and the .NET 10 SDK to build the operator and the sample API.
3. The lesson Custom Resource Definitions — Extending the Kubernetes API. Schemas, printer columns and the status subresource are used here without being re-explained.

Core Concepts

The Operator Pattern

An operator is a custom resource plus a controller that knows how to run one kind of application. The custom resource is the interface for humans and tools: short, declarative and validated. The controller holds the operational knowledge: which objects to create, in which order, how to upgrade, how to clean up.

Level-Triggered Reconciliation

A controller does not react to "what just happened". It reacts to "what is the state now". Every event, whether a create, an update, a deleted child or a timer, only puts the object's key (namespace/name) into a queue. The reconcile function then reads the current object and the current children and makes them match. It never relies on the content of the event.

This is called level-triggered, as opposed to edge-triggered. It is what makes controllers robust: if the operator was down while three changes happened, it does not need to replay them. It reads the current state once and converges. The reconcile function must therefore be idempotent: running it twice in a row must have the same effect as running it once.

Watches, the Work Queue and Resync

The operator keeps three sources of work:

1. A watch on DotnetApp objects in all namespaces. Every event enqueues the object's key.
2. A watch on Deployments labeled app.kubernetes.io/managed-by=dotnetapp-operator. An event on a child is mapped to its owner through the owner reference, so a deleted, scaled or newly ready Deployment re-runs the owner's reconcile.
3. A resync every two minutes that lists all DotnetApp objects and enqueues them. Watches can miss events after network problems; the resync guarantees convergence anyway.

The work queue de-duplicates: while a key waits in the queue, more events for it do not add more entries. A single worker processes the queue, so the same object is never reconciled twice at the same time. When a reconcile throws, the key is re-queued with exponential backoff (1 s, 2 s, 4 s and so on, capped at 5 minutes), so a broken object does not hammer the API server.

Frameworks such as controller-runtime in Go, or KubeOps and the Java Operator SDK, give you these pieces ready-made, including shared informer caches. Building them by hand once in plain C# makes it clear what those frameworks do.

Owner References and Server-Side Apply

Every child the operator creates carries an owner reference to its DotnetApp:

When the owner is deleted, the garbage collector in the controller manager deletes the children. The operator never writes delete logic for them, which is why its ClusterRole has no delete verb on Deployments. Owner references only work within a namespace, or from a namespaced object to a cluster-scoped owner. They cannot point across namespaces.

The children are written with server-side apply (SSA): the operator sends the complete object it wants under a field manager name, dotnetapp-operator, with force: true. The API server merges it, records which manager owns which field, and changes nothing if the intent is already in place. That makes the "act" step idempotent for free. If a human changes a field the operator owns, for example spec.replicas with kubectl scale, the next apply takes it back.

Status, Conditions and observedGeneration

The status follows the Kubernetes API conventions:

Two rules keep status from causing trouble. lastTransitionTime changes only when the condition's status flips, not on every write. And the operator writes status only when it differs from what is stored. A status write is itself a change that produces a watch event; writing unconditionally would make the operator reconcile itself in an endless loop.

Finalizers

A finalizer is a string in metadata.finalizers. While it is present, a delete request does not remove the object. It only sets metadata.deletionTimestamp. The object stays visible, now in a terminating state, until every finalizer has been removed. The controller sees the timestamp in its reconcile, does its cleanup, and removes its finalizer. Then the API server deletes the object.

Use a finalizer for cleanup the garbage collector cannot do: objects in another namespace, DNS records, cloud resources, entries in external systems. In this lesson it is a shared app-catalog ConfigMap in the operator's own namespace. Do not use a finalizer for children in the same namespace; owner references already handle those. And remember the cost: if the controller is gone for good, objects with its finalizer never finish deleting until someone removes the finalizer by hand.

RBAC and Leader Election

An operator needs exactly the permissions its reconcile uses. Here that is read and patch on dotnetapps, patch on dotnetapps/status, update on dotnetapps/finalizers (required for blockOwnerDeletion when the OwnerReferencesPermissionEnforcement admission plugin is on), get, list, watch, create and patch on Deployments and Services, and, in its own namespace only, the catalog ConfigMap and the leader-election Lease.

Running two replicas avoids a gap when a node fails, but two active controllers would fight over the same objects. Leader election solves it with a Lease object in coordination.k8s.io. Each replica tries to write its own identity into the Lease. The holder renews it every few seconds; the others watch. If the holder stops renewing for LeaseDuration, another replica takes over. KubernetesClient ships this as LeaderElector with a LeaseLock.

Hands-On: Kubernetes Commands

See which subresources a CRD enables

Find the children of a custom resource

The owner reference connects them.

See who owns which fields

Server-side apply records a manager per field set.

Check whether a controller has caught up

Scale a custom resource

Works because the CRD defines a scale subresource.

Find the current leader

Unblock an object stuck on a finalizer

Only when the controller that owns the finalizer is gone for good. The object disappears immediately, but its cleanup never runs: in a test of exactly this command, the app's entry stayed in app-catalog.

Step-by-Step Example

The Scenario

A platform team wants every .NET service to be deployed the same way: a readiness probe on /healthz, CPU and memory requests, a memory limit, standard labels and a ClusterIP Service on port 80. Teams should only describe what differs, in a DotnetApp. The platform also keeps an app-catalog ConfigMap in its own namespace listing every app and its URL, for a developer portal to read.

The operator runs in dotnetapp-system. The apps live in operator-lab. Both namespaces, the CRD, one ClusterRole and one ClusterRoleBinding are removed in the last step.

Step 1 — The Sample .NET API

The operator needs something to deploy. greeter-api is a minimal ASP.NET Core app that answers with a greeting taken from configuration:

Build it twice, as 1.0 and 1.1, so there is something to upgrade to later:

Step 2 — The DotnetApp CRD

Save as 1-dotnetapp-crd.yaml. The scale subresource points at .spec.replicas, .status.replicas and .status.selector. The Status column reads the reason of the Ready condition with a JSONPath filter.

Step 3 — The Operator in C#

Create a worker project and add the client library:

DotnetApp.cs maps the resource to C#. The top of the file declares the DotnetApp class itself (apiVersion, kind, metadata, spec, status) and the list class, exactly like FeatureFlag in the previous lesson. Spec and status:

WorkQueue.cs is the de-duplicating queue with per-key backoff:

Reconciler.cs is the heart of the operator. ReconcileAsync follows the table from Core Concepts: observe, handle deletion, ensure the finalizer, apply the children, report.

BuildDeployment and BuildService (not shown) create plain V1Deployment and V1Service objects from the spec: labels app.kubernetes.io/name and app.kubernetes.io/managed-by, the container with its port, environment variables, readiness probe on healthPath, requests of 50m CPU and 64Mi memory and a 128Mi memory limit, and a Service from port 80 to the app port. Both carry this owner reference and are written with server-side apply:

The catalog lives in another namespace, so owner references cannot clean it up. AddToCatalogAsync (not shown) server-side applies a ConfigMap app-catalog in the operator's namespace containing a single key, <namespace>.<name>, with the image and URL as value. It uses a separate field manager per app, dotnetapp-operator-<namespace>.<name>, so each app owns only its own key and applies never remove other apps' entries. RemoveFromCatalogAsync deletes the key with a JSON merge patch that sets it to null. Finally the status. "Ready" means more than "enough Pods are ready": during a rolling update old and new Pods are both ready, so the operator also requires that the Deployment controller has seen the latest template and that every replica runs it.

Controller.cs feeds the queue. The watch on DotnetApp objects follows the list-then-watch pattern from the previous lesson, across all namespaces with ListClusterCustomObjectAsync and WatchListClusterCustomObjectAsync. The rest of the class:

Program.cs registers the services and wraps the controller in leader election. Only the replica that holds the Lease runs controller.RunAsync. If it loses the Lease, it stops reconciling and exits, and Kubernetes restarts the Pod as a follower.

The image uses the chiseled .NET runtime image, since the operator serves no HTTP:

Step 4 — RBAC and Deploy the Operator

2-operator-rbac.yaml contains the dotnetapp-system namespace, the dotnetapp-operator ServiceAccount, a ClusterRole with a ClusterRoleBinding, and a Role with a RoleBinding. The two roles:

The operator can never delete a Deployment, and it can touch ConfigMaps only in its own namespace. Deploy two replicas with 3-operator.yaml:

The log lines are shortened to pod message for readability. One replica holds the Lease; the other only follows. The last line is a real-world detail worth knowing: right after a CRD is created, the API server's watch cache for it is still initializing and answers 429 TooManyRequests with retryAfterSeconds: 1. The watch loop retries after two seconds, and that is the end of it. Every watch in an operator needs such a retry loop.

Step 5 — Create a DotnetApp

Save as 4-greeter.yaml. Ten lines, and the defaults for port and health path come from the CRD:

Twelve seconds later:

Look at what the operator wrote: the owner reference on the child, the finalizer on the app, the status, and the catalog entry in the other namespace.

Call the app through its Service from inside the cluster, and read the operator's log:

Four status writes for one app, each caused by a change on the Deployment: created with no Pods, Pods created but not ready (replicas went from 0 to 2, hence the second 0/2), one ready, two ready. Between those, many more events arrived (the finalizer patch, each status write), but they changed nothing and produced no writes. That is the idempotency rule at work.

Step 6 — Self-Healing and Drift

Delete the Deployment, as an over-eager colleague might:

One second later a new Deployment exists. The garbage collector removed the old ReplicaSet and its Pods together with the deleted Deployment, and the new Pods of this small API were ready within about two seconds. The operator log shows the chain:

Now drift: scale the Deployment directly, bypassing the DotnetApp.

Three seconds later the Deployment is back at 2. The scale changed spec.replicas, the Deployment watch fired, the reconcile re-applied the desired object with force: true, and the field went back to its owner, dotnetapp-operator. kubectl's manager entry is gone from managedFields. The place to change replicas is the DotnetApp.

Step 7 — Scale Through the Scale Subresource

The spec changed, so the generation went to 2. The operator has already acted on it (observedGeneration=2), but only two replicas are ready so far. A few seconds later:

kubectl scale worked on a custom resource because the CRD maps the Scale API onto .spec.replicas. The same mapping lets a HorizontalPodAutoscaler target a DotnetApp directly, using .status.selector to find its Pods.

Step 8 — Roll Out a New Version

Change the image and the configuration in one patch, and watch the status column:

The first 1.1 line is the spec change itself, with the old status. One reconcile later the condition is WaitingForReplicas. During the rolling update READY briefly shows 4, the surge Pod, and the condition stays False although three or more Pods are ready, because old Pods are still part of the count. Only when all three replicas run the new template does it flip to AllReplicasReady. An earlier version of this operator compared only readyReplicas with replicas and reported AllReplicasReady in the middle of the rollout. That is the kind of bug a careful status definition prevents.

Step 9 — Leader Failover

Delete the Pod that holds the Lease:

Twenty seconds later:

The leadership went to the replacement Pod 8p7gn, not to the old follower 6xs5s. Leader election only guarantees that there is at most one leader, not which replica wins; whoever retries first after the lease expires gets it. Prove that the new leader is working:

Step 10 — Deletion with a Finalizer

Add a second app, 5-pricing.yaml:

Delete greeter while the operator runs:

Two mechanisms worked together. The finalizer let the operator remove the catalog entry in dotnetapp-system before the object vanished. Once it was gone, the garbage collector deleted its Deployment and Service through their owner references; the operator wrote no code for that. The later events for greeter find nothing and do nothing.

Now the failure case: delete pricing while no operator runs.

"Deleted", says kubectl, but the object is still there with a deletionTimestamp, held by the finalizer. Its Deployment keeps running too: the garbage collector waits until the owner is really gone. Without --wait=false, kubectl would hang here. Start the operator again:

The new replicas first saw the stale Lease of the scaled-down leader 8p7gn, waited for it to expire, and then jklp7 took over. Its initial list found pricing with a deletion timestamp, ran the cleanup and released the finalizer. Level-triggered reconciliation needed no record of the missed delete event, only the current state. The catalog is empty and the namespace has no apps left.

Step 11 — Clean Up

Delete the apps before the operator, so that it can still release its finalizers. In the other order, every remaining DotnetApp waits for an operator that no longer exists, and its namespace stays in Terminating until you remove the finalizers by hand with the patch from the Hands-On section.

Summary

1. An operator is a custom resource plus a controller that turns it into ordinary Kubernetes objects and keeps them that way.
2. Controllers are level-triggered: events only enqueue a key, and the reconcile reads the current state. Every reconcile must be idempotent; a periodic resync covers missed events.
3. A de-duplicating work queue with one worker prevents parallel reconciles of the same object, and exponential backoff protects the API server from failing objects.
4. Watch the children too and map their events back through the owner reference. That is how a deleted or hand-edited Deployment is repaired within a second.
5. Owner references let the garbage collector delete children; server-side apply with a field manager makes "create or update" a single idempotent call and takes back fields others changed.
6. Report progress with status, a Ready condition and observedGeneration, write status only when it changes, and define "ready" carefully enough to be false during a rollout.
7. The scale subresource makes kubectl scale and the HPA work on your resource.
8. Use a finalizer only for cleanup owner references cannot do, such as objects in another namespace. While the controller is down, deletion waits.
9. Grant the operator only what its reconcile uses, and run several replicas with leader election on a Lease, so exactly one of them acts.