Custom Resource Definitions — Extending the Kubernetes API

Everything you have used so far in this series, from Pods and Deployments to Services, ConfigMaps and Gateways, is a resource type that somebody added to the Kubernetes API. Some were compiled into the API server. Others, like the Gateway API, Argo CD Applications, KEDA ScaledObjects or cert-manager Certificates, were added from the outside with a CustomResourceDefinition (CRD). A CRD teaches the API server a new kind of object. From then on, kubectl get, RBAC, watches, validation, kubectl explain and etcd storage all work for it, without writing a line of server code.

In this lesson you define your own resource, a FeatureFlag, and let a .NET 10 API read it. The API server validates every flag with an OpenAPI schema and CEL rules, fills in defaults, prints custom columns, keeps status separate from spec, and serves the resource under two versions. The .NET side lists and watches the flags with the official KubernetesClient package, using a ServiceAccount that may only read flags in its own namespace. Changing a flag with kubectl patch changes the API's behavior within a second, without a restart.

We will cover the following topics:

1. What a CRD is, and the difference between a custom resource and a controller
2. The anatomy of a CRD: group, versions, scope, names, short names and categories
3. OpenAPI v3 schema validation, defaults and strict field validation
4. CEL validation rules with x-kubernetes-validations, including immutable fields
5. additionalPrinterColumns, and the status and scale subresources
6. Serving several versions with served, storage, deprecated and the conversion strategy None
7. Reading and watching custom resources from .NET with KubernetesClient
8. RBAC for custom resources, including aggregation into the built-in view role

Technical Requirements

1. A Kubernetes cluster and kubectl. The examples run on Docker Desktop's built-in Kubernetes with the kind provisioner (1 control-plane node and 3 workers, Kubernetes v1.36). A cluster created with kind create cluster and the same layout behaves the same; there, load the image you build with kind load docker-image.
2. Docker and the .NET 10 SDK to build the sample API.
3. jq and curl for the test commands.
4. The RBAC and ServiceAccounts lessons. Roles, RoleBindings and kubectl auth can-i are used here without being re-explained.

Core Concepts

Resources, Custom Resources and Controllers

The Kubernetes API is a collection of REST resources grouped by API group and version. deployments live in the group apps, version v1, at /apis/apps/v1/namespaces/<ns>/deployments. A CRD adds a new resource to that collection. After you apply a CRD for featureflags in the group flags.dotnetacademy.dev, the API server serves /apis/flags.dotnetacademy.dev/v1/namespaces/<ns>/featureflags and stores the objects in etcd like any built-in object.

A CRD on its own is only data. Nothing happens when you create a FeatureFlag except that it is validated and stored. Behavior comes from programs that read or watch these objects:

This split is why CRDs are so useful even without an operator. A FeatureFlag CRD gives you a typed, validated, RBAC-protected configuration store with a change feed, and every tool that speaks the Kubernetes API (kubectl, Argo CD, Helm, Lens) already understands it.

Anatomy of a CRD

The name of a CRD is always <plural>.<group>. The important fields:

Structural Schemas, Defaults and Unknown Fields

Every version must have a structural OpenAPI v3 schema: every field has a type, and objects list their properties. The API server uses it for three things:

1. Validation. minimum, maximum, pattern, enum, minLength, maxItems, required and so on are checked on every create and update.
2. Defaulting. A field with default is filled in when it is missing, both when an object is written and when it is read from etcd.
3. Pruning. Fields that are not in the schema are dropped. Current kubectl versions ask the server for strict field validation by default (--validate=strict), so a typo like rolloutPercnt is rejected with an error instead of being silently pruned.

kubectl explain reads the same schema, which is why the description fields are worth writing.

CEL Validation Rules

Some rules involve more than one field or compare the new object with the old one. Those are written in CEL (Common Expression Language) under x-kubernetes-validations. The API server evaluates them in-process. No webhook to run, deploy or keep available.

self is the value of the field the rule is attached to. oldSelf is the previous value and only exists on updates. A rule that uses oldSelf is skipped on create. CEL rules run only after the plain schema checks pass, which is why you will see the message "some validation rules were not checked because the object was invalid".

The ValidatingAdmissionPolicy lesson uses the same language for rules across many resource types. CRD rules are the right place for rules that belong to the type itself.

Printer Columns

By default kubectl get on a custom resource shows only NAME and AGE. additionalPrinterColumns adds columns from JSONPath expressions. Each column has a type (string, integer, number, boolean, date) that controls formatting. A date column is printed as an age, like the built-in AGE.

The Status and Scale Subresources

Kubernetes objects split spec (what the user wants) from status (what a controller observed). With subresources.status: {} the API server enforces that split:

It also makes metadata.generation meaningful: generation increases only when the spec changes, not on status writes. Controllers compare it with status.observedGeneration to tell whether they have caught up. RBAC can grant the two endpoints separately, so users edit specs and only the controller writes status.

subresources.scale maps a replica count inside your spec to the standard Scale API, through specReplicasPath, statusReplicasPath and labelSelectorPath. Then kubectl scale and the HorizontalPodAutoscaler work on your custom resource. The next lesson's DotnetApp resource uses it; a feature flag has no replicas, so this lesson does not.

Versions and Conversion

APIs evolve. A CRD can serve several versions at once, for example a deprecated v1alpha1 next to v1. Exactly one version has storage: true; objects are written to etcd in that version. status.storedVersions records every version that has ever been used for storage, which matters when you later want to remove one.

With conversion.strategy: None, the API server converts between versions by changing only the apiVersion string. That is fine when the schemas are identical, as in a rename from v1alpha1 to v1. It is dangerous when they are not: each version validates with its own schema, so a loose old version becomes a back door for invalid data, as the step-by-step example shows. When the field layout really changes between versions, you need a conversion webhook.

deprecated: true with a deprecationWarning makes the API server return a warning header on every request to that version. kubectl prints it, and client libraries can log it.

Talking to Custom Resources from .NET

The official KubernetesClient NuGet package (version 21.0.8 in this lesson) has no generated classes for your types, but its CustomObjects API takes the group, version, namespace and plural as strings: ListNamespacedCustomObjectAsync<T>, WatchListNamespacedCustomObjectAsync, GetNamespacedCustomObjectAsync<T>, PatchNamespacedCustomObjectAsync and PatchNamespacedCustomObjectStatusAsync. You write the C# classes yourself, implementing IKubernetesObject<V1ObjectMeta> for the object and IItems<T> for the list. The watch returns each object untyped, as a JsonElement, which KubernetesJson.Deserialize<T> maps onto your class.

The standard way to keep a local copy is list, then watch: list once, remember the list's resourceVersion, and start the watch from it, so no event between the two calls is lost. When the server closes the watch, list again.

RBAC for Custom Resources

Custom resources are ordinary RBAC resources. A rule names the API group and the plural:

The built-in view, edit and admin ClusterRoles do not know about your type. They are aggregated roles: any ClusterRole labeled rbac.authorization.k8s.io/aggregate-to-view: "true" has its rules merged into view by the controller manager. Shipping such a ClusterRole with your CRD means every existing view binding immediately covers the new type.

Hands-On: Kubernetes Commands

List the resources an API group serves

Shows the plural, short names and subresources a CRD added.

Read the schema of a custom resource

Works for every CRD in the cluster, including the ones Helm charts install.

List custom resources by short name and category

Write only the status

Without --subresource=status, status changes are silently ignored.

Ask for a specific version

The fully qualified resource name selects the version.

Check which versions exist and which are stored

Check what a ServiceAccount may do with a custom resource

Step-by-Step Example

The Scenario

A team wants feature flags without running a separate feature-flag service. Each flag is a Kubernetes object in the application's namespace, owned by a team, with a rollout percentage and the environments it applies to. The rules:

1. rolloutPercent is between 0 and 100 and defaults to 100.
2. enabled defaults to true. A disabled flag must have rolloutPercent: 0.
3. environments may only contain dev, staging and prod.
4. owner is required, lowercase, and can never change.

A .NET 10 API, flags-api, keeps all flags of its namespace in memory and answers "is flag X on for user Y?". It must pick up changes immediately and must not be able to change flags.

Everything runs in a new namespace, crd-lab. The CRD itself and one ClusterRole are cluster-scoped and are removed in the last step.

Step 1 — Define the FeatureFlag CRD

Save this as 1-featureflag-crd.yaml:

Apply it and wait for the API server to accept the names and start serving the resource:

The API server now serves a new resource. Note the second entry: the status subresource is its own endpoint with its own verbs.

kubectl explain works from the schema. If it reports couldn't find resource right after the CRD was created, kubectl's discovery cache is a few seconds behind; run it again.

Step 2 — Create Flags and See the Defaults

Save three flags as 2-flags.yaml. dark-mode sets only the owner; legacy-search is switched off.

The columns come from additionalPrinterColumns. dark-mode shows true and 100 although the manifest did not set them: the API server applied the schema defaults before storing the object.

kubectl get flags -n crd-lab prints the same table through the category, and the object is also a plain JSON document at /apis/flags.dotnetacademy.dev/v1/namespaces/crd-lab/featureflags/<name>, which kubectl get --raw can fetch like any other REST path.

Step 3 — Let the API Server Reject Bad Flags

Each of the following files breaks one rule. First, three schema violations in one object, 3-invalid-flags.yaml:

All three errors come back at once. The last line tells you the CEL rules were skipped because the plain schema checks already failed.

Now a flag that passes the schema but breaks the cross-field CEL rule, 4-disabled-with-rollout.yaml:

The transition rule on owner turns it into a write-once field:

Two more files: 4b-no-owner.yaml leaves out owner, and 4c-typo.yaml has a valid owner but spells rolloutPercent as rolloutPercnt: 10.

The typo is caught by strict field validation. With an older client, or with --validate=false, the API server would have pruned the unknown field and stored a flag with the default rolloutPercent: 100, the opposite of what the author meant.

Step 4 — Status Is a Separate Endpoint

Suppose a review tool records who last reviewed a flag in .status. Watch metadata.generation while you try three different writes:

A normal patch cannot touch status: "no change", and the status is still empty. The status endpoint can:

The status changed and the generation did not. Change the spec, and the generation moves while the status stays:

Step 5 — Serve a Second Version, and the Conversion Trap

Older clients still send flags.dotnetacademy.dev/v1alpha1. A tempting shortcut is to add that version with a schema that accepts anything. This is 5-crd-two-versions-wrong.yaml: the same CRD as Step 1, with this entry appended to versions and the conversion strategy set:

The same three objects are visible through both versions, and the old version carries a deprecation warning. Its table has no custom columns, because those are defined per version. Now send an invalid flag through the old version, 5b-sneaky-v1alpha1.yaml:

sneaky was validated against the loose v1alpha1 schema, converted with strategy None (only apiVersion changed) and stored as v1. Every v1 client, including the .NET API you are about to build, now receives a flag with a rollout of 150 percent and an owner that the v1 schema forbids. Remove it and fix the CRD.

With strategy None, every served version must have the same schema. A YAML anchor keeps the two copies identical. This is 5-crd-two-versions.yaml: in the v1 entry, the line schema: becomes schema: &featureflag-schema, and the v1alpha1 entry reuses it:

The old version is still served, but it now enforces the same rules. Reading through it returns the same data with a different apiVersion, and etcd still holds only v1:

Because storedVersions contains only v1, you can later set served: false on v1alpha1 and remove it without migrating any stored objects.

Step 6 — The flags-api in .NET 10

Create the project and add the client library. System.IO.Hashing provides a fast, stable hash for putting users into rollout buckets.

FeatureFlag.cs describes the custom resource as C# classes. Only the fields the API needs are mapped; the JSON names match the CRD schema.

FlagCache.cs is a BackgroundService that implements list-then-watch. It starts the watch at the list's resourceVersion, so no change between the two calls is lost, and it re-lists whenever the watch ends or fails.

Program.cs wires it up. InClusterConfig() reads the ServiceAccount token that Kubernetes mounts into the Pod; outside the cluster the same code uses your kubeconfig. A user always lands in the same bucket for a flag, so raising the rollout from 25 to 50 percent keeps the first 25 percent of users in and adds new ones.

The image uses the chiseled ASP.NET Core runtime, which runs as a non-root user by default:

Step 7 — A ServiceAccount That Can Only Read Flags

Save as 6-flags-api-rbac.yaml:

Read and watch in crd-lab, nothing else: no writes, no other namespaces, and no access to the status subresource, which RBAC treats as a separate resource.

Deploy the API with 7-flags-api.yaml. The Downward API passes the Pod's namespace to the app, and the readiness probe stays red until the first list has filled the cache.

Step 8 — Evaluate Flags and Change Them Live

Forward a local port to the Service. Port 18091 is arbitrary; pick any free port.

new-checkout is at 50 percent since Step 4, and the API runs as staging. Users with a bucket below 50 get the feature:

Roll the feature out to everyone. No redeploy, no restart:

Restrict it to dev, and the staging API switches it off for everyone:

Add one flag (9-search-v2.yaml) and delete another:

Both replicas received every event over their watch: two Modified events for the two patches, one Added, one Deleted. The cache follows the cluster without polling.

Step 9 — Make the Built-in view Role Cover FeatureFlags

An auditor bound to the built-in view ClusterRole can read almost everything in the namespace, but not your new type:

Ship an aggregated ClusterRole with the CRD, 8-aggregate-to-view.yaml:

The controller manager copied the rule into view. Every existing view binding in the cluster now covers FeatureFlags, read-only. Use aggregate-to-edit and aggregate-to-admin the same way for write access.

Step 10 — Clean Up, and What Deleting a CRD Does

Deleting a CRD deletes every object of that type in every namespace. There is no confirmation and no undo, so treat CRD deletion like dropping a database table.

The three flags are gone, the API path no longer exists, and the .NET cache keeps retrying every five seconds, which is the behavior you want from a consumer: it recovers on its own when the CRD comes back. Finish the cleanup:

Summary

1. A CRD adds a resource type to the Kubernetes API: REST endpoints, etcd storage, watches, RBAC and kubectl support come for free. Behavior comes from programs that read or watch the objects.
2. The structural schema validates, defaults and prunes; strict field validation rejects typos. CEL rules add cross-field rules and transition rules such as self == oldSelf, without a webhook.
3. additionalPrinterColumns, shortNames and categories shape the kubectl experience.
4. The status subresource separates intent from observation, and metadata.generation moves only with the spec. The scale subresource connects a replica field to kubectl scale and the HPA.
5. One version is the storage version, recorded in status.storedVersions. With conversion None, give every served version the same schema, or an old version becomes a way around your validation.
6. From .NET, use KubernetesClient's CustomObjects API with list-then-watch, and re-list when the watch ends.
7. Grant custom resources like any other resource, keep /status separate, and ship an aggregate-to-view ClusterRole.
8. Deleting a CRD deletes all of its objects in every namespace.