Kustomize — Environment Overlays Without Templates

In the previous article we packaged the Catalog API as a Helm chart. Helm solves the multi-environment problem with templates: you write {{ .Values.replicaCount }} in the manifest and fill it in per environment. That is powerful, but it has a cost. The files in your repository are no longer valid Kubernetes YAML, and reading a chart means reading Go templates.

Kustomize takes the opposite approach. Every file stays plain, valid YAML. You write the manifests once as a base, and each environment is an overlay that says what to change: a different namespace, a different image tag, three replicas instead of one, an extra Secret. Kustomize is built into kubectl, so there is nothing to install.

In this article we deploy the same Catalog API to a development and a production namespace with Kustomize, and test every feature on a real cluster.

We will cover the following topics:

1. Bases, overlays and the kustomization.yaml file
2. Namespaces, name prefixes and labels
3. The images transformer for changing tags without editing manifests
4. Strategic merge patches and JSON 6902 patches
5. ConfigMap and Secret generators, and why their names end in a hash
6. Components for optional, reusable features
7. kubectl kustomize, kubectl apply -k, kubectl diff -k and pruning with ApplySets
8. When to choose Kustomize and when to choose Helm

Technical Requirements

1. A Kubernetes cluster and kubectl. The examples run on Docker Desktop's built-in Kubernetes with the kind provisioner (1 control-plane node and 3 workers, Kubernetes v1.36). A plain kind cluster with the same layout behaves the same.
2. kubectl 1.36 ships with Kustomize v5.8.1 built in. Check yours with kubectl version --client.
3. The catalog-api:1.0.0 and catalog-api:1.1.0 images from the previous article. On a plain kind cluster, load them with kind load docker-image catalog-api:1.0.0 catalog-api:1.1.0.

Core Concepts

Base and Overlays

A kustomization is a folder containing a file called kustomization.yaml. That file lists resources and the transformations to apply to them. A kustomization can use another kustomization as a resource, which is how layering works.

The folder layout used in this article:

The base never knows that overlays exist. Overlays point down at the base with a relative path such as ../../base. By default Kustomize refuses to load files from outside the kustomization's own folder tree, except through resources and components entries. That is the load restrictor, and it is a security feature, not a bug.

Transformers

Most of what an overlay does is set a field. Each of these fields is a built-in transformer:

The reference-updating part is what makes Kustomize more than sed. Rename a ConfigMap and every Deployment that mounts it is updated to match.

Avoid the older commonLabels field. It also adds labels to selectors, and changing a selector label on an existing Deployment fails, because selectors cannot be changed after creation. The labels field with includeSelectors: false is the safe replacement.

Two Kinds of Patches

When a transformer is not enough, you patch.

Both go under the patches field. A patch either sits in its own file (path:) or inline (patch:), and a target selects which resources it applies to by kind, name, label or annotation.

Generators and Hash Suffixes

configMapGenerator and secretGenerator build ConfigMaps and Secrets from literals, files or .env files. Kustomize appends a hash of the content to the name, for example catalog-config-hc622gmtk2, and rewrites every reference to use that name.

This fixes a real problem. Pods read environment variables from a ConfigMap only when they start. If you edit a ConfigMap in place, running Pods never notice. With a hash suffix, new content means a new name, the Deployment's Pod template changes, and Kubernetes rolls the Pods. It is the same idea as the checksum/config annotation in the Helm chart, done for you.

An overlay can extend a generator defined in the base with behavior: merge (add or override keys) or replace it entirely with behavior: replace.

Remember that secretGenerator only base64-encodes values. If the literal is in your Git repository, the secret is in your Git repository. For real secrets, use the Vault integration covered earlier in this course or a tool such as Sealed Secrets or External Secrets.

Components

A component is a kustomization of kind Component. It can add resources and apply patches, but it is only included where an overlay lists it under components. Use components for features that are optional rather than environment-specific: a PodDisruptionBudget, a debug sidecar, an OpenTelemetry configuration.

Hands-On: Kubernetes Commands

Render an overlay without applying it

Prints the final YAML. Always look at this before applying.

Apply an overlay

See what would change on the cluster

Shows a diff between the live objects and the rendered overlay. Exits with code 1 when there are differences, which is handy in CI.

Delete everything an overlay created

Apply with pruning through an ApplySet

Deletes objects that were applied earlier through the same ApplySet but are no longer rendered. ApplySet support in kubectl is still alpha, hence the environment variable.

Check the built-in Kustomize version

Step-by-Step Example

The Scenario

The Catalog API must run in two namespaces. Development runs one replica of the newest image, 1.1.0, with the ASP.NET Core environment set to Development. Production runs three replicas of the stable 1.0.0 image, with larger resources, a database connection string from a Secret, and a PodDisruptionBudget. Both share one set of base manifests.

Step 1 — The Base

base/deployment.yaml is a complete, valid manifest. You could kubectl apply -f it on its own.

base/service.yaml:

base/kustomization.yaml lists the resources, adds a label to everything, and generates the ConfigMap the Deployment refers to as catalog-config:

As in the Helm article, Catalog__Greeting becomes the .NET configuration key Catalog:Greeting.

Step 2 — The Development Overlay

overlays/dev/kustomization.yaml uses five features at once: a namespace, a name prefix, the images transformer, a generator merge, and a JSON 6902 patch that replaces the first environment variable's value.

Render it:

Check each change against the overlay. The ConfigMap was renamed twice: dev- from the prefix and -hg56f9dbt7 from the hash. The Deployment's configMapRef was rewritten to match. The image tag is 1.1.0 although no manifest says so. And the selector still only contains app: catalog-api, because the labels were added with includeSelectors: false.

Step 3 — The Production Overlay

Production needs more resources and three replicas. That is a strategic merge patch: a partial Deployment, merged into the base by name. Containers merge by their name, so only catalog-api's resources change.

overlays/prod/replicas-resources.yaml:

The connection string comes from a generated Secret. A JSON 6902 add operation with a path ending in /- appends the Secret to the existing envFrom list. A strategic merge patch could not do this cleanly, because envFrom entries have no merge key.

overlays/prod/kustomization.yaml:

The Secret is referenced as plain catalog-db in the patch. Kustomize rewrites it to the hashed name, just like the ConfigMap.

Step 4 — A Component for the PodDisruptionBudget

Only production wants a PodDisruptionBudget. Put it in a component so any future overlay, such as staging, can opt in with one line.

components/pdb/kustomization.yaml:

components/pdb/pdb.yaml:

The component receives the overlay's transformations, so the PDB lands in catalog-prod with the environment: prod label without saying so itself.

Render production and pick out the interesting lines:

Step 5 — Apply Both Overlays

The namespace field sets the namespace on resources but does not create it, so create the namespaces first. Alternatively, add a Namespace manifest to each overlay's resources.

Step 6 — Verify Each Environment

Port-forward both Services in two terminals:

Different version, environment and greeting, from one set of base manifests. Confirm that the Secret reached the production container:

In .NET that value is available as builder.Configuration.GetConnectionString("Catalog").

Step 7 — Change Configuration and Watch the Hash

Change the production greeting in overlays/prod/kustomization.yaml to Hello from production, v2, then diff before applying:

The diff tells the whole story: a new ConfigMap, and the Deployment pointing at it. Apply:

The Pods rolled, and they now read the new greeting. Look at the ConfigMaps, though:

The old one is still there. kubectl apply only creates and updates; it never deletes something you stopped rendering. Keeping the previous ConfigMap is useful for a rollback, but over months they pile up.

Step 8 — Prune Stale Objects with an ApplySet

An ApplySet is a small parent object, a Secret by default, that records which objects belong to one apply. With --prune, kubectl deletes members that are no longer in the rendered output. It is still alpha in kubectl, so it is enabled with an environment variable.

Change the greeting again, to v3, and apply with the same command:

The v2 ConfigMap was pruned. The very first one, hc622gmtk2, survived, because it was applied before the ApplySet existed and was never labelled as a member. Adopt an ApplySet from the first apply, or delete pre-existing leftovers once by hand. In the next article, Argo CD does this pruning for you.

Step 9 — Clean Up

kubectl delete -k deletes exactly what the overlay renders:

Then remove both namespaces, which also removes the leftover ConfigMap and the ApplySet Secret:

Helm or Kustomize?

They solve the same problem from opposite ends, and many teams use both.

A common pattern combines them: install third-party software such as cert-manager or Prometheus from its Helm chart, and manage your own .NET services with Kustomize overlays. You can also render a chart with helm template and use the output as a Kustomize resource, to patch the parts the chart author did not expose.

Summary

1. A base holds the shared manifests; an overlay references the base and declares one environment's differences. Every file stays plain, valid YAML.
2. Built-in transformers set the namespace, name prefixes, labels and image tags, and update references when names change. Use labels with includeSelectors: false instead of commonLabels.
3. Strategic merge patches look like partial manifests and merge lists by key; JSON 6902 patches make precise edits such as appending with /-.
4. Generators create ConfigMaps and Secrets with a content hash suffix, so a configuration change automatically rolls the Pods.
5. Components package optional features, such as a PodDisruptionBudget, that overlays opt into.
6. kubectl apply -k never deletes old objects. Use kubectl diff -k before applying, and an ApplySet with --prune to remove what you stopped rendering.
7. The namespace field does not create the namespace.

In the next article we stop running kubectl apply by hand altogether and let Argo CD keep the cluster in sync with Git.