Running ASP.NET Core on Kubernetes — Health Checks, Graceful Shutdown and Configuration

Kubernetes treats every container the same way. It starts it, probes it, sends it traffic, and one day sends it SIGTERM. Whether that goes smoothly depends on how well the application inside plays along. An ASP.NET Core app that ignores the platform still runs, but it drops requests on every deployment, gets restarted for the wrong reasons, never notices configuration changes, and runs as root with more memory than it is allowed to use.

This lesson is about the ASP.NET Core side of the contract. You build one small .NET 10 API and a Deployment for it, and fix each of those problems with real measurements on a four-node kind cluster. The headline result: during a rolling update under load, a naive Deployment dropped 7 of 280 requests; the finished one dropped 0 of 276.

We will cover the following topics:

1. Health checks in ASP.NET Core, and mapping them to startup, readiness and liveness probes
2. What happens between kubectl delete pod and process exit, and how graceful shutdown works in .NET
3. HostOptions.ShutdownTimeout, terminationGracePeriodSeconds and the native preStop sleep action
4. Configuration from environment variables, ConfigMap files and Secret files, including reload on change
5. Chiseled images, non-root users and a read-only root filesystem
6. How .NET reacts to CPU and memory limits: processor count, GC mode and heap limit

Technical Requirements

1. The multi-node kind cluster used throughout this series: Docker Desktop's Kubernetes with the kind provisioner (one control-plane and three workers, Kubernetes v1.36), or kind create cluster with an equivalent config.
2. kubectl, Docker and the .NET 10 SDK.
3. Familiarity with Deployments, Services, probes, ConfigMaps and Secrets from earlier lessons.

Core Concepts

Health Checks Mapped to Probes

Kubernetes asks three different questions, and each deserves its own answer from the application:

The last row is the classic mistake. If liveness checks the database and the database has a hiccup, Kubernetes restarts every replica at once. That turns a short dependency blip into a full outage. Dependencies belong in readiness: the Pod stops getting traffic, but keeps running and recovers by itself.

ASP.NET Core's health checks support this with tags. You register all checks once, tag them, and map one endpoint per probe that runs only the checks with a given tag.

What Happens When a Pod Is Deleted

During a rolling update, scale-down, node drain or kubectl delete pod, two things start at the same moment:

1. The control plane marks the Pod as terminating. The EndpointSlice controller removes it from the Service, and every node's kube-proxy (and every ingress or gateway) updates its rules. This takes some time to propagate across the cluster.
2. The kubelet on the Pod's node runs the preStop hook, if there is one, and then sends SIGTERM to the container.

Without a preStop hook, step 2 usually wins the race. The app receives SIGTERM and stops accepting connections, while some nodes still send it new requests for a moment. Those requests fail with "connection refused" or "connection reset". On top of that, if the app does not wait for requests already in flight, those are cut off as well.

The fix has three parts:

The budget must add up: terminationGracePeriodSeconds > preStop sleep + ShutdownTimeout. The default grace period is 30 seconds, and .NET's default ShutdownTimeout is 30 seconds as well, which leaves no room for a preStop delay.

The sleep preStop action is built into the kubelet and generally available in current Kubernetes versions. It matters for .NET because the recommended chiseled images have no shell and no sleep binary, so the older exec: ["sh", "-c", "sleep 5"] workaround does not work there.

How .NET Handles SIGTERM

The generic host listens for SIGTERM. When it arrives:

1. IHostApplicationLifetime.ApplicationStopping fires.
2. Kestrel stops accepting new connections and lets running requests finish.
3. Hosted services get StopAsync calls.
4. ApplicationStopped fires, and the process exits with code 0.

If step 2 and 3 take longer than ShutdownTimeout, the host stops waiting and in-flight work is abandoned. A sensible pattern is to make readiness fail as soon as ApplicationStopping fires, so anything that still checks readiness stops sending traffic.

Configuration Sources and Precedence

WebApplication.CreateBuilder reads configuration in this order, later sources overriding earlier ones:

1. appsettings.json
2. appsettings.{Environment}.json
3. User secrets (Development only)
4. Environment variables
5. Command-line arguments

Anything you add with builder.Configuration.Add... afterwards comes after those and wins over them, environment variables included. Keep this in mind when you mount files from ConfigMaps.

Kubernetes gives you three ways to feed an app configuration:

AddKeyPerFile is part of the ASP.NET Core shared framework in .NET 10, so no extra NuGet package is needed. A file named Payment__ApiKey becomes the key Payment:ApiKey.

Never mount a ConfigMap or Secret with subPath if you want updates: subPath mounts are copied once and never refreshed.

Chiseled Images and Non-Root

Since .NET 8, Microsoft publishes chiseled Ubuntu images such as mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled. They contain only what the runtime needs: no shell, no package manager, and a non-root user named app with UID 1654 as the default user. All .NET images listen on port 8080 by default (ASPNETCORE_HTTP_PORTS=8080), because non-root users cannot bind to ports below 1024.

Combined with a strict securityContext (runAsNonRoot, readOnlyRootFilesystem, no privilege escalation, all capabilities dropped), an attacker who finds a bug in your code lands in a container with almost nothing to work with.

.NET and Container Limits

The .NET runtime reads the container's cgroup limits:

1. CPU limit sets Environment.ProcessorCount. That number sizes the thread pool and decides the GC mode: with a single CPU, .NET uses workstation GC even if the app asks for server GC.
2. Memory limit sets the GC heap hard limit. By default the GC heap may use 75% of the container's memory limit, leaving room for native memory and the runtime itself. DOTNET_GCHeapHardLimitPercent (a hexadecimal value) changes that.

Without a memory limit, .NET sees the whole node's memory and sizes itself for it. That is a common reason for surprising OOM kills.

Hands-On: Kubernetes Commands

Watch readiness change during startup

The READY column flips to 1/1 only after the readiness endpoint returns 200.

Follow a Pod's log while it shuts down

Shows the exact order of Application is shutting down and Stopped cleanly.

See the termination settings of a running Deployment

Check the shutdown budget at a glance.

Debug a chiseled container that has no shell

An ephemeral container shares the process namespace and can read the app's files through /proc/1/root.

Change settings and resources without editing YAML

Each command triggers a rolling update.

Step-by-Step Example

The Scenario

You run orders-api, an ASP.NET Core service behind a Kubernetes Service. It takes a few seconds to warm its caches after start, and some requests legitimately take over a second. Its currency and free-shipping threshold come from a ConfigMap, and a payment API key from a Secret. Your goals:

1. No dropped requests during rolling updates.
2. No traffic before warm-up has finished.
3. Configuration changes without redeploying.
4. A minimal, non-root image with correct resource limits.

Step 1 — The Application

Create the project:

Replace Program.cs:

In a real service, ReadinessCheck would also check the dependencies the API cannot work without, for example with AddNpgSql or AddRedis from the AspNetCore.HealthChecks packages, all tagged ready. The /runtime endpoint exists only so you can see what .NET thinks of its container.

Step 2 — A Chiseled, Non-Root Image

Copying the .csproj and restoring before copying the rest lets Docker cache the restore layer, so code changes do not re-download packages. Add a .dockerignore with bin/ and obj/, then build:

A note on tags in local clusters. Docker Desktop's Kubernetes nodes keep their own copy of every image they have run. If you rebuild and reuse the same tag with imagePullPolicy: IfNotPresent, the nodes keep running the old copy. While testing this lesson that is exactly what happened: a Pod started an older image with the same name, which ran as root, and the kubelet refused it with container has runAsNonRoot and image will run as root. Give every build a new tag (1.0, 1.1, ...) or use imagePullPolicy: Always with a registry. With a plain kind cluster, run kind load docker-image net-orders-api:1.0 after each build.

Step 3 — Configuration and Secret

Save as 1-config.yaml:

The Secret key is the file name the app will see, and AddKeyPerFile turns __ into the configuration separator :.

Step 4 — The Naive Deployment, Measured

Start with what many first deployments look like: no probes, no preStop, default rolling update, and a short shutdown timeout. Save as 2-deployment-naive.yaml:

To measure, run a load generator inside the cluster. A port-forward is useless here, because it is bound to a single Pod and dies with it. Save the script as loadgen.sh:

And a Pod that runs it, loadgen-pod.yaml:

Start the load, trigger a rolling update ten seconds later, and count the status codes:

000 means curl got no HTTP response at all: the connection was refused or reset. Seven failed requests in one minute, from a single routine deployment. Some hit Pods that had already stopped listening but were still in the endpoint list; others were in flight when a Pod gave up after one second. New Pods also received traffic the moment the process started, before warm-up.

Step 5 — The Production-Grade Deployment

Save as 3-deployment.yaml:

Each block has a job:

1. startupProbe gives the app up to 60 seconds to start (2 s × 30) before liveness is allowed to kill it.
2. readinessProbe checks every 2 seconds and reacts after one failure, so the Pod joins the Service only after warm-up and leaves it as soon as shutdown begins. You can see the warm-up in the log of a new Pod: Health check ready with status Unhealthy ... with message 'warming up caches' a few times, then nothing.
3. livenessProbe is slow and forgiving on purpose.
4. maxUnavailable: 0 and maxSurge: 1 mean a new Pod must become ready before an old one is taken away.
5. preStop sleep 5, ShutdownTimeout 25 and a grace period of 40 seconds form the shutdown budget.
6. The securityContext runs the app as UID 1654 with a read-only root filesystem and no capabilities. ASP.NET Core runs fine without a writable root filesystem. If your app needs a scratch directory, mount an emptyDir at /tmp.

Run exactly the same test again:

Zero failures. To see why, follow one Pod's log, start an 8-second request against it, and delete the Pod while the request runs:

The delete was issued at 08:58:10. SIGTERM arrived at 08:58:15, exactly after the 5-second preStop sleep. During those five seconds the Pod was already removed from the Service but still served normally. After SIGTERM, Kestrel waited for the 8-second request to complete, and the process exited cleanly 0.35 seconds later. The client got its answer.

Step 6 — Configuration: Precedence, Secrets and Reload

Ask the app what it sees:

Two things to notice:

1. freeShippingThreshold is 50, from the ConfigMap file, even though the environment variable Shop__FreeShippingThreshold says 75. The file source was added after CreateBuilder registered environment variables, so the file wins. If you want environment variables to override mounted files, call builder.Configuration.AddEnvironmentVariables() again after adding the files.
2. The Secret arrived as a file, /secrets/Payment__ApiKey, and became Payment:ApiKey. Unlike an environment variable, it does not show up in kubectl describe pod, in crash dumps of the environment, or in child processes.

Now change the ConfigMap:

The kubelet refreshes ConfigMap volumes on its sync loop, so it takes up to about a minute. In our test the app reported the new values 65 seconds after the patch, without a restart:

That only worked because of DOTNET_USE_POLLING_FILE_WATCHER=true. To see why, we removed that variable, rolled out, and changed the ConfigMap again to GBP. Four minutes later the app still answered USD. Yet the file had changed, which you can prove with an ephemeral debug container, since the chiseled image has no cat or ls:

The listing explains the problem. orders.json is a symlink to ..data/orders.json, and ..data is itself a symlink to a timestamped directory. The kubelet updates the volume by writing a new timestamped directory and atomically swapping the ..data link. The file .NET watches is never written to, so the default event-based file watcher never fires. The polling watcher compares timestamps every few seconds and notices the swap. Set DOTNET_USE_POLLING_FILE_WATCHER=true whenever you rely on reloadOnChange with ConfigMap or Secret volumes.

Also remember that only IOptionsMonitor and IOptionsSnapshot see reloaded values. A plain IOptions<T> is read once at startup.

Step 7 — What .NET Sees Inside Its Limits

With the limits from Step 5 (cpu: "1", memory: 256Mi):

1. processorCount is 1, derived from the CPU limit, not the node's 8 cores.
2. totalAvailableMemoryMb is 192, which is 75% of 256 MiB: the default GC heap hard limit.
3. isServerGC is false. ASP.NET Core asks for server GC, but with a single CPU the runtime falls back to workstation GC.
4. user is app, the non-root user.

Raise the CPU limit to 2 and cap the heap at 50% (hexadecimal 0x32):

Two processors, server GC, and a 128 MB heap limit. Practical guidance:

1. Always set a memory limit for .NET containers, equal to the request for predictable behaviour.
2. Small APIs with one CPU or less are fine with workstation GC. For memory-sensitive services with more CPUs, consider DOTNET_GCConserveMemory or switching to workstation GC with DOTNET_gcServer=0.
3. Lower the heap percentage if the app uses a lot of native memory (large buffers, native libraries), so the container is not OOM-killed by memory outside the GC heap.

Step 8 — Clean Up

Summary

1. Give each probe its own endpoint with tagged health checks: liveness checks only the process, readiness checks warm-up and dependencies, startup protects slow starts.
2. On deletion, endpoint removal and SIGTERM happen in parallel. A preStop sleep lets endpoint removal win, HostOptions.ShutdownTimeout lets in-flight requests finish, and terminationGracePeriodSeconds must cover both.
3. Fail readiness in ApplicationStopping, and use maxUnavailable: 0 so capacity never drops during a rollout. Measured result: 7 dropped requests became 0.
4. Sources added after CreateBuilder override environment variables. Mount Secrets as files and read them with AddKeyPerFile.
5. ConfigMap and Secret volumes are updated by swapping a symlink. Use DOTNET_USE_POLLING_FILE_WATCHER=true for reloadOnChange, never subPath, and read values through IOptionsMonitor or IOptionsSnapshot.
6. Chiseled images run as UID 1654 without a shell. Combine them with runAsNonRoot, readOnlyRootFilesystem and dropped capabilities, and debug them with kubectl debug.
7. .NET derives processor count, GC mode and heap limit from the container limits. Always set a memory limit.

In the next lesson you package a multi-service .NET Aspire application and deploy it to the same cluster.