Kubernetes Probes: Liveness, Readiness and Startup

Lesson 10 of 40 9 min read Updated

Kubernetes can tell that your container process is running, but it cannot tell whether your application is actually working. A .NET API can be alive as a process and still be stuck in a deadlock, waiting for a database connection, or busy warming up its caches. Probes are how you teach Kubernetes to see the difference.

Think of a hospital ward. A nurse checks a patient's pulse regularly (is the patient alive?), a doctor decides when the patient is well enough to receive visitors (is the patient ready?), and during recovery from surgery nobody checks the usual vitals until the patient has woken up (is the patient still starting?). Kubernetes has one probe for each of these questions.

This lesson covers the three probe types, the four ways to run a probe, the timing parameters, and how to wire them to ASP.NET Core health checks so your Pods restart when they are broken and receive traffic only when they are ready.

What you will learn

  1. The difference between liveness, readiness and startup probes and what Kubernetes does when each one fails
  2. The four probe mechanisms: httpGet, tcpSocket, exec and grpc
  3. How to tune initialDelaySeconds, periodSeconds, timeoutSeconds, successThreshold and failureThreshold
  4. How to expose separate liveness and readiness endpoints in ASP.NET Core
  5. The most common probe mistakes and how to avoid them

Why probes matter

Without probes, Kubernetes only knows whether the container's main process is running. That leads to two classic problems:

  • A new Pod is added to the Service as soon as the container starts, so users hit an application that is still booting and get errors.
  • A Pod whose application has hung keeps running forever, because the process has not exited, and keeps receiving traffic it cannot serve.

Probes solve both. The kubelet on each node runs them periodically and acts on the result.

The three probe types

ProbeQuestion it answersOn failure
livenessProbeIs the container still healthy, or is it stuck?The kubelet kills the container and restarts it according to restartPolicy.
readinessProbeIs the container ready to receive traffic right now?The Pod is removed from the Service endpoints. The container is not restarted.
startupProbeHas the application finished starting?Liveness and readiness probes are held back until it succeeds. If it never succeeds within its budget, the container is restarted.
Readiness is reversible, liveness is not. A Pod that fails readiness can become ready again and will be put back into the Service. A Pod that fails liveness is killed. Use readiness for temporary conditions such as a slow dependency, and liveness only for states that a restart can fix.

Probe mechanisms

Every probe uses exactly one of four mechanisms to check the container:

MechanismHow it worksBest for
httpGetSends an HTTP GET. Any status from 200 to 399 is a success.ASP.NET Core and other web apps
tcpSocketTries to open a TCP connection to a port. Success if the connection opens.Databases, brokers, non-HTTP servers
execRuns a command inside the container. Exit code 0 is a success.Workers and legacy apps without a port
grpcCalls the standard gRPC Health Checking Protocol.gRPC services

An HTTP probe looks like this:

readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
periodSeconds: 5

A TCP probe and an exec probe:

livenessProbe:
tcpSocket:
port: 5672
periodSeconds: 10
---
livenessProbe:
exec:
command: ["cat", "/tmp/healthy"]
periodSeconds: 10

A gRPC probe needs the service to implement grpc.health.v1.Health, which ASP.NET Core gRPC supports through the Grpc.AspNetCore.HealthChecks package:

livenessProbe:
grpc:
port: 8080
periodSeconds: 10

Probe parameters

All three probe types share the same timing parameters:

FieldDefaultMeaning
initialDelaySeconds0Seconds to wait after the container starts before the first probe
periodSeconds10How often to run the probe
timeoutSeconds1How long to wait for a response before counting a failure
successThreshold1Consecutive successes needed to become healthy again (must be 1 for liveness and startup)
failureThreshold3Consecutive failures before Kubernetes acts

The time Kubernetes tolerates before acting is roughly failureThreshold × periodSeconds. With the defaults, a failing liveness probe restarts the container after about 30 seconds.

Startup probes for slow applications

Some .NET applications take a while to start: they run EF Core migrations, load large caches or warm up a JIT. If the liveness probe starts too early, it kills the container before it has finished booting, and the Pod ends up in CrashLoopBackOff.

You could raise initialDelaySeconds on the liveness probe, but then a genuinely broken container is detected late forever. The startup probe fixes this: it allows a long startup budget once, and then gets out of the way.

startupProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 5
failureThreshold: 30

This gives the app up to 30 × 5 = 150 seconds to start. As soon as the startup probe succeeds once, the liveness and readiness probes take over with their own, tighter settings.

Health checks in ASP.NET Core

ASP.NET Core ships with a health check middleware. The key design decision is to expose two endpoints with different meanings:

  • /healthz/live answers "is the process able to respond at all?" and checks nothing external.
  • /healthz/ready answers "can I serve requests right now?" and checks dependencies such as the database.
using Microsoft.AspNetCore.Diagnostics.HealthChecks;
var builder = WebApplication.CreateBuilder(args);
builder.Services.AddHealthChecks()
.AddNpgSql(builder.Configuration.GetConnectionString("Default")!,
name: "database", tags: new[] { "ready" });
var app = builder.Build();
// Liveness: no checks run, so it only proves the app responds.
app.MapHealthChecks("/healthz/live", new HealthCheckOptions
{
Predicate = _ => false
});
// Readiness: runs only the checks tagged "ready".
app.MapHealthChecks("/healthz/ready", new HealthCheckOptions
{
Predicate = check => check.Tags.Contains("ready")
});
app.Run();
Never check dependencies in the liveness probe. If the database goes down and every Pod's liveness probe fails because of it, Kubernetes restarts all your Pods at once. Restarting does not fix the database, and now your whole API is down as well. Dependency checks belong in the readiness probe only.

A complete example

This Deployment combines all three probes for a .NET Order API:

apiVersion: apps/v1
kind: Deployment
metadata:
name: order-api
spec:
replicas: 3
selector:
matchLabels:
app: order-api
template:
metadata:
labels:
app: order-api
spec:
containers:
- name: order-api
image: myregistry.azurecr.io/order-api:1.0
ports:
- containerPort: 8080
startupProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 5
failureThreshold: 30
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
periodSeconds: 10
timeoutSeconds: 2
failureThreshold: 3
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
periodSeconds: 5
timeoutSeconds: 2
failureThreshold: 3
successThreshold: 1

Here is what happens when the Pod starts. Only the startup probe runs for the first few seconds. When /healthz/live answers, the startup probe is done, and the liveness and readiness probes begin. The Pod joins the Service only after /healthz/ready succeeds, which means the database connection works.

Common mistakes

  • Liveness probe that checks dependencies. As shown above, this can cause a cascading restart of every Pod.
  • Identical liveness and readiness endpoints with no startup probe. A slow start then triggers restarts before the app is up.
  • Timeouts that are too tight. The default timeoutSeconds of 1 is often too short for a busy .NET app during a garbage collection or a CPU spike. Raise it to 2 or 3 seconds.
  • Expensive probe endpoints. The probe runs every few seconds on every replica. Keep it cheap and do not run heavy queries.
  • Redirects to HTTPS. If UseHttpsRedirection answers a plain HTTP probe with a 307, it is still a success for Kubernetes (200 to 399), but the probe is not testing what you think. Probe the HTTP port directly.
Rule of thumb: use a startup probe to protect slow starts, a liveness probe for "restarting would help", and a readiness probe for "can serve traffic right now".

Hands-on: see probes in action

Step 1 — Deploy a Pod with a failing readiness probe

Save this as probe-demo.yaml. It uses nginx and probes a file that does not exist yet:

apiVersion: v1
kind: Pod
metadata:
name: probe-demo
spec:
containers:
- name: web
image: nginx:1.27
readinessProbe:
exec:
command: ["cat", "/tmp/ready"]
periodSeconds: 5
livenessProbe:
httpGet:
path: /
port: 80
periodSeconds: 10
kubectl apply -f probe-demo.yaml
kubectl get pod probe-demo

The Pod shows 0/1 in the READY column while its status is Running. The container is running, but it is not ready.

Step 2 — Inspect the probe failures

kubectl describe pod probe-demo

In the Events section you will see Readiness probe failed messages. Because it is only a readiness failure, the container is not restarted.

Step 3 — Make the Pod ready

kubectl exec probe-demo -- touch /tmp/ready
kubectl get pod probe-demo -w

Within five seconds the probe succeeds and the Pod switches to 1/1. It would now be included in a Service's endpoints.

Step 4 — Break liveness on purpose

kubectl exec probe-demo -- rm /usr/share/nginx/html/index.html
kubectl get pod probe-demo -w

The liveness probe now gets a 404, fails three times in a row, and the kubelet restarts the container. The RESTARTS column goes to 1 and kubectl describe pod probe-demo shows Liveness probe failed: HTTP probe failed with statuscode: 404. Clean up with kubectl delete pod probe-demo.

Summary

  1. Probes let the kubelet check the health of your application, not just whether its process is running.
  2. A failed liveness probe restarts the container. A failed readiness probe only removes the Pod from Service endpoints. A startup probe delays the other two while a slow app boots.
  3. Probes use one of four mechanisms: httpGet, tcpSocket, exec or grpc.
  4. Tune periodSeconds, timeoutSeconds and failureThreshold. Kubernetes acts after about failureThreshold × periodSeconds.
  5. In ASP.NET Core, expose a dependency-free /healthz/live endpoint and a /healthz/ready endpoint that checks dependencies, using tagged health checks.
  6. Never put external dependency checks in the liveness probe, and keep probe endpoints cheap.

With probes in place, Kubernetes restarts broken Pods automatically and sends traffic only to Pods that are ready to handle it.