Lesson 66 of 72 · 31 min read · Updated
Cluster Maintenance — Cordon, Drain, Certificates and Upgrades
On this page
Sooner or later every node in a cluster has to be taken out of service. The operating system needs a security patch, the disk is full, the VM is moved to a new host, or Kubernetes itself must be upgraded. Day-2 operations like these are where clusters lose availability. Usually the cause is not a bug but a routine command run in the wrong order.
This lesson is about doing that maintenance without your users noticing. You build a small .NET 10 inventory API, run it on a disposable three-node kind cluster, and keep a load generator firing requests at it the whole time. You drain nodes, watch a node go NotReady, renew the control plane certificates, and finally perform a real Kubernetes upgrade from v1.35.8 to v1.36.4 with kubeadm. The headline result: a naive single-replica Deployment lost requests on every drain, while the prepared one answered 4,243 of 4,243 requests during the complete cluster upgrade.
We will cover the following topics:
- The node maintenance workflow: cordon, drain and uncordon
- What
kubectl drainreally does, its important flags, and why it uses the Eviction API instead of deleting Pods - How PodDisruptionBudgets, replicas, topology spread and readiness together make a drain invisible to clients
- What happens when a node fails instead of being drained: NotReady, the
node.kubernetes.io/unreachabletaint andtolerationSeconds - Checking and renewing kubeadm certificates
- The version skew policy and the order in which components must be upgraded
- A complete kubeadm upgrade:
upgrade plan,upgrade apply, kubelet upgrades andupgrade node - How managed clusters (AKS, EKS, GKE) handle the same work with surge upgrades
Technical Requirements
- A disposable kind cluster created with the kind CLI, not Docker Desktop's built-in cluster used in most other lessons of this series. Draining nodes, stopping a kubelet, restarting the control plane and upgrading Kubernetes would disrupt every workload you run on your everyday cluster, and Docker Desktop manages its own Kubernetes version. A throw-away cluster costs a minute to create and can be deleted afterwards. The examples use kind v0.33.0 with the images
kindest/node:v1.35.8andkindest/node:v1.36.4. The lesson Setting Up a Local Multi-Node Cluster with kind and Docker Desktop explains the kind CLI in detail. - Docker with about 3 CPUs and 4 GB of memory free for a three-node cluster.
kubectl(the examples use v1.36.1) and the .NET 10 SDK to build the sample API.- The lesson PriorityClass, Preemption and PodDisruptionBudgets introduces PodDisruptionBudgets. This lesson uses them as one tool among several and does not re-explain every field.
Core Concepts
Voluntary and Involuntary Disruptions
Kubernetes distinguishes two kinds of trouble for a Pod:
| Kind | Examples | Can Kubernetes coordinate it? |
|---|---|---|
| Voluntary disruption | kubectl drain, node pool upgrades, cluster autoscaler scale-down, deleting a Deployment | Yes. Tools that use the Eviction API ask first and respect PodDisruptionBudgets. |
| Involuntary disruption | Hardware failure, kernel panic, a VM deleted by mistake, network partition, node out of memory | No. Pods simply disappear. Only spare replicas on other nodes help. |
Maintenance is the voluntary kind, and that is good news: you decide when it happens, so you can make sure the application is ready for it. Everything in this lesson either prepares an application for voluntary disruptions or limits the damage of involuntary ones.
Cordon, Drain and Uncordon
A maintenance window for one node always follows the same three steps:
| Step | Command | What changes |
|---|---|---|
| Cordon | kubectl cordon <node> | Sets spec.unschedulable: true. The scheduler places no new Pods on the node. Running Pods are not touched. The node shows Ready,SchedulingDisabled. |
| Drain | kubectl drain <node> --ignore-daemonsets --delete-emptydir-data | Cordons the node if needed, then evicts every Pod on it and waits until they are gone. |
| Uncordon | kubectl uncordon <node> | Clears unschedulable. New Pods may land on the node again. |
Uncordoning does not move anything back. Pods that were evicted stay where the scheduler put them, so after a series of drains the load is often uneven. A kubectl rollout restart of the affected Deployments spreads them again, and the Descheduler project can do it automatically.
What kubectl drain Really Does
kubectl drain is a client-side loop, not a single API call. It lists the Pods on the node, filters them, and then for each Pod sends an Eviction to the API server. An eviction is a request to delete a Pod that the API server is allowed to refuse. It refuses with HTTP 429 Too Many Requests when the eviction would violate a PodDisruptionBudget, and drain then retries every 5 seconds until it succeeds or its --timeout runs out.
Before evicting anything, drain refuses to continue when it finds Pods it cannot handle safely. Each refusal has its own flag:
| Pod on the node | Default behaviour | Flag | Why the default is cautious |
|---|---|---|---|
| Managed by a DaemonSet | Drain stops | --ignore-daemonsets | The DaemonSet controller would immediately recreate the Pod on the same node. Skipping them is almost always what you want. |
Uses an emptyDir volume | Drain stops | --delete-emptydir-data | The data in the emptyDir is lost when the Pod leaves the node. |
| Has no controller (a bare Pod) | Drain stops | --force | Nothing will recreate the Pod elsewhere. It is gone for good. |
| Static Pod (mirror Pod) | Ignored | none | The kubelet owns it through a manifest file. Drain cannot remove it. |
A few more flags matter in practice:
--timeout=5mlimits the whole drain. Without it a drain blocked by a budget waits forever.--pod-selector app=inventory-apievicts only matching Pods. Useful to move one application without draining everything.--grace-period=Noverrides each Pod'sterminationGracePeriodSeconds. Leave it at the default so applications can shut down cleanly.--dry-run=servershows what would happen, including the refusals, without changing anything.--disable-evictiondeletes Pods instead of evicting them. This bypasses every PodDisruptionBudget. It exists for clusters where the Eviction API is unavailable or for emergencies, and it should never be part of a normal runbook.
Why Eviction and Not Delete
Deleting a Pod and evicting it end the same way, with a graceful termination, but only an eviction checks PodDisruptionBudgets first. That is why every well-behaved tool that removes Pods for maintenance, drain, the cluster autoscaler and managed node pool upgrades, uses evictions. A budget does not protect you from someone running kubectl delete pod in a loop. The lesson PriorityClass, Preemption and PodDisruptionBudgets compares the two paths in detail.
Making a Drain Invisible to Clients
A PodDisruptionBudget on its own does not prevent errors. Four things have to work together:
| Ingredient | What it prevents |
|---|---|
| More than one replica | With one replica, every eviction is a full outage until the replacement is ready. |
| Topology spread or anti-affinity | Replicas on different nodes, so one drain cannot take them all at once. |
| PodDisruptionBudget | Drain evicts replicas one at a time and waits until the replacement counts as healthy. |
| Readiness probe plus graceful shutdown | The replacement receives traffic only when it can answer, and the old Pod leaves the Service endpoints before it stops. The lesson Running ASP.NET Core on Kubernetes — Health Checks, Graceful Shutdown and Configuration covers the .NET side in depth. |
The Step-by-Step section measures the difference: the same API with and without these ingredients, under the same drain.
When a Node Fails Instead of Being Drained
Drain is planned. A crashed node is not, and Kubernetes reacts on a timer:
| Time after the kubelet stops reporting | What happens |
|---|---|
| 0 s | The kubelet stops renewing its Lease in kube-node-lease. |
| about 50 s | The node lifecycle controller in kube-controller-manager marks the node Ready=Unknown (NotReady), adds the taints node.kubernetes.io/unreachable:NoSchedule and node.kubernetes.io/unreachable:NoExecute, and marks its Pods not ready, which removes them from Service endpoints. |
tolerationSeconds later | Pods whose toleration for the NoExecute taint has expired are evicted. Their controllers create replacements on healthy nodes. |
Unless a Pod declares its own, the DefaultTolerationSeconds admission plugin gives it tolerations for node.kubernetes.io/not-ready and node.kubernetes.io/unreachable with tolerationSeconds: 300: five minutes during which replicas on a dead node still count. A fast-starting stateless API is better served by about 30 seconds; a stateful workload that is expensive to move after a short network blip by the default or more.
The evicted Pods stay in Terminating until the node comes back or someone force-deletes them, because only the kubelet can confirm that the containers have really stopped.
kubeadm Certificates
A kubeadm cluster runs on certificates: the API server's serving certificate, the client certificates the control plane components use to talk to each other and to etcd, and the certificates embedded in admin.conf, controller-manager.conf and scheduler.conf. kubeadm issues these leaf certificates for one year. The certificate authorities (ca, etcd-ca, front-proxy-ca) are valid for ten years.
When a leaf certificate expires, the control plane stops working: components can no longer authenticate to the API server or to etcd. Three facts keep that from happening:
kubeadm certs check-expirationlists every certificate with its expiry date.kubeadm upgrade applyandkubeadm upgrade noderenew all certificates on the node they run on. A cluster that is upgraded at least once a year never reaches expiry.kubeadm certs renew allrenews them by hand. Afterwards the control plane static Pods must be restarted, because some components read their certificates and kubeconfig files only at startup.
Kubelet client certificates are separate. With rotateCertificates: true in the kubelet configuration, the default in kubeadm clusters, each kubelet requests a new client certificate from the API server before the old one expires.
Version Skew Policy
Kubernetes does not upgrade atomically. During an upgrade, old and new components run side by side, and the version skew policy defines which combinations are supported:
| Component | Allowed relative to kube-apiserver |
|---|---|
| kube-apiserver (HA control plane) | Instances may differ by at most one minor version during the upgrade |
| kube-controller-manager, kube-scheduler, cloud-controller-manager | Same minor as the API server or one older. Never newer. |
| kubelet, kube-proxy | Up to three minors older than the API server. Never newer. |
| kubectl | One minor newer or older than the API server |
Two rules follow from the table and shape every upgrade:
- Upgrade the control plane first, then the nodes. A kubelet must never be newer than the API server it talks to.
- Upgrade one minor version at a time. kubeadm refuses to skip from 1.34 to 1.36. To go two minors, upgrade the control plane twice. The kubelets may follow later, because they tolerate up to three minors of skew.
The kubeadm Upgrade Flow
| Phase | Where | Commands |
|---|---|---|
| 1. Prepare | Your laptop | Read the release notes of the target minor, check for removed APIs, take an etcd snapshot (see the next lesson, Backing Up and Restoring etcd) |
| 2. Upgrade kubeadm | First control plane node | Install the new kubeadm binary or package |
| 3. Plan | First control plane node | kubeadm upgrade plan checks cluster health and shows the target versions |
| 4. Apply | First control plane node | kubeadm upgrade apply v1.36.4 upgrades etcd, the API server, controller manager and scheduler one by one, then CoreDNS and kube-proxy, and renews certificates |
| 5. Other control plane nodes | Each additional control plane node | kubeadm upgrade node |
| 6. Kubelets | Every node, one at a time | kubectl drain, install the new kubelet and kubectl, systemctl daemon-reload && systemctl restart kubelet, kubectl uncordon. Worker nodes run kubeadm upgrade node first to update their kubelet configuration. |
On a real cluster the binaries come from the Kubernetes package repositories (pkgs.k8s.io), one repository per minor version. kind nodes have no package manager setup for Kubernetes, so the Step-by-Step example copies the binaries out of the newer kindest/node image instead. Every kubeadm command is the same as on a production node.
Managed Clusters
On AKS, EKS and GKE the provider upgrades the control plane, and node pool upgrades do the node work: they add surge nodes with the new version (AKS and GKE maxSurge, EKS managed node groups maxUnavailable), then cordon, drain and delete old nodes. The drains use the Eviction API, so your budgets are respected, and a budget that never allows a disruption stalls the upgrade until the provider times out or forces it, each by its own rules. The skew policy still applies. The preparation in the Step-by-Step section is exactly what makes these upgrades uneventful too.
Hands-On: Kubernetes Commands
See which nodes are cordoned
A cordoned node shows SchedulingDisabled in its status.
Preview a drain without changing anything
--dry-run=server runs every check, including PodDisruptionBudgets, and prints what would be evicted.
Drain one application only
Moves just the Pods that match the selector off the node.
Check whether a budget currently allows a disruption
ALLOWED DISRUPTIONS of 0 means every eviction will be refused.
Check certificate expiry on a kubeadm control plane
On a real node run it with sudo. On kind, docker exec into the control plane container.
Show the kubelet version of every node
Step-by-Step Example
The Scenario
A small online shop runs an inventory API. Operations has three jobs this week: patch the worker nodes, renew the control plane certificates before they expire, and upgrade the cluster from Kubernetes 1.35 to 1.36. The business has one requirement: the inventory API must keep answering during all of it.
You will:
- Build the API and a disposable cluster on Kubernetes v1.35.8.
- Drain a node under load with a naive Deployment and count the failures.
- Prepare the Deployment for maintenance and repeat the drain.
- See how drain refuses unsafe Pods and how a budget can block it.
- Simulate a node failure and shorten the time to recovery.
- Renew the certificates.
- Upgrade the whole cluster to v1.36.4 while a load generator keeps counting.
Step 1 — The Inventory API
Create the project:
Replace Program.cs. The API reports which Pod and node answered, and its readiness endpoint starts failing the moment the host begins to shut down, so Kubernetes stops routing traffic to a Pod that is about to stop.
The Dockerfile uses a chiseled runtime image, which runs as a non-root user and contains no shell:
Build it. A quick docker run -p 18080:8080 inventory-api:1.0 answers curl localhost:18080/api/stock/SKU-1 with {"sku":"SKU-1","quantity":361,"pod":"c947a5ac56e4","node":"unknown"}.
Step 2 — A Disposable Cluster on v1.35.8
The cluster starts one minor version behind, so it can be upgraded at the end. Pinning the image digest makes the setup reproducible. Save this as ops-lab.yaml:
Create it with its own kubeconfig file, so your normal kubectl context is never touched, and load the image into the nodes:
Step 3 — The Naive Deployment and a Load Generator
Start the way many Deployments start: one replica, no probes, no budget. Save as 1-naive.yaml:
The load generator sends about eight requests per second and prints the HTTP status of each. A status of 000 means no answer within one second. It runs on the control plane node, so draining the workers never moves it. Save as loadgen.yaml:
Step 4 — Drain the Naive Deployment
Remember how many lines the load generator has written, drain the node that hosts the API, and count the results of everything after that point:
Three requests got no answer. The drain itself took about two seconds, which is the problem: the only replica was gone before its replacement on the other worker could answer. On a busy production API those three seconds are hundreds of failed requests. An earlier run of the same drain on ops-lab-worker lost two requests with a two-second client timeout, so the exact number varies, but it is never zero.
Bring the node back:
Step 5 — Prepare the Deployment for Maintenance
Save as 2-ready-for-maintenance.yaml. It adds three replicas spread across nodes, readiness and liveness probes, a preStop sleep that gives the endpoint removal time to reach every node before the app stops, and a PodDisruptionBudget that keeps at least two replicas available:
whenUnsatisfiable: ScheduleAnyway is deliberate. With DoNotSchedule, a drain of one of only two workers would leave replacements Pending, because they could not satisfy the spread, and the budget would block the drain forever.
Step 6 — Drain Again, With the Budget
ops-lab-worker2 hosts two of the three replicas. Drain it under the same load:
The output tells the whole story. Drain asked for both Pods at once. The API server allowed one eviction and refused the second with the budget message. Drain retried five seconds later, by which time the replacement for the first Pod was ready, and the second eviction went through. According to time, the drain took 12.6 seconds instead of 2, and all 266 requests succeeded.
After the drain, all three replicas run on ops-lab-worker, the only schedulable worker. Uncordon the node and spread them again:
(The three old Pods were still Terminating at that moment and are left out.)
Step 7 — When Drain Refuses or Gets Stuck
Drain stops on Pods it cannot move safely. Start a bare Pod with an emptyDir volume on ops-lab-worker2, the kind of debugging Pod people forget about:
A server-side dry run shows each refusal in turn:
With --force the bare Pod would be deleted and never come back. The selector variant moves only the API and leaves everything else alone:
Delete the scratch Pod with kubectl -n shop delete pod scratch.
Now the opposite problem: a budget that never allows a disruption. Raise minAvailable to the replica count and drain the node that hosts one replica, with a timeout:
Without --timeout this drain would retry forever. Note that the node stays cordoned after the failed drain. Uncordon it, or fix the budget and drain again. The Eviction API shows the same decision as an HTTP status. This is the request drain sends for every Pod:
With minAvailable back at 2, the same request succeeds with {"kind":"Status","apiVersion":"v1","metadata":{},"status":"Success","code":201}. Restore the budget and the node:
Step 8 — A Node Fails
Look at the tolerations every Pod received by default:
Five minutes is long for an API that starts in a second. Shorten it to 30 seconds with a patch. Save as 3-tolerations-patch.yaml:
Simulate a failed node by stopping its kubelet. The kind node is a container, so docker exec reaches it. Then watch the node and the Pods every ten seconds:
Condensed from the run (local time):
The node turned NotReady after roughly 50 seconds, the replicas were replaced about 30 seconds after that, as the shortened toleration demands. The old Pods stay Terminating because no kubelet can confirm their end. The node carries both unreachable taints and explains itself:
The load generator logged 1,165 successful requests and no failures during this experiment. That is partly luck of the simulation: stopping only the kubelet left the API containers running, so they kept answering until Kubernetes moved traffic away. When a machine loses power, requests routed to its Pods fail until the node is marked NotReady and the endpoints are updated. Spare replicas on other nodes keep that window survivable.
Start the kubelet again. The node returns, the taints disappear, and the stuck Pods are cleaned up:
Step 9 — Check and Renew Certificates
Renew one certificate first and compare serial numbers on disk and on the wire:
The running API server already serves the new certificate, because it watches its serving certificate files and reloads them. That is not true for everything: the client certificates inside controller-manager.conf and scheduler.conf, and etcd's own certificates, are read at startup. So after renew all, kubeadm asks for a restart of the control plane:
Static Pods cannot be restarted with kubectl delete: deleting the mirror Pod only makes the kubelet create the mirror object again, the containers keep running. The kubelet restarts them when their manifest leaves and returns to /etc/kubernetes/manifests:
On a single control plane node this means about 20 seconds without an API server. Running workloads are not affected, but deployments and scaling pause. With three control plane nodes, renew and restart them one at a time.
admin.conf was renewed as well. Kubeconfig files that were copied from it earlier, such as the one kind wrote for you, keep working until their own copy of the certificate expires. Copy the new file again before that.
The kubelets take care of their own client certificates:
Step 10 — Plan the Upgrade to v1.36.4
The cluster runs v1.35.8. The target is v1.36.4, exactly one minor version newer. On a real node you would install the new kubeadm package from the v1.36 repository on pkgs.k8s.io. Here the binaries come from the kindest/node:v1.36.4 image, which contains exactly the files the packages would install:
To prove that the whole upgrade is invisible to clients, replace the in-cluster load generator, which lives on the control plane node that is about to be drained, with one outside the cluster. A NodePort Service makes the API reachable from a container on the kind Docker network. Save as 4-nodeport.yaml:
Install the new kubeadm on the control plane and ask it for a plan:
The plan checks cluster health, looks up the newest patch releases online, and offers two paths: the newest patch of the current minor (v1.35.9) or the newest stable release (v1.36.5). For each path it also prints a second table, left out above, with what apply will not do for you: the kubelet on every node. kubeadm can always upgrade to its own version, so the next step targets v1.36.4, the version of the binaries at hand.
Step 11 — Upgrade the Control Plane
Condensed output:
time reported 3 minutes 36 seconds, most of it spent pulling the new images. kubeadm replaced the static Pod manifests one component at a time, kept backups of the old ones under /etc/kubernetes/tmp, renewed every certificate on the way, and rolled out new CoreDNS and kube-proxy versions. The API server now reports the new version, while every kubelet still runs the old one, which the skew policy allows:
The kube-proxy DaemonSet is still rolling out at this moment. Wait for it before touching the nodes:
Step 12 — Upgrade the Kubelet on the Control Plane Node
The kubelet on the control plane node is upgraded like any other: drain, replace, restart, uncordon.
The static Pods of the control plane are not evicted by the drain; they keep running while the kubelet restarts.
Step 13 — Upgrade the Workers One at a Time
Each worker goes through the same loop. kubeadm upgrade node on a worker only updates the kubelet configuration; the binary replacement and restart do the rest.
Condensed output for the first worker:
The budget again forced the evictions into single file. When the loop finishes, every component runs v1.36.4:
Step 14 — Count the Requests
The external load generator ran from before kubeadm upgrade plan until after the last uncordon:
Every one of the 4,243 requests sent between 12:13:26 and 12:20:50 was answered, through an etcd upgrade, three API server restarts, new CoreDNS and kube-proxy versions and three drained nodes. That is the result of the preparation in Step 5, not of luck in the upgrade itself.
Step 15 — Clean Up
Summary
- Node maintenance is always cordon, drain, work, uncordon. Uncordoning does not move Pods back; use
kubectl rollout restartor the Descheduler to rebalance. kubectl drainevicts Pods through the Eviction API, which respects PodDisruptionBudgets and answers 429 when a budget allows no disruption.kubectl delete podand--disable-evictionbypass budgets.- Drain refuses DaemonSet Pods,
emptyDirPods and bare Pods until you pass--ignore-daemonsets,--delete-emptydir-dataor--force. Use--dry-run=serverto preview,--pod-selectorto scope and always--timeoutto avoid endless waits. - A budget alone does not stop errors. Replicas, topology spread with
ScheduleAnyway, readiness and graceful shutdown together turned 3 failed requests on a naive drain into 0 of 4,243 during a full upgrade. - A failed node becomes NotReady after about 50 seconds and gets the
node.kubernetes.io/unreachabletaints. Pods are evicted after theirtolerationSeconds, 300 by default; 30 seconds suits fast-starting stateless APIs. - kubeadm leaf certificates last one year.
kubeadm certs check-expirationshows them,kubeadm certs renew allrenews them, and everykubeadm upgraderenews them too. Restart the control plane static Pods afterwards by moving their manifests. - Upgrade control plane first, one minor at a time. Kubelets may lag up to three minors but must never be newer than the API server.
- The kubeadm flow is: new kubeadm,
kubeadm upgrade plan,kubeadm upgrade apply, then per node drain,kubeadm upgrade node, new kubelet, restart, uncordon. Managed clusters automate the same steps with surge nodes, and depend on the same budgets.