Lesson 67 of 72 · 24 min read · Updated
Backing Up and Restoring etcd
On this page
Every object you have ever created in a Kubernetes cluster lives in one place: etcd. Deployments, Services, ConfigMaps, Secrets, RBAC rules, CRDs and their custom resources are all keys in a small, consistent key-value store on the control plane. The worker nodes, kubelets and controllers only hold copies of what etcd says. Lose etcd, and the cluster forgets what it is supposed to run.
That makes an etcd snapshot the most basic backup of a self-managed cluster. This lesson shows what is inside etcd, takes a snapshot from a kind control plane, causes a disaster by deleting a namespace that holds a .NET 10 API with its ConfigMap and Secret, and restores the whole cluster state from the snapshot. You will also see what a snapshot does not protect, why changes made after the snapshot disappear, and how to schedule snapshots with a CronJob.
We will cover the following topics:
- What Kubernetes stores in etcd and how the keys are organised
- What an etcd snapshot contains and what it does not: no volume data, no container images
etcdctlversusetcdutl, and the certificates a client needs- Taking a snapshot from the etcd static Pod, checking it with
etcdutl snapshot status, and copying it off the node - Restoring a snapshot into a new data directory and pointing the etcd static Pod at it
- What happens after a restore: revision bumps, controllers reconciling, caches catching up
- Restore caveats for multi-member clusters and encryption at rest
- Scheduling snapshots with a CronJob on the control plane node
- When to use Velero instead
Technical Requirements
- A disposable kind cluster created with the kind CLI, not Docker Desktop's built-in cluster used in most other lessons of this series. Restoring etcd rewinds the entire cluster to the moment of the snapshot and requires stopping the API server, which would disrupt everything else you run on your everyday cluster. Docker Desktop also manages its control plane itself. The examples use kind v0.33.0 and
kindest/node:v1.36.4with one control-plane node and one worker. The lesson Setting Up a Local Multi-Node Cluster with kind and Docker Desktop explains the kind CLI. - Docker,
kubectl(v1.36.1 in the examples) and the .NET 10 SDK to build the sample API. - No local etcd tools are needed. The etcd container image that kubeadm already runs on the control plane contains both
etcdctlandetcdutlin version 3.6.8. - The lesson Cluster Maintenance — Cordon, Drain, Certificates and Upgrades explains static Pods and kubeadm certificates, which this lesson uses.
Core Concepts
What Lives in etcd
The API server is the only component that talks to etcd. Every object it accepts is stored under a key that follows the pattern /registry/<resource>/<namespace>/<name>:
| Key | Object |
|---|---|
/registry/deployments/pricing/pricing-api | The Deployment pricing-api in namespace pricing |
/registry/configmaps/pricing/pricing-config | A ConfigMap |
/registry/secrets/pricing/partner-credentials | A Secret |
/registry/pods/pricing/pricing-api-777d9fc7b7-hz7xs | A Pod, including its status and the node it is bound to |
/registry/namespaces/pricing | A cluster-scoped object has no namespace segment |
/registry/apiextensions.k8s.io/customresourcedefinitions/... | CRDs, and below other prefixes their custom resources |
Values are stored as Kubernetes protobuf with a k8s magic prefix, not as JSON. Secrets are not encrypted unless you configure encryption at rest on the API server. Anyone who can read etcd or a snapshot file can read every Secret in the cluster. The Step-by-Step section proves that in one command.
What a Snapshot Contains, and What It Does Not
An etcd snapshot is a consistent copy of the whole key space at one revision. It is all or nothing: you restore the entire cluster state, not a single namespace.
| In the snapshot | Not in the snapshot |
|---|---|
| Every API object: workloads, Services, ConfigMaps, Secrets, RBAC, CRDs and custom resources | Data inside PersistentVolumes. A PVC object comes back, but the database files on the disk behind it do not. |
| PersistentVolume and PersistentVolumeClaim objects | Container images. Registries must still have them. |
| Leases, Events (only for their one-hour lifetime), node objects | Certificates and keys in /etc/kubernetes/pki, kubeconfig files and static Pod manifests on the control plane nodes |
| Status fields, such as which node a Pod was bound to | The encryption-at-rest configuration and its keys |
| Anything outside the cluster: DNS records, cloud load balancers, external databases |
So a complete recovery plan for a self-managed control plane needs two things next to the snapshot: a copy of /etc/kubernetes/pki (at least the CA keys), and whatever protects the volume data, such as storage snapshots or Velero.
etcdctl and etcdutl
etcd 3.6 ships two command line tools with different jobs:
| Tool | Talks to | Used for |
|---|---|---|
etcdctl | A running etcd over the network (gRPC on port 2379) | snapshot save, endpoint status, member list, get, put, defragmentation |
etcdutl | Files on disk, no server needed | snapshot status, snapshot restore, offline defragmentation, migrations |
etcdctl snapshot status and etcdctl snapshot restore were deprecated in etcd 3.5 and removed in 3.6: in the 3.6.8 image, etcdctl snapshot has a single subcommand, save. Inspecting and restoring snapshots is always an etcdutl job now.
Certificates for etcd Clients
In a kubeadm cluster etcd accepts only clients with a certificate signed by the etcd CA. All the files are in /etc/kubernetes/pki/etcd on the control plane:
| File | Purpose |
|---|---|
ca.crt | The etcd CA. Clients use it to verify the server (--cacert). |
server.crt / server.key | etcd's serving certificate. It is also a valid client certificate, so it works for quick manual commands. |
healthcheck-client.crt / healthcheck-client.key | A client-only certificate kubeadm creates for health checks. A good choice for backup jobs. |
peer.crt / peer.key | Used between etcd members. Not for clients. |
Every etcdctl call therefore needs --endpoints, --cacert, --cert and --key. The Hands-On section wraps them in a shell function.
The Restore Procedure
A restore does not load data into a running etcd. It builds a new data directory from the snapshot file and starts etcd on it as if it were a new cluster that happens to contain all the old data:
| Step | Why |
|---|---|
| 1. Stop the API server and etcd | Nothing may write to the old data while you switch. On kubeadm, moving the static Pod manifests out of /etc/kubernetes/manifests stops them. |
2. etcdutl snapshot restore into a new directory | Creates a fresh member with a new cluster ID. The old data directory stays untouched as a fallback. |
| 3. Point etcd at the new directory | Change the hostPath of the etcd-data volume in etcd.yaml, or swap the directories. |
| 4. Start the control plane again | Move the manifests back. The kubelet recreates the static Pods. |
| 5. Restart components that cache state | kubelets, kube-controller-manager, kube-scheduler and other controllers such as CoreDNS hold watches and caches built before the restore. |
Revisions, Watches and Why to Bump Them
etcd numbers every change with a global revision. Kubernetes exposes it as metadata.resourceVersion, and every controller, kubelet and informer remembers the last revision it has seen so it can resume a watch from there.
A restore rewinds the revision to the snapshot. Clients that saw newer revisions before the disaster would then wait for revisions that already happened in their eyes, and miss changes. etcdutl solves this with two flags that the Kubernetes documentation recommends for every restore:
--bump-revision 1000000000adds a large number to the revision, so the restored store is ahead of anything a client has seen.--mark-compactedmarks all old revisions as compacted, so clients that try to resume an old watch receive an error and do a fresh list instead.
Multi-Member Clusters and Encryption at Rest
- Multi-member etcd. Production control planes run three or five etcd members. A snapshot is taken from one member, but the restore must recreate every member from the same snapshot file, each with its own
--nameand--initial-advertise-peer-urlsand the full--initial-clusterlist. Restoring only one member and letting it rejoin the old ones does not work, because the restored cluster has a new cluster ID. Stop all API servers first and start them only when all members are back. - Encryption at rest. If the API server encrypts Secrets with an
EncryptionConfiguration, the snapshot holds only ciphertext. Restoring it is useless without the same configuration file and keys. Back them up separately and never next to the snapshot, or the encryption gains you nothing. - Version. Restore with an
etcdutlof the same minor version as the etcd that will run on the data, and restore into a cluster of the same Kubernetes minor version.
When etcd Snapshots Are the Wrong Tool
| Need | Better tool |
|---|---|
| Recover the whole control plane after losing etcd | etcd snapshot |
| Restore one namespace or one application, leaving the rest of the cluster as it is | Velero |
| Restore volume data along with the objects | Velero with volume snapshots or file-system backup, or the storage system's own snapshots |
| Move workloads to a different cluster or another Kubernetes version | Velero, or GitOps from the lesson GitOps with Argo CD |
| A managed cluster (AKS, EKS, GKE) | Velero. You have no access to etcd there; the provider protects the control plane. |
The next lesson, Backup and Disaster Recovery with Velero, covers the application-level side.
Hands-On: Kubernetes Commands
A shell function for etcdctl
Runs etcdctl inside the etcd static Pod with the right certificates. Works in bash and zsh.
Check etcd health, size and revision
List the keys of one namespace
Take a snapshot
The file lands on the control plane node, inside the directory mounted into the etcd Pod.
Inspect a snapshot file with etcdutl
ctr on the node starts the etcd image already present in containerd, so this works even when the API server is down.
Find the etcd flags a restore must match
Check whether Secrets are encrypted at rest
A count of 0 means no encryption provider is configured.
Step-by-Step Example
The Scenario
A pricing team runs a .NET 10 API that reads its currency and discount from a ConfigMap and a partner API key from a Secret. You will take an etcd snapshot, change the configuration, and then an operator deletes the wrong namespace. The task is to bring the API, its ConfigMap and its Secret back from the snapshot, and to understand exactly which state you get back.
Step 1 — The Pricing API
Replace Program.cs. ASP.NET Core maps environment variables such as Pricing__Currency to the configuration key Pricing:Currency, so the API needs no Kubernetes-specific code:
The list price is derived from the characters of the SKU, so every replica returns the same price for the same SKU. The Dockerfile:
Step 2 — A Disposable Cluster
Save as etcd-lab.yaml:
Look at how kubeadm runs etcd. It is a static Pod with its data in /var/lib/etcd on the node:
Step 3 — Deploy the API With Its Configuration
Save as pricing.yaml:
Step 4 — Look Inside etcd
Define the etcdctl function from the Hands-On section, then ask etcd about itself:
Condensed:
One member, 1.5 MB. Count the keys and list those of the pricing namespace, leaving out the Events:
Now read the Secret straight from etcd, bypassing the API server and its RBAC:
The value starts with the k8s protobuf prefix, and the API key is there in clear text, twice: in the data and in the last-applied-configuration annotation that kubectl apply writes. This cluster has no encryption at rest:
Every snapshot of this etcd is therefore as sensitive as all the Secrets in the cluster together. Store snapshots encrypted and with tight access control.
Step 5 — Take a Snapshot
The etcd Pod can write only to the directories mounted into it, so the file lands in the data directory first. Move it to a backup directory on the node and copy it off the node. On a real cluster that last step is scp or an upload to object storage; with kind it is docker cp:
A snapshot that exists only on the node it protects is not a backup: when the control plane node dies, the snapshot dies with it.
Check the file with etcdutl. It runs from the same etcd image, started directly in containerd with ctr, so it does not depend on the API server:
The snapshot is consistent at revision 670 and contains the same 417 keys you counted.
Step 6 — Life Goes On After the Snapshot
The team ships a new configuration and rolls the API:
Another team creates a namespace of its own:
Step 7 — The Disaster
Someone cleans up the wrong namespace:
The Deployment, both Pods, the Service, the ConfigMap and the Secret are gone, and with the Secret the partner API key, which nobody has written down anywhere else.
Step 8 — Stop the Control Plane
Move all static Pod manifests out of the directory the kubelet watches. The kubelet stops the containers within a few seconds:
No control plane container is running, and the API is down. The workloads on the worker keep running as they were; only the API is unavailable.
Step 9 — Restore Into a New Data Directory
A restore needs the member name and peer URL the etcd static Pod uses. Read them from its manifest:
Run etcdutl snapshot restore from the etcd image, with /var/lib of the node mounted so it can create the new directory next to the old one:
The interesting lines of the output:
The old data directory is still there, untouched, in case the restore has to be rolled back:
On a node with its own package manager you would install etcdutl from the etcd release archive of the same version instead of using ctr. The command line is identical.
Step 10 — Point etcd at the Restored Data and Start the Control Plane
The etcd container always uses --data-dir=/var/lib/etcd. What changes is the host directory mounted there, the etcd-data volume:
Move the manifests back and wait for the API:
The API server answered four seconds after the manifests were back.
Step 11 — Restart the Components That Cache State
The kubelets, controllers and CoreDNS all hold watches and caches built before the restore. Start with the kubelets:
The controller manager and scheduler were already restarted in Steps 8 and 10, together with the API server, because they are static Pods too. CoreDNS follows in the next step.
Step 12 — Verify What Came Back
Everything is back with its original names, the original Service ClusterIP and even the original Pod names. The Pod objects came back from etcd still bound to etcd-lab-worker, whose containers had been removed when the namespace was deleted, so the kubelet started them again; that is the RESTARTS 1. The events show it:
Now look at which state came back:
The configuration is v1, the state at the snapshot. The v2 change from Step 6, the rollout it triggered, and the whole reports namespace of the other team are gone. A restore rewinds the entire cluster, not just the namespace you lost. That is the price of etcd snapshots, and the reason to take them often.
The API answers with the restored configuration:
This answer did not come immediately. During the first minute or so after the API came back, the same command failed, and the Pod logged curl: (6) Could not resolve host: pricing-api.pricing. A little later, name resolution and the request worked without any further intervention. Cluster DNS, like every component with a cache, needs time to catch up after a restore. Rather than waiting for that, make a restart of CoreDNS and of every other controller in the cluster, such as ingress controllers, cert-manager or operators, part of the runbook, so none of them acts on a stale cache:
Then verify the application end to end before you declare the incident over.
Finally, the revision. New writes continue from the bumped number, so no client can mistake the restored store for an older one:
Step 13 — Schedule Snapshots With a CronJob
A manual snapshot protects one moment. A CronJob on the control plane node takes one every six hours and keeps the newest seven. The etcd image has no shell, so the work is split: an init container runs etcdctl snapshot save, and a small busybox container renames the file with a timestamp and deletes old ones. Save as etcd-backup-cronjob.yaml:
hostNetwork: true lets the job reach etcd on 127.0.0.1:2379, where kubeadm's etcd listens, and the healthcheck-client certificate keeps the job away from etcd's own server key. Run it twice by hand instead of waiting six hours:
The snapshots still sit on the control plane node they are meant to protect. In production, add a step that uploads each file to object storage in another failure domain, encrypted, or run the job from a backup tool that does this for you. With several control plane nodes, the job needs to run on only one of them; a snapshot from any healthy member contains the whole cluster.
Step 14 — Clean Up
The local copy in backups/ contains every Secret of the cluster in clear text. Delete it too.
Summary
- etcd holds every API object under
/registry/<resource>/<namespace>/<name>. Without encryption at rest, Secrets are stored in clear text, in etcd and in every snapshot. - A snapshot is a consistent copy of the whole cluster state at one revision. It contains no PersistentVolume data, no container images, no certificates from
/etc/kubernetes/pkiand no encryption keys. Back those up separately. etcdctltalks to a running etcd (snapshot save,endpoint status);etcdutlworks on files (snapshot status,snapshot restore). Both are in the etcd image kubeadm already runs.- Copy snapshots off the node. A snapshot that dies with the control plane is not a backup.
- To restore: stop the static Pods by moving their manifests,
etcdutl snapshot restoreinto a new data directory with the member's--name,--initial-clusterand--initial-advertise-peer-urls, point theetcd-datahostPath at it, start the control plane, restart kubelets and controllers. - Always use
--bump-revisionand--mark-compacted, so watches restart cleanly and resource versions keep increasing. - A restore rewinds everything: the v2 configuration and the unrelated
reportsnamespace created after the snapshot were lost. Expect a short period of stale caches, such as DNS, and verify end to end. - Multi-member clusters must restore every member from the same snapshot. Encrypted clusters need the same encryption configuration and keys.
- Schedule snapshots with a CronJob on the control plane, keep several, ship them off-site. For single namespaces, volume data, other clusters or managed Kubernetes, use Velero, covered in the next lesson, Backup and Disaster Recovery with Velero.