Lesson 68 of 72 · 25 min read · Updated

Backup and Disaster Recovery with Velero

The previous lesson showed that an etcd snapshot is a full copy of the cluster's API state, and that it is the right tool when the control plane itself is damaged. For most of an application team's bad days it is the wrong tool. You cannot restore one namespace from it, it holds no data from your PersistentVolumes, and it only fits back into the cluster it came from. When someone runs kubectl delete namespace against the wrong context, or a cluster has to be rebuilt, you need something else.

That something is Velero. It backs up Kubernetes objects and the data in their volumes to object storage outside the cluster, and restores them into the same cluster or into a new one. In this lesson you protect a .NET 10 API and its Postgres database with Velero on kind, break it twice, and measure what comes back. The headline result: after the entire cluster was deleted, a new cluster served all four bookings again about three minutes after kind create cluster started.

We will cover the following topics:

  1. Disaster recovery vocabulary: RPO, RTO, and why replication is not a backup
  2. What an etcd snapshot misses, and where Velero fits
  3. Velero's architecture: server, plugins, BackupStorageLocation, VolumeSnapshotLocation and the node-agent
  4. File-system backup with Kopia versus CSI snapshots
  5. The Backup, Restore and Schedule resources, filters and TTL
  6. Backup hooks for a consistent database backup
  7. Restore ordering, skipped resources, the existing-resources policy and namespace mappings
  8. Restoring into a new cluster, and rehearsing restores

Technical Requirements

  1. Docker Desktop (macOS or Windows) or Docker Engine (Linux), with about 2 CPUs and 4 GB of memory free.
  2. The kind CLI v0.33.0 and the node image kindest/node:v1.36.4, the same Kubernetes version as the rest of this series.
  3. The Velero CLI v1.18.4 (brew install velero on macOS), plus kubectl, curl, jq and the .NET 10 SDK.
  4. Familiarity with "Kubernetes Persistent Volumes, PVCs, and Storage" and "Backing Up and Restoring etcd".

Most lessons in this series use Docker Desktop's built-in four-node cluster. This one does not. You will delete a namespace, then a whole cluster, and you need a second cluster to restore into. Velero also installs cluster-wide CRDs and a privileged DaemonSet that reads every Pod's volumes on the node. None of that belongs on the cluster that hosts the rest of your work. So you create two disposable clusters with the kind CLI, dr-a and dr-b, each with one control-plane node and one worker, and each with its own kubeconfig file. Your ~/.kube/config and its current context are never touched.

Velero needs S3-compatible object storage. Older guides use MinIO, but MinIO no longer publishes container images, so docker pull minio/minio fails. This lesson uses RustFS 1.0.3, an Apache-2.0 S3-compatible server. Any S3-compatible store works with the same configuration.

Core Concepts

Disaster Recovery Vocabulary

TermQuestionSet by
RPO, Recovery Point ObjectiveHow much data can we lose?Backup frequency. Nightly backups mean up to 24 hours of lost writes.
RTO, Recovery Time ObjectiveHow long can we be down?How fast you rebuild and restore. Measure it, do not guess it.

A backup is a point-in-time copy kept somewhere else. Replication, such as a Postgres replica or a multi-zone cluster, keeps you running when hardware fails, but it also copies a DROP TABLE within milliseconds. Only a backup lets you go back in time. A common rule is 3-2-1: three copies, two kinds of media, one off-site. For Kubernetes, off-site starts with "not inside the cluster being protected".

What an etcd Snapshot Does Not Cover

etcd snapshotVelero
Volume dataNo, only the PV and PVC objectsYes, file-system backup or CSI snapshots
GranularityAll or nothingNamespace, resource type, label selector
Restore targetSame cluster, by replacing etcdSame cluster, another namespace, a new cluster
Managed clusters (AKS, EKS, GKE)No access to etcdWorks everywhere

On self-managed clusters you want both. On managed clusters, Velero or a product built on it is usually all you have.

Velero's Architecture

You never call Velero directly. The velero CLI creates custom resources, and the server reconciles them.

ComponentJob
Velero server (Deployment in namespace velero)Reconciles Backups, Restores and Schedules, and reads and writes objects through the API
Object store plugin (init container)Talks to storage. velero-plugin-for-aws covers AWS S3 and every S3-compatible store.
BackupStorageLocation (BSL)Bucket, prefix, endpoint, credentials and access mode for backups
VolumeSnapshotLocation (VSL)Where native cloud disk snapshots go. Not used here.
node-agent (DaemonSet)Copies volume data from the node with Kopia, and back again on restore
BackupRepositoryAn encrypted, deduplicating Kopia repository per namespace inside the bucket

A backup writes a tarball of JSON manifests plus metadata under backups/<name>/, and volume data into the Kopia repository under kopia/<namespace>/. The repository is encrypted with the password in the Secret velero-repo-credentials.

File-System Backup and CSI Snapshots

CSI snapshotsFile-system backup (node-agent + Kopia)
HowThe CSI driver takes a VolumeSnapshot. The optional data mover copies it to object storage.Reads the mounted volume's files on the node and uploads them
NeedsA CSI driver with snapshot supportA running Pod that mounts the volume
Portable to another cloudOnly with the data moverAlways
Volume typesWhatever the driver provisionsMost types, including local and NFS. Not hostPath.

The last row matters on local clusters. kind's default StorageClass standard (local-path provisioner) creates hostPath volumes. Velero skips them with a warning, and the backup still says Completed. You will hit this in Step 6. File-system backup is opt-in per volume, either through the Pod annotation backup.velero.io/backup-volumes: <volume> or for every volume with --default-volumes-to-fs-backup.

Backups, Restores and Schedules

velero backup create, velero restore create and velero schedule create each create one custom resource of the same name. A Schedule is a cron expression plus a Backup template. While a backup or restore runs, the server creates one PodVolumeBackup or PodVolumeRestore per volume as a work item for the node-agent.

Filters select what goes in. --include-namespaces / --exclude-namespaces take namespaces, --include-resources / --exclude-resources take resource types (for example events,events.events.k8s.io), and --selector takes a label selector that matches the labels on each object. --ttl sets how long a backup is kept, with a default of 720h (30 days). Cluster-scoped objects such as PVs are added automatically when a namespaced object needs them.

Backup Hooks

Copying a running database's files gives a crash-consistent copy, like pulling the power plug. Hooks make it application-consistent by running commands inside a container before or after its volumes are copied:

AnnotationMeaning
pre.hook.backup.velero.io/containerContainer to run in
pre.hook.backup.velero.io/commandCommand as a JSON array
pre.hook.backup.velero.io/timeoutWait limit. The default is 30 seconds.
pre.hook.backup.velero.io/on-errorFail (the default) or Continue

post.hook.backup.velero.io/... works the same way after the copy. The hook in this lesson runs pg_dump into the data volume, which gives a portable logical dump, and then CHECKPOINT, which flushes dirty pages to disk.

How a Restore Works

Velero recreates objects in priority order. The default from velero server --help starts with customresourcedefinitions, namespaces, storageclasses, ..., persistentvolumes, persistentvolumeclaims, ..., serviceaccounts, ..., secrets, configmaps, limitranges, priorityclasses, pods, replicasets.apps, ..., endpoints, services. Unlisted types come alphabetically in between. nodes, events, csinodes, volumeattachments and Velero's own objects are never restored.

An object that already exists is skipped with a warning. With --existing-resource-policy update, Velero patches it instead, which fails for immutable fields. For file-system volumes, Velero recreates the PVC, lets the provisioner create a fresh PV, and injects an init container named restore-wait so the application starts only after the node-agent has written the data.

--namespace-mappings old:new restores under a new name, which is the safe way to test next to production. A ConfigMap labeled velero.io/change-storage-class: RestoreItemAction in the velero namespace maps StorageClass names between clusters.

Restoring Into a New Cluster

The new cluster needs:

  1. Velero with a BSL pointing at the same bucket and prefix. The backup-sync controller then creates Backup objects for what it finds.
  2. The same Kopia repository password. velero install uses the default, so if you changed it, recreate velero-repo-credentials first.
  3. StorageClasses with the same names, or a mapping.
  4. Access to the images, and the CRDs and operators the app depends on.

Set the BSL to ReadOnly during recovery, so a half-built cluster cannot create or garbage-collect backups. Schedules are not stored in the bucket, so recreate them from Git when the new cluster takes over. Finally, rehearse: a backup that has never been restored is only a hope.

Hands-On: Kubernetes Commands

These commands come from the step-by-step run. KUBECONFIG points at the cluster in use.

Check Velero and its storage location

PHASE must be Available.

velero version
velero backup-location get

Back up a namespace and check the volume data

Completed only means the run finished. Look at WARNINGS and at Pod Volume Backups.

velero backup create hotel-backup-1 --include-namespaces hotel --ttl 72h --wait
velero backup describe hotel-backup-1 --details
kubectl -n velero get podvolumebackups

Read hook output and download a backup

velero backup logs hotel-backup-1 | grep -E 'running exec hook|stdout: CHECKPOINT'
velero backup download hotel-backup-1 -o hotel-backup-1.tar.gz

Restore into another namespace, or only some resource types

velero restore create hotel-restore-test --from-backup hotel-backup-1 --namespace-mappings hotel:hotel-restore-test --wait
velero restore create hotel-services --from-backup hotel-nightly-20261021122254 --include-resources services --wait

Back up every volume without annotations

velero backup create hotel-fs-default --include-namespaces hotel --default-volumes-to-fs-backup --wait

Back up by label, and see what you miss

velero backup create hotel-db-only --include-namespaces hotel --selector app=postgres --wait
velero backup describe hotel-db-only --details

The resource list in this run contained only the ReplicaSet, the Namespace, the PV, the PVC and the Pod. The Deployment, the Service and the Secret were missing. Only the Pod template carries app: postgres. The ReplicaSet inherits that label, and the PVC and PV come in through the Pod. If you back up by label, label every object.

Schedules and deletion

velero schedule create hotel-nightly --schedule="0 2 * * *" --include-namespaces hotel --exclude-resources events,events.events.k8s.io --ttl 168h
velero backup create --from-schedule hotel-nightly --wait
velero schedule pause hotel-nightly
velero schedule unpause hotel-nightly
velero backup delete hotel-first --confirm

Make the storage location read-only

velero backup-location set has no access-mode flag, so patch the resource:

kubectl -n velero patch backupstoragelocation default --type merge -p '{"spec":{"accessMode":"ReadOnly"}}'

Step-by-Step Example

The Scenario

A hotel runs a .NET 10 bookings API with Postgres 17 in the namespace hotel. You will run an object store outside every cluster, deploy the app on cluster dr-a, install Velero, back up and restore after deleting the namespace, rehearse a restore under another name, schedule backups, then delete dr-a completely and restore onto a new cluster dr-b.

Pinned versions: Velero v1.18.4, velero-plugin-for-aws v1.14.4, rustfs/rustfs 1.0.3, amazon/aws-cli 2.38.3, postgres 17.11, kindest/node v1.36.4, Npgsql 10.0.3.

Step 1 — An Object Store Outside the Cluster

Run the object store as a plain container on the kind Docker network, the bridge network kind attaches its nodes to. Pods resolve it by container name, and your machine reaches it through the published port. kind creates this network with its first cluster, so if docker network inspect kind fails, do Step 2 first.

docker run -d --name velero-s3 --network kind -p 9000:9000 -p 9001:9001 \
-e RUSTFS_ACCESS_KEY=velero -e RUSTFS_SECRET_KEY=velero-secret-123 \
-e RUSTFS_CONSOLE_ENABLE=true \
-v velero-s3-data:/data \
rustfs/rustfs:1.0.3 /data
docker ps --filter name=velero-s3 --format 'table {{.Names}}\t{{.Image}}\t{{.Status}}'
NAMES IMAGE STATUS
velero-s3 rustfs/rustfs:1.0.3 Up 5 seconds

Create the bucket with the AWS CLI in a container on the same network:

s3() {
docker run --rm --network kind \
-e AWS_ACCESS_KEY_ID=velero -e AWS_SECRET_ACCESS_KEY=velero-secret-123 \
-e AWS_DEFAULT_REGION=us-east-1 \
amazon/aws-cli:2.38.3 --endpoint-url http://velero-s3:9000 "$@"
}
s3 s3 mb s3://velero-backups
make_bucket: velero-backups

Step 2 — Create the First Cluster

Save kind-dr.yaml. The node image is pinned, because kind v0.33.0 would otherwise pick its newer default.

kind: Cluster
apiVersion: kind.x-k8s.io/v1alpha4
nodes:
- role: control-plane
image: kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed
- role: worker
image: kindest/node:v1.36.4@sha256:099e049362a1526b2db71494e1947aae99bd16290d7c895f2b7ea312e3cbfaed
kind create cluster --name dr-a --config kind-dr.yaml --kubeconfig $PWD/kubeconfig-dr-a
export KUBECONFIG=$PWD/kubeconfig-dr-a
kubectl get nodes
NAME STATUS ROLES AGE VERSION
dr-a-control-plane Ready control-plane 26s v1.36.4
dr-a-worker Ready <none> 12s v1.36.4

Step 3 — The .NET Bookings API

dotnet new web -n BookingsApi -o bookings-api --framework net10.0
cd bookings-api
dotnet add package Npgsql

Replace Program.cs. The API creates its table on startup and retries while Postgres starts. The PostgresException filter is there because two replicas raced on the first deployment, and one failed with duplicate key value violates unique constraint "pg_class_relname_nsp_index". /info reports the Pod, its node and the booking count, so you can tell which cluster is answering.

using Npgsql;

var builder = WebApplication.CreateBuilder(args);

var connectionString = builder.Configuration.GetConnectionString("Bookings")
?? throw new InvalidOperationException("ConnectionStrings__Bookings is not set");
builder.Services.AddSingleton(NpgsqlDataSource.Create(connectionString));

var app = builder.Build();

// Create the table on startup. Retry while Postgres is still starting.
var db = app.Services.GetRequiredService<NpgsqlDataSource>();
for (var attempt = 1; ; attempt++)
{
try
{
await using var cmd = db.CreateCommand("""
CREATE TABLE IF NOT EXISTS bookings (
id serial PRIMARY KEY,
guest text NOT NULL,
room int NOT NULL,
created_at timestamptz NOT NULL DEFAULT now())
""");
await cmd.ExecuteNonQueryAsync();
break;
}
catch (PostgresException ex) when (ex.SqlState is "23505" or "42P07")
{
// Another replica created the table at the same moment.
break;
}
catch (NpgsqlException) when (attempt < 30)
{
app.Logger.LogWarning("Database not ready (attempt {Attempt}), retrying in 2s", attempt);
await Task.Delay(2000);
}
}

app.MapGet("/bookings", async (NpgsqlDataSource db) =>
{
var list = new List<Booking>();
await using var cmd = db.CreateCommand("SELECT id, guest, room, created_at FROM bookings ORDER BY id");
await using var reader = await cmd.ExecuteReaderAsync();
while (await reader.ReadAsync())
list.Add(new Booking(reader.GetInt32(0), reader.GetString(1), reader.GetInt32(2), reader.GetDateTime(3)));
return list;
});

app.MapPost("/bookings", async (NewBooking input, NpgsqlDataSource db) =>
{
await using var cmd = db.CreateCommand(
"INSERT INTO bookings (guest, room) VALUES ($1, $2) RETURNING id, created_at");
cmd.Parameters.AddWithValue(input.Guest);
cmd.Parameters.AddWithValue(input.Room);
await using var reader = await cmd.ExecuteReaderAsync();
await reader.ReadAsync();
var booking = new Booking(reader.GetInt32(0), input.Guest, input.Room, reader.GetDateTime(1));
return Results.Created($"/bookings/{booking.Id}", booking);
});

app.MapGet("/info", async (NpgsqlDataSource db) =>
{
await using var cmd = db.CreateCommand("SELECT count(*) FROM bookings");
var count = (long)(await cmd.ExecuteScalarAsync())!;
return new
{
pod = Environment.MachineName,
node = Environment.GetEnvironmentVariable("NODE_NAME") ?? "unknown",
bookings = count
};
});

app.MapGet("/healthz", () => Results.Ok("ok"));

app.Run();

record Booking(int Id, string Guest, int Room, DateTime CreatedAt);
record NewBooking(string Guest, int Room);

The Dockerfile uses the chiseled ASP.NET Core runtime image. Add a .dockerignore with bin/ and obj/.

FROM mcr.microsoft.com/dotnet/sdk:10.0 AS build
WORKDIR /src
COPY BookingsApi.csproj .
RUN dotnet restore
COPY . .
RUN dotnet publish -c Release -o /app --no-restore

FROM mcr.microsoft.com/dotnet/aspnet:10.0-noble-chiseled
WORKDIR /app
COPY --from=build /app .
ENV ASPNETCORE_HTTP_PORTS=8080
EXPOSE 8080
ENTRYPOINT ["dotnet", "BookingsApi.dll"]

kind nodes run their own containerd, so load the image into them:

docker build -t bookings-api:1.0 .
kind load docker-image bookings-api:1.0 --name dr-a

Step 4 — Deploy the API and Postgres

Save hotel.yaml. The Postgres Pod template's annotations opt the data volume into file-system backup and define the pre-backup hook. For now the PVC uses kind's default StorageClass standard.

apiVersion: v1
kind: Namespace
metadata:
name: hotel
---
apiVersion: v1
kind: Secret
metadata:
name: postgres-credentials
namespace: hotel
type: Opaque
stringData:
POSTGRES_USER: hotel
POSTGRES_PASSWORD: hotel-pass-123
POSTGRES_DB: bookings
ConnectionStrings__Bookings: Host=postgres;Database=bookings;Username=hotel;Password=hotel-pass-123
---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: postgres-data
namespace: hotel
spec:
storageClassName: standard
accessModes: ["ReadWriteOnce"]
resources:
requests:
storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: postgres
namespace: hotel
spec:
replicas: 1
strategy:
type: Recreate
selector:
matchLabels:
app: postgres
template:
metadata:
labels:
app: postgres
annotations:
# Velero: copy this volume with the node-agent (file-system backup)
backup.velero.io/backup-volumes: data
# Velero: dump the database and flush it to disk before the copy starts
pre.hook.backup.velero.io/container: postgres
pre.hook.backup.velero.io/timeout: 60s
pre.hook.backup.velero.io/command: >-
["/bin/bash", "-c",
"pg_dump -U hotel -d bookings -f /var/lib/postgresql/data/bookings.sql && psql -U hotel -d bookings -c CHECKPOINT"]
spec:
containers:
- name: postgres
image: postgres:17.11
envFrom:
- secretRef:
name: postgres-credentials
env:
- name: PGDATA
value: /var/lib/postgresql/data/pgdata
ports:
- containerPort: 5432
readinessProbe:
exec:
command: ["pg_isready", "-U", "hotel", "-d", "bookings"]
periodSeconds: 5
volumeMounts:
- name: data
mountPath: /var/lib/postgresql/data
volumes:
- name: data
persistentVolumeClaim:
claimName: postgres-data
---
apiVersion: v1
kind: Service
metadata:
name: postgres
namespace: hotel
spec:
selector:
app: postgres
ports:
- port: 5432
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: bookings-api
namespace: hotel
spec:
replicas: 2
selector:
matchLabels:
app: bookings-api
template:
metadata:
labels:
app: bookings-api
spec:
containers:
- name: api
image: bookings-api:1.0
imagePullPolicy: IfNotPresent
ports:
- containerPort: 8080
env:
- name: ConnectionStrings__Bookings
valueFrom:
secretKeyRef:
name: postgres-credentials
key: ConnectionStrings__Bookings
- name: NODE_NAME
valueFrom:
fieldRef:
fieldPath: spec.nodeName
readinessProbe:
httpGet:
path: /healthz
port: 8080
---
apiVersion: v1
kind: Service
metadata:
name: bookings-api
namespace: hotel
spec:
selector:
app: bookings-api
ports:
- port: 80
targetPort: 8080

Apply it and write three bookings:

kubectl apply -f hotel.yaml
kubectl -n hotel rollout status deploy/bookings-api
kubectl -n hotel port-forward svc/bookings-api 8080:80 &
for g in "Ada Lovelace:101" "Grace Hopper:102" "Linus Torvalds:203"; do
curl -s -X POST localhost:8080/bookings -H 'Content-Type: application/json' \
-d "{\"guest\":\"${g%%:*}\",\"room\":${g##*:}}"; echo
done
curl -s localhost:8080/info
{"id":1,"guest":"Ada Lovelace","room":101,"createdAt":"2026-10-21T12:17:43.485237Z"}
{"id":2,"guest":"Grace Hopper","room":102,"createdAt":"2026-10-21T12:17:43.514081Z"}
{"id":3,"guest":"Linus Torvalds","room":203,"createdAt":"2026-10-21T12:17:43.53482Z"}
{"pod":"bookings-api-5b56d87769-jz48b","node":"dr-a-worker","bookings":3}

Step 5 — Install Velero

Save the S3 credentials as credentials-velero:

[default]
aws_access_key_id = velero
aws_secret_access_key = velero-secret-123
velero install \
--provider aws \
--image velero/velero:v1.18.4 \
--plugins velero/velero-plugin-for-aws:v1.14.4 \
--bucket velero-backups \
--secret-file ./credentials-velero \
--backup-location-config region=us-east-1,s3ForcePathStyle="true",s3Url=http://velero-s3:9000,publicUrl=http://localhost:9000 \
--use-volume-snapshots=false \
--use-node-agent \
--wait
CustomResourceDefinition/backuprepositories.velero.io: created
...
DaemonSet/node-agent: created
Waiting for Velero deployment to be ready.
Waiting for node-agent daemonset to be ready.
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
SettingWhy
s3Url=http://velero-s3:9000The address the server uses. Pods resolve the container name through the kind network's DNS.
publicUrl=http://localhost:9000The address the CLI on your machine uses for logs, describe and download. In a first attempt without it, velero backup describe failed with lookup velero-s3: no such host.
s3ForcePathStyle="true"http://host/bucket instead of http://bucket.host, which self-hosted stores need
--use-volume-snapshots=falseNo VolumeSnapshotLocation, because there is no cloud disk API
--use-node-agentDeploys the node-agent DaemonSet
velero backup-location get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws velero-backups Available 2026-10-21 15:18:45 +0300 +03 ReadWrite true

The install created 13 CRDs (kubectl get crd -o name | grep velero.io), including backups, restores, schedules, backupstoragelocations, podvolumebackups and backuprepositories.

Step 6 — The First Backup, and a Silent Gap

velero backup create hotel-first --include-namespaces hotel --wait
velero backup get
Backup completed with status: Completed. You may check for more information using the commands `velero backup describe hotel-first` and `velero backup logs hotel-first`.
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION QUEUE POSITION SELECTOR
hotel-first Completed 0 1 2026-10-21 15:19:16 +0300 +03 29d default <none>

Completed in under a second is suspiciously fast for a database:

velero backup describe hotel-first --details
Warnings:
Namespaces:
hotel: resource: /pods name: /postgres-544f4454b-rz2lv message: /Volume data in pod hotel/postgres-544f4454b-rz2lv is a hostPath volume which is not supported for pod volume backup, skipping
...
Items backed up: 44
...
Pod Volume Backups: <none included>

HooksAttempted: 1
HooksFailed: 0

The hook ran and 44 objects were saved, but not one byte of the database. Restoring this would give you an empty Postgres. Always check the WARNINGS column, and check that every important volume appears under Pod Volume Backups or CSI Snapshots.

Step 7 — Fix the Storage and Back Up Again

The local-path provisioner creates local volumes for a StorageClass annotated with defaultVolumeType: local. Save storageclass.yaml:

apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
name: local-fs
annotations:
# local-path-provisioner: create "local" PVs instead of "hostPath" PVs.
# Velero's file-system backup skips hostPath volumes.
defaultVolumeType: local
provisioner: rancher.io/local-path
reclaimPolicy: Delete
volumeBindingMode: WaitForFirstConsumer

A PVC's StorageClass is immutable, and this is a lab, so recreate the namespace on local-fs. Then start the port-forward again and post the same three bookings as in Step 4.

kubectl apply -f storageclass.yaml
sed -i '' 's/storageClassName: standard/storageClassName: local-fs/' hotel.yaml # GNU sed: sed -i
kubectl delete namespace hotel
kubectl apply -f hotel.yaml
kubectl get pv -o custom-columns='CLAIM:.spec.claimRef.name,CLASS:.spec.storageClassName,HOSTPATH:.spec.hostPath.path,LOCAL:.spec.local.path'
CLAIM CLASS HOSTPATH LOCAL
postgres-data local-fs <none> /var/local-path-provisioner/pvc-ab0a0a07-857b-47aa-b60d-db6c911abece_hotel_postgres-data
velero backup create hotel-backup-1 --include-namespaces hotel --ttl 72h --wait
velero backup describe hotel-backup-1 --details
kubectl -n velero get podvolumebackups
Pod Volume Backups - kopia:
Completed:
hotel/postgres-544f4454b-2mwtj: data (size: 47805222, incremental size: 47805222)

HooksAttempted: 1
HooksFailed: 0
NAME STATUS STARTED BYTES DONE TOTAL BYTES INCREMENTAL BYTES STORAGE LOCATION AGE
hotel-backup-1-57vtq Completed 4s 47805222 47805222 47805222 default 5s

This time 47.8 MB were copied, the data directory plus the dump. The hook output is in the backup log:

velero backup logs hotel-backup-1 | grep -E 'running exec hook|stdout: CHECKPOINT'
time="2026-10-21T12:20:17Z" level=info msg="running exec hook" backup=velero/hotel-backup-1 hookCommand="[/bin/bash -c pg_dump -U hotel -d bookings -f /var/lib/postgresql/data/bookings.sql && psql -U hotel -d bookings -c CHECKPOINT]" ...
time="2026-10-21T12:20:17Z" level=info msg="stdout: CHECKPOINT\n" backup=velero/hotel-backup-1 ...

In the bucket, manifests and metadata sit under backups/ and the encrypted volume data under kopia/hotel/:

s3 s3 ls s3://velero-backups/backups/hotel-backup-1/
2026-10-21 12:20:22 5685 hotel-backup-1-logs.gz
2026-10-21 12:20:22 14407 hotel-backup-1.tar.gz
2026-10-21 12:20:22 3490 velero-backup.json
...
velero backup download hotel-backup-1 -o hotel-backup-1.tar.gz
tar -tzf hotel-backup-1.tar.gz | grep -v preferredversion | grep ^resources | cut -d/ -f2 | sort | uniq -c

The tarball holds one JSON file per object, grouped by type: 24 events, 3 pods, 2 deployments.apps, 2 replicasets.apps, 2 services, 1 persistentvolumeclaims, 1 persistentvolumes, 1 secrets, and so on, across 13 resource types. The Secret is in that tarball, only base64-encoded. Your bucket is now as sensitive as the cluster, so encrypt it and restrict access to it.

Step 8 — Disaster One: The Namespace Is Gone

Deleting the namespace also deletes the PVC, and the Delete reclaim policy removes the PV with its data:

kubectl delete namespace hotel
kubectl get pv
namespace "hotel" deleted
No resources found
velero restore create hotel-restore-1 --from-backup hotel-backup-1 --wait
velero restore describe hotel-restore-1 --details
Phase: Completed
Total items to be restored: 19
Items restored: 19

Started: 2026-10-21 15:21:08 +0300 +03
Completed: 2026-10-21 15:21:16 +0300 +03

Warnings:
Namespaces:
hotel: could not restore, ConfigMap:kube-root-ca.crt already exists. Warning: the in-cluster version is different than the backed-up version
...
kopia Restores:
Completed:
hotel/postgres-544f4454b-2mwtj: data (size: 47805222)
...
v1/PersistentVolume:
- pvc-ab0a0a07-857b-47aa-b60d-db6c911abece(skipped)
v1/PersistentVolumeClaim:
- hotel/postgres-data(created)
...
v1/ServiceAccount:
- hotel/default(skipped)

The restore took eight seconds. Of the 44 backed-up items, the 24 events are excluded by design and the old PV is skipped, which leaves the 19 restored. Kubernetes creates kube-root-ca.crt and the default ServiceAccount in every new namespace before Velero gets to them, so those existing objects are skipped. The PVC was recreated, the provisioner made a new PV, and the node-agent filled it.

kubectl -n hotel get pod -l app=postgres -o jsonpath='{range .items[0].spec.initContainers[*]}{.name}{" "}{.image}{"\n"}{end}'
curl -s localhost:8080/bookings | jq -c '.[]'
curl -s localhost:8080/info
kubectl -n hotel exec deploy/postgres -- ls -la /var/lib/postgresql/data
restore-wait velero/velero:v1.18.4
{"id":1,"guest":"Ada Lovelace","room":101,"createdAt":"2026-10-21T12:19:58.75573Z"}
{"id":2,"guest":"Grace Hopper","room":102,"createdAt":"2026-10-21T12:19:58.78081Z"}
{"id":3,"guest":"Linus Torvalds","room":203,"createdAt":"2026-10-21T12:19:58.798857Z"}
{"pod":"bookings-api-5b56d87769-qxrl6","node":"dr-a-worker","bookings":3}
Defaulted container "postgres" out of: postgres, restore-wait (init)
...
-rw-r--r-- 1 root root 2306 Oct 21 12:20 bookings.sql
drwx------ 19 postgres root 4096 Oct 21 12:21 pgdata
drwxr-xr-x 2 root root 4096 Oct 21 12:21 .velero

The injected restore-wait init container kept Postgres from starting until the data was back. The bookings and the hook's dump file are both there.

Step 9 — A Restore Drill in a Separate Namespace

Add a fourth booking in production, then restore hotel-backup-1 next to it under another name:

curl -s -X POST localhost:8080/bookings -H 'Content-Type: application/json' -d '{"guest":"Margaret Hamilton","room":304}'
velero restore create hotel-restore-test --from-backup hotel-backup-1 \
--namespace-mappings hotel:hotel-restore-test --wait
for ns in hotel hotel-restore-test; do
kubectl -n $ns port-forward svc/bookings-api 8081:80 >/dev/null 2>&1 & PF=$!; sleep 3
echo "$ns: $(curl -s localhost:8081/info)"; kill $PF; sleep 1
done
{"id":4,"guest":"Margaret Hamilton","room":304,"createdAt":"2026-10-21T12:21:45.981427Z"}
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe hotel-restore-test` and `velero restore logs hotel-restore-test`.
hotel: {"pod":"bookings-api-5b56d87769-qxrl6","node":"dr-a-worker","bookings":4}
hotel-restore-test: {"pod":"bookings-api-5b56d87769-cs9gz","node":"dr-a-worker","bookings":3}

The copy holds the backup's point in time, three bookings, on its own PVC and database. That works because the connection string uses the short name Host=postgres, which resolves inside the Pod's own namespace. With postgres.hotel.svc.cluster.local, the drill copy would have written to production's database. Delete the drill when you are done:

kubectl delete namespace hotel-restore-test

Step 10 — Schedule Backups

velero schedule create hotel-nightly --schedule="0 2 * * *" --include-namespaces hotel \
--exclude-resources events,events.events.k8s.io --ttl 168h
velero schedule get
Schedule "hotel-nightly" created successfully.
NAME STATUS CREATED SCHEDULE BACKUP TTL LAST BACKUP SELECTOR PAUSED
hotel-nightly Enabled 2026-10-21 15:22:31 +0300 +03 0 2 * * * 168h0m0s n/a <none> false

In this run the schedule waited for its first cron tick and did not back up right away. Trigger one from it:

velero backup create --from-schedule hotel-nightly --wait
velero backup describe hotel-nightly-20261021122254 --details
Creating backup from schedule, all other filters are ignored.
...
Pod Volume Backups - kopia:
Completed:
hotel/postgres-544f4454b-2mwtj: data (size: 47805276, incremental size: 17034913)

Kopia deduplicates, so the second backup uploaded only 17 MB of 47.8 MB. The useless first backup can go:

velero backup delete hotel-first --confirm
Request to delete backup "hotel-first" submitted successfully.
The backup will be fully deleted after all associated data (disk snapshots, backup files, restores) are removed.

Step 11 — Disaster Two: The Whole Cluster Is Gone

Delete the cluster with everything in it. Only the bucket survives.

kind delete cluster --name dr-a --kubeconfig $PWD/kubeconfig-dr-a
Deleting cluster "dr-a" ...
Deleted nodes: ["dr-a-control-plane" "dr-a-worker"]

Rebuild the way you would in an emergency: cluster first, then the platform pieces the app expects (image, StorageClass), then Velero with the same velero install command as in Step 5.

kind create cluster --name dr-b --config kind-dr.yaml --kubeconfig $PWD/kubeconfig-dr-b
export KUBECONFIG=$PWD/kubeconfig-dr-b
kind load docker-image bookings-api:1.0 --name dr-b
kubectl apply -f storageclass.yaml

After the install, make the location read-only, and wait about a minute for the sync:

kubectl -n velero patch backupstoragelocation default --type merge -p '{"spec":{"accessMode":"ReadOnly"}}'
velero backup-location get
velero backup get
velero schedule get
NAME PROVIDER BUCKET/PREFIX PHASE LAST VALIDATED ACCESS MODE DEFAULT
default aws velero-backups Available 2026-10-21 15:25:08 +0300 +03 ReadOnly true
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION QUEUE POSITION SELECTOR
hotel-backup-1 Completed 0 0 2026-10-21 15:20:17 +0300 +03 2d default <none>
hotel-nightly-20261021122254 Completed 0 0 2026-10-21 15:22:54 +0300 +03 6d default <none>

Both backups are back. velero schedule get printed nothing, because Schedules never go to the bucket. Restore the newest backup:

velero restore create hotel-dr --from-backup hotel-nightly-20261021122254 --wait
curl -s localhost:8080/bookings | jq -c '.[]'
curl -s localhost:8080/info
Restore completed with status: Completed. You may check for more information using the commands `velero restore describe hotel-dr` and `velero restore logs hotel-dr`.
{"id":1,"guest":"Ada Lovelace","room":101,"createdAt":"2026-10-21T12:19:58.75573Z"}
{"id":2,"guest":"Grace Hopper","room":102,"createdAt":"2026-10-21T12:19:58.78081Z"}
{"id":3,"guest":"Linus Torvalds","room":203,"createdAt":"2026-10-21T12:19:58.798857Z"}
{"id":4,"guest":"Margaret Hamilton","room":304,"createdAt":"2026-10-21T12:21:45.981427Z"}
{"pod":"bookings-api-5b56d87769-cs9gz","node":"dr-b-worker","bookings":4}

Start a new port-forward against dr-b before running curl. All four bookings are back, including the one made after hotel-backup-1, and node comes from the Downward API on the new cluster. The timestamps of this run:

TimeEvent
15:22:54Last backup
15:23:28Cluster dr-a deleted
15:23:42Cluster dr-b created
15:24:40Velero installed
15:25:35Backups synced from the bucket
15:25:50Restore completed
15:26:28API answering with all four bookings

The RTO was about three minutes, for a tiny lab. Measure your own. The RPO was 34 seconds with no writes in between, so nothing was lost. With only the nightly schedule, the worst case is 24 hours.

Step 12 — Partial Restores and Promoting the New Cluster

If only a Service is deleted, restore just that resource type. Existing objects are left alone:

kubectl -n hotel delete service bookings-api
velero restore create hotel-services --from-backup hotel-nightly-20261021122254 --include-resources services --wait
velero restore describe hotel-services --details
Warnings:
Namespaces:
hotel: could not restore, Service:postgres already exists. Warning: the in-cluster version is different than the backed-up version
...
v1/Service:
- hotel/bookings-api(created)
- hotel/postgres(failed)

failed in the list is only a warning for the existing Service. Restoring the whole backup with --existing-resource-policy update over the running namespace produced warnings such as PersistentVolumeClaim "postgres-data" is invalid: spec: Forbidden: spec is immutable after creation and Pod "bookings-api-5b56d87769-cs9gz" is invalid: spec: Forbidden: pod updates may not change fields other than .... Use update for configuration such as ConfigMaps, Secrets and Deployments. To repair a database, restore into a clean namespace.

Once you trust dr-b, give the location write access back and recreate the schedule. In real life it comes from Git.

kubectl -n velero patch backupstoragelocation default --type merge -p '{"spec":{"accessMode":"ReadWrite"}}'
velero schedule create hotel-nightly --schedule="0 2 * * *" --include-namespaces hotel \
--exclude-resources events,events.events.k8s.io --ttl 168h
backupstoragelocation.velero.io/default patched
Schedule "hotel-nightly" created successfully.

Step 13 — Clean Up

kind delete cluster --name dr-b --kubeconfig $PWD/kubeconfig-dr-b
docker rm -f velero-s3
docker volume rm velero-s3-data
rm kubeconfig-dr-a kubeconfig-dr-b

Summary

  1. Define RPO (set by backup frequency) and RTO (measured by rehearsal) first. Replication is not a backup, because it copies mistakes too.
  2. An etcd snapshot restores one cluster's whole API state without volume data. Velero backs up selected objects and their volumes to object storage outside the cluster, and restores them anywhere.
  3. Velero is controllers plus CRDs: Backup, Restore, Schedule, BackupStorageLocation, and per-volume work items for the node-agent. The CLI only creates these objects.
  4. File-system backup with Kopia skips hostPath volumes with only a warning, so a Completed backup can still lack your data. Check --details and the warnings every time.
  5. Pre-backup hooks such as pg_dump and CHECKPOINT make database backups application-consistent.
  6. Restores run in priority order, never restore nodes or events, and skip existing objects unless you use --existing-resource-policy update, which cannot change immutable fields. --namespace-mappings gives safe drills next to production, but watch for fully qualified service names.
  7. For a new cluster: same bucket, same repository password, matching StorageClasses and images, the location set to ReadOnly during recovery, and Schedules recreated from Git. Here, a deleted cluster was serving its data again about three minutes after the rebuild started.
  8. Protect the bucket (it holds your Secrets), control retention with TTLs, and rehearse restores regularly.