Kubernetes Secret

Lesson 11 of 39 15 min read Updated

In the previous lesson you stored non-sensitive settings in ConfigMaps. Passwords, API keys, TLS certificates and database credentials need more care. ConfigMap data appears in plain text whenever someone runs kubectl get configmap -o yaml, and ConfigMaps are not designed to be locked down.

Kubernetes has a dedicated object for sensitive data: the Secret. If a ConfigMap is the settings sheet you hand to an app, a Secret is the sealed envelope with the keys to the building. You hand it over the same way, but you are much stricter about who may open it.

Secrets let you build container images without any credentials inside them. The same image runs in development, staging and production, and only the Secrets differ. Pods receive Secrets through explicit references in their manifests, and Secret volumes live on tmpfs (a RAM disk), so the files are never written to the node's disk.

What you will learn

  1. How Secrets differ from ConfigMaps, and which built-in Secret types exist
  2. Why base64 is encoding, not encryption, and how to protect Secrets in a real cluster
  3. How to create, inspect, decode and edit Secrets with kubectl
  4. How to give a .NET app database credentials as environment variables and TLS certificates as files
  5. How to pull images from a private registry with imagePullSecrets

Secrets vs ConfigMaps

Secrets and ConfigMaps look alike: both are key-value stores that Pods consume. The differences matter as soon as security matters:

FeatureConfigMapSecret
Intended dataNon-sensitive configurationPasswords, tokens, keys, certificates
How values are writtenPlain text in data (binary in binaryData)Base64 in data, or plain text in stringData
Volume backingFiles on the node's disktmpfs (RAM disk), never written to the node's disk
Size limit1 MiB1 MiB
kubectl describeShows the valuesShows only key names and sizes
Access controlRBAC, often readable by many usersRBAC, same model, but you grant it to as few users as possible
Encryption at rest in etcdPossible, rarely enabledShould be enabled with an EncryptionConfiguration
Same mechanics, more care. Pods consume Secrets exactly like ConfigMaps: as environment variables or as files. The difference is how Kubernetes stores them and who you allow to read them.

Secret types

Every Secret has a type. It tells Kubernetes, and the people reading the manifest, what kind of data the Secret holds. Some types also make the API server check that the required keys are present.

TypeDescription
OpaqueThe default type. Holds any key-value data.
kubernetes.io/tlsA TLS certificate and its private key. Requires the keys tls.crt and tls.key.
kubernetes.io/dockerconfigjsonCredentials for private container registries.
kubernetes.io/basic-authA username and a password.
kubernetes.io/ssh-authAn SSH private key.
kubernetes.io/service-account-tokenA long-lived ServiceAccount token. Legacy: modern clusters give Pods short-lived tokens automatically.

Day to day you will mostly use Opaque (database passwords, API keys, tokens) and kubernetes.io/tls (HTTPS certificates).

Base64 is encoding, not encryption

Values in a Secret's data field are base64-encoded. Base64 is reversible, so anyone who can read the Secret can decode it in one command:

echo "YmlsbGluZ19zdmM=" | base64 --decode

Expected output:

billing_svc

Base64 exists so that a Secret can carry binary data, such as certificates, inside YAML, which is a text format. On its own it provides no security at all.

If you want to write plain-text values in a manifest, use stringData instead of data. Kubernetes encodes the values for you when it stores the Secret. If the same key appears in both fields, the stringData value wins.

apiVersion: v1
kind: Secret
metadata:
name: example-stringdata
type: Opaque
stringData:
DB_USER: billing_svc
DB_PASSWORD: "X9k!vLp2@dQm7!"
Base64 hides nothing. Treat a Secret manifest with real values like a password written on paper. Never commit it to Git, paste it into a ticket or share it in chat.

Protect Secrets in a real cluster

By default, Kubernetes stores Secrets unencrypted in etcd, the cluster's database. Anyone with access to etcd or with broad admin rights can read every Secret. Also, anyone who can create a Pod in a namespace can mount any Secret in that namespace. For production clusters:

  1. Enable encryption at rest with an EncryptionConfiguration and a key you control, ideally from a cloud KMS. Managed clusters (AKS, EKS, GKE) offer this as a setting.
  2. Use RBAC rules to limit who can get, list and watch Secrets, and who can create Pods in sensitive namespaces.
  3. Consider the Secrets Store CSI Driver to read secrets straight from Azure Key Vault, AWS Secrets Manager or HashiCorp Vault, so they are never stored in etcd at all.
  4. Prefer mounted files over environment variables for the most sensitive values. Environment variables leak more easily through logs, crash dumps and child processes.

Naming rules for keys

Keys in Secrets and ConfigMaps follow the same rules:

  1. Only letters, digits, dashes (-), underscores (_) and dots (.). No spaces, slashes or colons.
  2. At most 253 characters.
  3. A key cannot be . or .., and cannot start with ... Kubernetes uses names like ..data internally inside mounted volumes.
Valid key nameInvalid key name
.auth_tokenauth file.json (space)
key.pemcerts/key.pem (slash)
config_file..data (starts with ..)
_password.txtapi:key (colon)

Keys become file names when you mount a Secret, so pick names that read well as files: key.pem is clearer than tls-key when an app expects certificate files. If you load keys with envFrom, stick to letters, digits and underscores so every key is also a normal environment variable name.

Working with Secrets from the command line

The kubectl create secret command has a sub-command for each common Secret type.

Generic (Opaque) Secrets

Use generic to create an Opaque Secret from literal values. Wrap values in single quotes so your shell does not interpret characters such as ! or $:

kubectl create secret generic my-db-secret \
--from-literal=DB_USER=admin \
--from-literal=DB_PASSWORD='s3cr3t!Pass'

You can also load values from files. Each file name becomes a key and the file content becomes its value:

kubectl create secret generic my-tls-secret \
--from-file=server.crt \
--from-file=server.key

This produces an Opaque Secret with the keys server.crt and server.key. For a proper TLS Secret, use the next command.

TLS Secrets

The tls sub-command checks that the certificate and key are valid and belong together:

kubectl create secret tls my-app-tls \
--cert=server.crt \
--key=server.key

This creates a Secret of type kubernetes.io/tls with the keys tls.crt and tls.key.

Docker registry Secrets

To store credentials for pulling images from a private registry:

kubectl create secret docker-registry my-registry-secret \
--docker-server=myregistry.azurecr.io \
--docker-username=myuser \
--docker-password='myP@ssw0rd' \

This creates a Secret of type kubernetes.io/dockerconfigjson. The --docker-email flag is optional.

View Secrets

List the Secrets in the current namespace, then look at one of them:

kubectl get secrets
kubectl describe secret my-db-secret

describe shows key names and sizes but not the values, as a deliberate safety measure. Expected output (trimmed):

Name: my-db-secret
Namespace: default
Type: Opaque

Data
====
DB_PASSWORD: 11 bytes
DB_USER: 5 bytes

To see the base64-encoded values, print the YAML:

kubectl get secret my-db-secret -o yaml

To decode a single value:

kubectl get secret my-db-secret -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode

Expected output:

s3cr3t!Pass

Edit a Secret

kubectl edit secret my-db-secret

The editor shows the values base64-encoded, so encode any new value before you paste it in:

echo -n "newPassword123" | base64

Expected output:

bmV3UGFzc3dvcmQxMjM=

Often it is easier to regenerate the whole Secret from plain values and apply it over the existing one:

kubectl create secret generic my-db-secret \
--from-literal=DB_USER=admin \
--from-literal=DB_PASSWORD='newPassword123' \
--dry-run=client -o yaml | kubectl apply -f -

Delete a Secret

kubectl delete secret my-db-secret

Just like with ConfigMaps, a Pod that references a deleted Secret keeps running, but it fails to start again (for example after a restart or a reschedule) until the Secret exists again.

Hands-on: secure a .NET Billing API

You will build a realistic setup: a .NET Billing API that needs database credentials as environment variables and a TLS certificate as files. You will create two Secrets and a Deployment that uses both.

As in the ConfigMaps lesson, the Deployment uses Microsoft's sample image mcr.microsoft.com/dotnet/samples:aspnetapp as a stand-in for your own Billing API image. It listens on port 8080 and answers health checks on /healthz. You also need openssl on your machine for Step 2.

Step 1 — Create the database credentials Secret

This Opaque Secret holds the database connection details. Save it as billing-api-db-credentials.yaml:

apiVersion: v1
kind: Secret
metadata:
name: billing-api-db-credentials
type: Opaque
data:
DB_HOST: cG9zdGdyZXMuaW50ZXJuYWwubG9jYWw=
DB_PORT: NTQzMg==
DB_NAME: YmlsbGluZ19kYg==
DB_USER: YmlsbGluZ19zdmM=
DB_PASSWORD: WDlrIXZMcDJAZFFtNyE=

Every value under data is base64-encoded. Here is what each one decodes to:

KeyBase64 valueDecoded value
DB_HOSTcG9zdGdyZXMuaW50ZXJuYWwubG9jYWw=postgres.internal.local
DB_PORTNTQzMg==5432
DB_NAMEYmlsbGluZ19kYg==billing_db
DB_USERYmlsbGluZ19zdmM=billing_svc
DB_PASSWORDWDlrIXZMcDJAZFFtNyE=X9k!vLp2@dQm7!

You can produce these values on the command line:

echo -n "postgres.internal.local" | base64

Expected output:

cG9zdGdyZXMuaW50ZXJuYWwubG9jYWw=
Always use echo -n. Without -n, echo adds a newline, the newline gets encoded into the value, and your app fails with confusing login or connection errors. Writing plain values under stringData avoids the problem entirely.

Apply the Secret:

kubectl apply -f billing-api-db-credentials.yaml

Step 2 — Create the TLS Secret

A TLS Secret stores an HTTPS certificate and its private key. In YAML it has this shape:

apiVersion: v1
kind: Secret
metadata:
name: billing-api-tls
type: kubernetes.io/tls
data:
tls.crt: <base64-encoded certificate>
tls.key: <base64-encoded private key>

The placeholders are not valid base64, so do not apply this file as it is. In production the certificate comes from a Certificate Authority such as Let's Encrypt. For this lesson, generate a self-signed certificate:

openssl req -x509 -nodes -days 365 -newkey rsa:2048 \
-keyout billing-api.key \
-out billing-api.crt \
-subj "/CN=billing-api.local"

Then create the TLS Secret from the two files. This is the easiest way, because kubectl encodes the files and checks that the certificate and key match:

kubectl create secret tls billing-api-tls \
--cert=billing-api.crt \
--key=billing-api.key

Expected output:

secret/billing-api-tls created

Step 3 — Create the Deployment that uses both Secrets

This Deployment injects the database credentials as environment variables and mounts the TLS certificate as files. Save it as billing-api-deployment.yaml:

apiVersion: apps/v1
kind: Deployment
metadata:
name: billing-api
labels:
app: billing-api
spec:
replicas: 2
selector:
matchLabels:
app: billing-api
template:
metadata:
labels:
app: billing-api
spec:
containers:
- name: billing-api
image: mcr.microsoft.com/dotnet/samples:aspnetapp
ports:
- containerPort: 8080
- containerPort: 8443
env:
- name: ASPNETCORE_ENVIRONMENT
value: "Production"
- name: Database__Host
valueFrom:
secretKeyRef:
name: billing-api-db-credentials
key: DB_HOST
- name: Database__Port
valueFrom:
secretKeyRef:
name: billing-api-db-credentials
key: DB_PORT
- name: Database__Name
valueFrom:
secretKeyRef:
name: billing-api-db-credentials
key: DB_NAME
- name: Database__User
valueFrom:
secretKeyRef:
name: billing-api-db-credentials
key: DB_USER
- name: Database__Password
valueFrom:
secretKeyRef:
name: billing-api-db-credentials
key: DB_PASSWORD
- name: ConnectionStrings__BillingDb
value: "Host=$(Database__Host);Port=$(Database__Port);Database=$(Database__Name);Username=$(Database__User);Password=$(Database__Password)"
volumeMounts:
- name: tls-certs
mountPath: /app/certs
readOnly: true
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
readinessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
livenessProbe:
httpGet:
path: /healthz
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
volumes:
- name: tls-certs
secret:
secretName: billing-api-tls

Apply it and wait for the rollout:

kubectl apply -f billing-api-deployment.yaml
kubectl rollout status deployment/billing-api

Expected output:

deployment "billing-api" successfully rolled out

Port 8443 is declared for when your own app serves HTTPS with the mounted certificate. The sample image only listens on 8080, which is why the probes use that port.

Step 4 — Inject individual keys with secretKeyRef

Each secretKeyRef picks one key from the Secret and exposes it under the environment variable name you choose:

env:
- name: Database__Host # Variable name the .NET app reads
valueFrom:
secretKeyRef:
name: billing-api-db-credentials # Which Secret
key: DB_HOST # Which key in that Secret

This works exactly like configMapKeyRef, but points at a Secret. In .NET, the double underscore in Database__Host maps to the configuration path Database:Host in IConfiguration.

The last variable, ConnectionStrings__BillingDb, is built from the others. Kubernetes replaces $(Database__Host) and the other references with their values, so your app can call builder.Configuration.GetConnectionString("BillingDb") and get a complete connection string. This only works for variables defined earlier in the same env list.

You can also inject every key at once with envFrom:

envFrom:
- secretRef:
name: billing-api-db-credentials

This creates the variables DB_HOST, DB_PORT, DB_NAME, DB_USER and DB_PASSWORD, one for each key in the Secret.

Step 5 — Mount the TLS certificate as files

The certificate reaches the container through a Secret volume:

volumeMounts:
- name: tls-certs
mountPath: /app/certs # Folder where the cert files appear
readOnly: true

volumes:
- name: tls-certs
secret:
secretName: billing-api-tls

Each key becomes one file. Check it from a running Pod:

kubectl exec deploy/billing-api -- ls /app/certs/

Expected output:

tls.crt
tls.key

Run ls -la instead and you will also see hidden entries such as ..data. They are symbolic links Kubernetes uses to swap in new files at once when the Secret changes. Like ConfigMap volumes, Secret volumes update automatically, unless you mount them with subPath.

The folder is a tmpfs volume in RAM, so the key never touches the node's disk. You can see it in the Filesystem column:

kubectl exec deploy/billing-api -- df -h /app/certs

Step 6 — Verify the environment variables

Confirm that the Secret values reached the container:

kubectl exec deploy/billing-api -- printenv | grep -E "Database__|ConnectionStrings__"

Expected output (the order may differ):

Database__Host=postgres.internal.local
Database__Port=5432
Database__Name=billing_db
Database__User=billing_svc
Database__Password=X9k!vLp2@dQm7!
ConnectionStrings__BillingDb=Host=postgres.internal.local;Port=5432;Database=billing_db;Username=billing_svc;Password=X9k!vLp2@dQm7!

The values are plain text. Kubernetes decodes base64 automatically when it injects Secret data into environment variables or volume files, so your app never deals with base64.

That is the payoff. Neither the image nor the Deployment manifest contains a single password. To rotate a credential, update the Secret and run kubectl rollout restart deployment/billing-api.

Step 7 — Pull images from a private registry

When a Pod pulls an image from a private registry, such as Azure Container Registry or a private Docker Hub repository, the kubelet needs credentials. A registry Secret provides them. Create one:

kubectl create secret docker-registry acr-pull-secret \
--docker-server=myregistry.azurecr.io \
--docker-username=myuser \
--docker-password='myP@ssw0rd' \

Then reference it in any Pod that pulls from that registry with spec.imagePullSecrets:

apiVersion: v1
kind: Pod
metadata:
name: private-app
spec:
containers:
- name: private-app
image: myregistry.azurecr.io/billing-api:1.0
ports:
- containerPort: 8080
resources:
requests:
cpu: "100m"
memory: "128Mi"
limits:
cpu: "500m"
memory: "256Mi"
imagePullSecrets:
- name: acr-pull-secret

imagePullSecrets tells the kubelet which Secret to use when it signs in to the registry. Without it, the registry rejects the pull and the Pod ends up in ImagePullBackOff. The Secret must live in the same namespace as the Pod.

If every Pod in a namespace pulls from the same registry, attach the Secret to the namespace's default ServiceAccount instead of repeating it in every manifest:

kubectl patch serviceaccount default \
-p '{"imagePullSecrets": [{"name": "acr-pull-secret"}]}'

Step 8 — Clean up

Remove the resources from the hands-on part:

kubectl delete deployment billing-api
kubectl delete secret billing-api-db-credentials billing-api-tls
kubectl delete pod private-app --ignore-not-found

Or delete them by manifest file:

kubectl delete -f billing-api-deployment.yaml
kubectl delete -f billing-api-db-credentials.yaml
kubectl delete secret billing-api-tls

Then remove the Secrets from the command-line section, undo the ServiceAccount patch and delete the local key files:

kubectl delete secret acr-pull-secret my-db-secret my-tls-secret my-app-tls my-registry-secret example-stringdata --ignore-not-found
kubectl patch serviceaccount default --type=json -p '[{"op": "remove", "path": "/imagePullSecrets"}]'
rm billing-api.crt billing-api.key

Common mistakes

SymptomLikely causeFix
App fails to log in with a correct-looking passwordA trailing newline was encoded (echo without -n)Re-encode with echo -n, or use stringData
kubectl apply fails with illegal base64 dataPlain text under dataMove the values to stringData, or encode them
Pod stuck in CreateContainerConfigErrorThe Secret or key does not exist, or lives in another namespaceCreate the Secret in the Pod's namespace and check key names
Pod stuck in ImagePullBackOff for a private imageNo imagePullSecrets, or wrong credentialsAdd the registry Secret to the Pod or the ServiceAccount
App still uses the old passwordThe value comes from an environment variablekubectl rollout restart deployment/<name>

Summary

Secrets are the Kubernetes way to store and distribute sensitive data such as passwords, tokens and certificates. Pods use them just like ConfigMaps, but you store and guard them more carefully.

  1. A Secret stores sensitive key-value data, base64-encoded. Base64 is not encryption: anyone who can read the Secret can decode it.
  2. Common types are Opaque for general data, kubernetes.io/tls for certificates and kubernetes.io/dockerconfigjson for registry credentials.
  3. Create Secrets imperatively with kubectl create secret generic | tls | docker-registry, or declaratively in YAML with data (base64) or stringData (plain text).
  4. Consume them as environment variables (secretKeyRef or envFrom with secretRef) or as files on a tmpfs volume.
  5. Image pull Secrets authenticate with private registries through spec.imagePullSecrets or the default ServiceAccount.
  6. Secrets are unencrypted in etcd by default. In production, enable encryption at rest, restrict access with RBAC, and consider the Secrets Store CSI Driver for external vaults.
  7. Key names may contain only letters, digits, -, _ and ., and cannot start with ... Choose names that work as file names, such as key.pem.

With ConfigMaps for settings and Secrets for credentials, your images stay generic and every environment gets exactly the configuration it needs.