Secrets

How secrets differ from config, and their real limits.

base64 is not encryption. That one sentence trips up more Kubernetes admins than anything else about Secrets. You drop a database password into a Secret, watch it turn into a wall of gibberish characters, and it feels protected. It isn't. Anyone allowed to read that Secret turns the gibberish back into your password with a single command.

A Secret is the object Kubernetes hands you for sensitive configuration: passwords, access tokens, TLS (Transport Layer Security) certificates, signing keys. It behaves almost exactly like a ConfigMap, the object for ordinary settings that aren't sensitive. Think of a ConfigMap as a settings sheet taped to the office fridge for anyone to read. A Secret is the same sheet for the sensitive values, folded shut. Folded, not locked. The real locks get bolted on separately, and bolting them on is your job, not something the object does for you.

How a Secret differs from a ConfigMap

Mechanically the two are twins. Both hold key-value data. Both get fed into a Pod (the smallest thing Kubernetes runs, one or more containers scheduled together as a unit) either as environment variables or as files, and your app reads them the same way. So why two object types at all? Intent and handling. Kubernetes treats anything typed as a Secret as the thing worth protecting. It keeps the values out of most command output. It can restrict a Secret so it only reaches the nodes actually running a Pod that needs it. And every security control worth having, encryption at rest, access rules, audit logs, points at Secrets specifically.

The values inside are base64-encoded. base64 is just a way of rewriting bytes in a printable alphabet so binary data survives being stored as text. It's reversible by design, and reversible by anyone. Treat it as a different font, not a lock.

Create one and read it straight back

kubectl create secret generic db-cred --from-literal=DB_PASSWORD='sup3rs3cr3t!'
kubectl get secret db-cred

secret/db-cred created
NAME      TYPE     DATA   AGE
db-cred   Opaque   1      2s

The imperative form is the fastest way in, and the one you'll grab for under exam pressure. Use --from-literal for values you type by hand, --from-file to load a file's contents as the value. Now watch how little that base64 actually buys you.

kubectl get secret db-cred -o jsonpath='{.data.DB_PASSWORD}' | base64 -d; echo

sup3rs3cr3t!

No key. No special permission. jsonpath (a way to pluck one field out of a reply) pulls the encoded value straight from the API server, the Application Programming Interface server, the cluster's single front door that every kubectl command talks to. Then base64 -d hands you the plaintext. The only thing standing between a person and that password was the right to read the Secret in the first place.

Env var or mounted file, and why it matters for rotation

An environment variable is like a value copied onto a sticky note when a worker clocks in for a shift. Change the master copy later and the sticky note still shows the old value, right up until that worker clocks out and back in. A mounted file is different. It's a live copy the worker re-reads every time they glance at it. Same Secret, very different behavior the moment you rotate a credential. Here's a Pod that reads the same Secret both ways at once.

apiVersion: v1
kind: Pod
metadata:
  name: app
spec:
  containers:
    - name: app
      image: busybox:1.36
      command: ["sleep", "3600"]
      env:
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-cred
              key: DB_PASSWORD
      volumeMounts:
        - name: db
          mountPath: /etc/db
          readOnly: true
  volumes:
    - name: db
      secret:
        secretName: db-cred

kubectl apply -f app.yaml
kubectl exec app -- cat /etc/db/DB_PASSWORD; echo
kubectl exec app -- mount | grep /etc/db

pod/app created
sup3rs3cr3t!
tmpfs on /etc/db type tmpfs (ro,relatime)

Look at the mount type: tmpfs. The kubelet (the Kubernetes agent that runs on every node and actually starts your containers) never writes Secret files to the node's disk. It keeps them in a memory-backed filesystem that vanishes on reboot. That protection comes free with a volume mount, and you lose it the second you switch to env vars. Rotation splits the two apart too. Update the Secret and the volume-mounted file refreshes on its own, usually within a minute or so. The lag is the kubelet's sync period plus a bit of cache propagation, and your app still has to notice the file changed and re-read it. An environment variable never refreshes. It's copied into the process once, at container start, and frozen there. Every Pod consuming that Secret as an env var keeps serving the old value until you restart it.

What actually protects it

Three controls do the real work here, and not one of them is the Secret simply being labeled a Secret.

Encryption at rest comes first. By default the API server writes Secret values into etcd (the cluster's key-value database, the place every object the cluster knows about actually lives) in the clear, base64 and nothing more. That's a diary written in plain ink sitting in an unlocked drawer. Grab an etcd snapshot or a nightly backup tarball and you've read every credential in the cluster, and RBAC never even entered the picture. An EncryptionConfiguration on the API server closes that hole, ideally backed by a KMS (Key Management Service, a cloud or hardware service that holds the encryption key so the cluster never stores it in the clear). Turn it on and Secret data is ciphertext before it ever touches disk.

Then there's tight RBAC. RBAC (Role-Based Access Control) is the cluster's guest list: it spells out who is allowed to do what. The part people miss is what a read actually returns. get and list hand back the whole Secret, values and all. They arrive base64-encoded, sure, but that's one base64 -d away from plaintext, the exact command you ran a minute ago. So read access to Secrets in a namespace is read access to every credential in that namespace. Check who has it before you trust your own setup.

kubectl auth can-i get secrets -n prod --as=system:serviceaccount:prod:ci-runner

yes

That yes means the ci-runner service account (a service account is the identity a program or Pod logs in as instead of a human, and this one belongs to an automated build pipeline) can pull every credential in the prod namespace. Grant Secret read to the exact namespaces and service accounts that genuinely need it, never cluster-wide out of habit. For the highest-value material, keep the source of truth outside the cluster entirely. The Secrets Store CSI (Container Storage Interface) driver mounts values from a system like HashiCorp Vault or a cloud secret store straight into the Pod, so unless you switch on its secret syncing the credential never becomes a Secret object in etcd at all. The External Secrets Operator works differently, and the difference matters: it reads from the same kind of external store but writes an ordinary Kubernetes Secret into the namespace, which then sits in etcd like any other. What it buys you is one source of truth and hands-off rotation, not a credential that has left etcd. And when a Secret never changes, set immutable: true. Think of it as laminating the card so nobody scribbles a new value onto it by accident. It also lets the kubelet on every node running a Pod that uses it drop its watch on that Secret, which takes real load off the API server on a big cluster.

stringData is for humans, data is for already-encoded bytes. Mixing both is fine, but stringData is write-only: the API server folds it into data on the way in, so a read never hands it back. And kubectl describe secret prints neither value, only key names and byte counts like DB_PASSWORD: 12 bytes. When you actually need to see a value, go back to get -o jsonpath piped through base64 -d.

Never commit Secret YAML with live values. Sealed Secrets, SOPS, or a vault CSI driver belong in the pipeline.

automountServiceAccountToken is a different secret path — the projected token. Turn it off when unused.

Try this

Create a generic Secret, get it, and base64-decode a key. Then mount it into a Pod, confirm the file arrives on a tmpfs mount, and rotate the value so you can watch the mounted file catch up while the environment variable stays stuck. Treat the demo as a reminder that etcd contents are sensitive.

kubectl create secret generic db-cred --from-literal=DB_PASSWORD='sup3rs3cr3t!'
kubectl get secret db-cred
kubectl get secret db-cred -o jsonpath='{.data.DB_PASSWORD}' | base64 -d; echo

secret/db-cred created
NAME      TYPE     DATA   AGE
db-cred   Opaque   1      2s
sup3rs3cr3t!

kubectl apply -f app.yaml
kubectl exec app -- cat /etc/db/DB_PASSWORD; echo
kubectl exec app -- mount | grep /etc/db

pod/app created
sup3rs3cr3t!
tmpfs on /etc/db type tmpfs (ro,relatime)

kubectl create secret generic db-cred --from-literal=DB_PASSWORD='n3wp4ssw0rd!' --dry-run=client -o yaml | kubectl replace -f -

secret/db-cred replaced

kubectl exec app -- cat /etc/db/DB_PASSWORD; echo
kubectl exec app -- sh -c 'echo "$DB_PASSWORD"'

n3wp4ssw0rd!
sup3rs3cr3t!

The mounted file followed the rotation. The environment variable is still the copy taken when the container started, and it stays wrong until the Pod restarts.

kubectl auth can-i get secrets -n prod --as=system:serviceaccount:prod:ci-runner

yes

Takeaway

Secrets are base64 in the API by default, not encryption. RBAC, encryption at rest, and external stores are what make them real.

Related