Kubernetes secrets, honestly

Base64 is not encryption; etcd is the real story.

In short. Kubernetes Secrets honestly: base64 is not encryption, and etcd is where the real risk sits.

Kubernetes — the system that schedules and runs your containers — ships a built-in object type called a Secret, and the name oversells it. A Secret is a sealed envelope, not a safe: it keeps the password out of casual view, but anyone allowed to touch the envelope can open it with one command. Real incidents trace back to teams who read "Secret," assumed "encrypted," and stopped thinking. This lesson covers what the object actually does, where the bytes really live, and which gaps you close with configuration versus which need different tools.

Concretely, a Secret is a small API object: a named bag of key/value pairs — a database password, an API token, a TLS private key — that the cluster stores centrally and delivers to pods that need it, as a mounted file or an environment variable. That is genuinely useful. It prevents the crudest failure: credentials baked into a container image, extractable by anyone who can pull the image, and it puts access behind the cluster's permission system. What it does not do, on a default cluster, is encrypt anything.

Base64 is encoding, not encryption

Secret values are stored base64-encoded. Base64 is a reversible text encoding — a way to write arbitrary bytes using only letters, digits, + and / so they survive being embedded in YAML or JSON. There is no key and no algorithm choice; decoding requires nothing but base64 -d:

$ kubectl create secret generic db-creds --from-literal=password='S3cr3t!'
secret/db-creds created
 
$ kubectl get secret db-creds -o jsonpath='{.data.password}'
UzNjcjN0IQ==
 
$ kubectl get secret db-creds -o jsonpath='{.data.password}' | base64 -d
S3cr3t!

Base64 is there because Secret values may be binary (a TLS key, a keystore) and binary does not embed cleanly in YAML — that is the entire reason. Treat "base64-encoded" as a synonym for *plaintext with extra steps*: anyone or anything that can read the object has the value.

Where the bytes actually live

Internally, kubectl sends the Secret to the kube-apiserver (the cluster's front door), which persists it in etcd — the key-value database holding all cluster state. By default it lands there unencrypted — and, as the hexdump below shows, not even base64-wrapped: the API's base64 is decoded and the raw bytes are written into etcd's protobuf serialization, exactly recoverable. When a pod referencing the Secret is scheduled, the kubelet (the agent on each node) fetches the value and materializes it for the container — as a file on a *tmpfs*, a RAM-backed filesystem that never touches the node's disk, or injected into the process environment. Verify the etcd claim yourself on a control-plane node:

# on a kubeadm control-plane node
$ sudo ETCDCTL_API=3 etcdctl \
    --cacert /etc/kubernetes/pki/etcd/ca.crt \
    --cert   /etc/kubernetes/pki/etcd/server.crt \
    --key    /etc/kubernetes/pki/etcd/server.key \
    get /registry/secrets/default/db-creds | hexdump -C | grep -A1 password
00000090  70 61 73 73 77 6f 72 64  12 07 53 33 63 72 33 74  |password..S3cr3t|
000000a0  21 1a 06 4f 70 61 71 75  65                       |!..Opaque|
# the password is sitting in the database — no decryption happened

The blast radius is larger than it first appears. etcd's data directory lives on the control-plane disk; etcd snapshots and cluster backups contain every secret in the cluster; and whoever can read a control-plane filesystem — or the bucket where backups land — holds all of them at once. At scale this is usually the real exposure: one over-shared backup bucket outweighs any number of carefully permissioned namespaces.

Turning on encryption at rest

Kubernetes can encrypt Secrets before they reach etcd. You hand the kube-apiserver an EncryptionConfiguration file listing providers in order: aescbc/aesgcm encrypt with a local key, kms (use API version v2, stable since Kubernetes 1.29) delegates to an external key service such as AWS KMS or Vault, and identity means "store plaintext." The first provider encrypts new writes; later ones exist so older data stays readable.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - aescbc:
          keys:
            - name: key1
              secret: 5FyIMHmYIm5XJ2fWCqA0M9TgUpwsPS62nBv+CWJ7RVA=  # head -c 32 /dev/urandom | base64
      - identity: {}   # fallback: read pre-existing plaintext data

# add to the kube-apiserver static pod manifest, then let it restart:
#   --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
 
# rewrite every existing secret so it gets encrypted on the way back in
$ kubectl get secrets -A -o json | kubectl replace -f -
secret/db-creds replaced
secret/registry-pull replaced
...
 
# verify: the stored value now carries an encryption header
$ sudo ETCDCTL_API=3 etcdctl get /registry/secrets/default/db-creds \
    --cacert ... --cert ... --key ... | head -2
/registry/secrets/default/db-creds
k8s:enc:aescbc:v1:key1:...binary ciphertext...

Two caveats keep this from being the full answer. With aescbc, the encryption key is a file on the same control-plane node as etcd — you have locked the safe and taped the key to its door, which mainly protects *offline* copies like backups and snapshots. A kms provider fixes that by keeping the key in an external service. On managed clusters you often get this for free: EKS, GKE, and AKS encrypt etcd with cloud KMS keys by default or one setting away — verify it rather than assume it.

Who can read a Secret

The other boundary is RBAC — Kubernetes role-based access control, the rules stating which users and service accounts may perform which verbs on which resources. For Secrets, the verbs get, list, and watch all return the full value, so a "read-only" role on secrets is a plaintext-disclosure role. Audit who holds those verbs, and scope roles to named secrets (resourceNames) where you can:

$ kubectl auth can-i get secrets -n prod \
    --as=system:serviceaccount:prod:ci-runner
no
 
$ kubectl auth can-i list secrets -n prod [email protected]
yes   # jenny can read every secret value in prod

Consumption shape matters too. Prefer mounting Secrets as files over environment variables: env vars leak through crash dumps, child processes, and debug endpoints (the mechanics were this course's first lesson), and a changed env var needs a pod restart, while a mounted Secret file updates in place within about a minute. Set defaultMode: 0400 on the mount, and mark stable secrets immutable: true — it prevents silent edits and cuts apiserver watch load in large clusters.

What Secrets don't do, and where this goes

Even hardened, the built-in object is a storage location, not a management system. There is no rotation: nothing expires, nothing regenerates, nothing notifies the app that a value changed. There is no versioning and no usage audit — API audit logs show who touched the object, never which process used which credential. Secrets are namespace-scoped, so one credential shared across fifty namespaces becomes fifty copies drifting apart. Those gaps — rotation, auditability, short-lived dynamic credentials — are precisely the case for external secret managers, which the final lesson of this course weighs.

There is a nearer problem first. A Secret manifest is YAML, and in a GitOps workflow YAML lives in git — and the previous lesson showed that anything committed lives in history, in every clone, effectively forever. Base64 in git is plaintext in git. The next lesson covers the two standard ways out: SOPS, which encrypts the values inside files you commit, and sealed-secrets, which encrypts a Secret so that only one specific cluster can ever open it.

Related