Secrets & encryption at rest

Encrypt etcd; pull from external stores.

A Kubernetes Secret isn't secret. The name oversells it. What actually lands in etcd is your password base64-encoded, and base64 is encoding, not encryption: it scrambles nothing and needs no key to reverse. Think of the object less as a locked safe and more as a labeled envelope the cluster passes around. Anyone who can read it through the API, read etcd directly, or walk off with an etcd backup can turn that envelope back into plaintext in a single command. So a Secret is really a distribution mechanism, a convenient way to hand a credential to a pod, and nothing more. Three separate controls turn it into something you can actually trust: encrypt it at rest in etcd, restrict who is allowed to read it, and watch how the pod consumes it. Skip any one of them and the other two won't cover the gap.

# "encrypted"? no. one base64 decode away from plaintext
$ kubectl get secret db-creds -n payments -o jsonpath='{.data.password}' \
    | base64 -d
S3cr3tP@ss           # anyone with get on the secret reads this

Encrypt at rest

Encryption at rest tells the API server to encrypt secret data before it writes to etcd and to decrypt it on the way back out. You switch it on with one file, an EncryptionConfiguration, handed to the API server through --encryption-provider-config. The order inside that file matters more than it looks. The first provider listed does the encrypting; on a read, the API server walks the list and lets whichever provider recognizes the stored format decrypt it. That's why you keep identity, the no-op plaintext provider, last while you migrate: it lets the API server still read the old plaintext secrets until you've rewritten them. A KMS (Key Management Service) v2 provider is the strong choice, because the key-encryption key lives in Vault or a cloud KMS, outside the cluster, and only short-lived data-encryption keys ever reach the API server. With aescbc or aesgcm the key sits right there in the config file, which makes that file exactly as sensitive as the data it's meant to protect, and rotating it means editing YAML on every control-plane node by hand.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
- resources: [secrets]
  providers:
  - aescbc:                 # encrypts new writes (first = write provider)
      keys:
      - name: key1
        secret: <32-byte base64 key>
  - identity: {}            # read fallback for still-plaintext secrets
# wire it in: kube-apiserver --encryption-provider-config=/etc/kubernetes/enc/enc.yaml

Flipping encryption on only touches future writes. Every secret already sitting in etcd stays in whatever form it had when it was last saved, so a cluster you just configured is still full of plaintext. You fix that by forcing a rewrite of every secret, which pushes each one back through the write provider. The one-liner below reads all secrets and replaces them in place; on a big cluster, do it in batches so you don't hammer the API server. Then you prove it at the storage layer instead of trusting the flag. Read the raw bytes straight out of etcd with etcdctl: an encrypted value begins with k8s:enc:, a plaintext one is legible base64. The flag being set is not proof. The etcdctl read is.

$ kubectl get secrets -A -o json | kubectl replace -f -   # rewrite = re-encrypt
 
$ sudo ETCDCTL_API=3 etcdctl \
    --cacert=/etc/kubernetes/pki/etcd/ca.crt \
    --cert=/etc/kubernetes/pki/etcd/server.crt \
    --key=/etc/kubernetes/pki/etcd/server.key \
    get /registry/secrets/payments/db-creds | hexdump -C | head
... k8s:enc:aescbc:v1:key1: ...      # not plaintext base64 = encrypted

A stolen backup vs a stolen token

Encryption at rest solves exactly one threat: someone who ends up with your etcd data but not your running API server. A stolen snapshot, a disk pulled from a decommissioned node, a backup bucket left world-readable. It does nothing about a pod or a user that just asks the API for the Secret, because the API server hands those back already decrypted. Different attacker, different control. This one is RBAC (Role-Based Access Control), the cluster's guest list: it decides which identities may get which objects. Most real secret theft looks like this, not like a stolen backup: an attacker pops one pod, finds its mounted service-account token, and starts asking the API what else it can read. So scope get on secrets as tightly as you can, per namespace, and stop handing every pod a token it never uses. Check the design instead of assuming it: kubectl auth can-i answers as any identity you name.

# who can actually read secrets in payments? verify before you trust the design
$ kubectl auth can-i get secrets -n payments \
    --as=system:serviceaccount:payments:web
no                        # the web SA has no secrets:get, good
 
$ kubectl auth can-i get secrets -n payments \
    --as=system:serviceaccount:payments:vault-agent
yes                       # only the workload that genuinely needs it
 
# stop mounting unused tokens: pin automount off on the SA
$ kubectl patch sa web -n payments \
    -p '{"automountServiceAccountToken": false}'
serviceaccount/web patched

Consume it as a file, not an env var

How the pod reads the secret is its own small decision with real consequences. Injecting a secret as an environment variable feels convenient, but env vars leak. They surface in /proc/<pid>/environ, in crash dumps, in child processes that inherit the environment, and in the occasional debug endpoint or logging library that dumps the whole config on startup. A secret mounted as a file stays in one place, and you can mark the mount read-only so nothing in the container can rewrite it. Files also refresh when the Secret changes, while an env var is frozen at the moment the container started. For rotation, and to keep the source of truth out of etcd altogether, pull from an external store instead: Vault through the External Secrets Operator (ESO), or the Secrets Store CSI (Container Storage Interface) Driver, both of which project the live value into the pod at runtime.

spec:
  containers:
  - name: app
    image: registry.internal/app:1.4.2
    volumeMounts:
    - { name: creds, mountPath: /etc/creds, readOnly: true }   # file, not env
  volumes:
  - name: creds
    secret: { secretName: db-creds }

# prove the secret is a read-only file and nowhere in the environment
$ kubectl exec app -n payments -- env | grep -i pass
# no output: nothing sensitive in the process environment
$ kubectl exec app -n payments -- sh -c 'mount | grep /etc/creds'
tmpfs on /etc/creds type tmpfs (ro,relatime,...)   # ro = read-only

env var consumption leaks into process listings, crash dumps, and child processes. Files with tight modes are easier to rotate and harder to scrape casually.

A stolen etcd backup without encryption at rest is a credential dump. Test restore procedures with encryption enabled, not only happy-path snapshots.

External stores (Vault, cloud secret managers) still need a boot identity. The cluster Secret often remains the delivery mechanism — harden both ends.

Rotation drills matter. Change a key in the external store, roll the Deployment, and confirm old pods die before old credentials expire. EncryptionConfig key rotation also needs a practice run; the first time should not be during an audit. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Decode a Secret to remember base64 is not encryption, then check EncryptionConfiguration is active and prefer a file mount over an env var.

$ kubectl -n payments get secret db-creds -o jsonpath='{.data.password}' | base64 -d; echo
s3cr3t-pass
 
$ sudo grep -A2 providers /etc/kubernetes/manifests/kube-apiserver.yaml | head -20
    - --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
...
  - aescbc:
      keys:
      - name: key1
 
$ kubectl -n payments apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: secret-file }
spec:
  containers:
  - name: c
    image: busybox:1.36
    command: ["sleep","3600"]
    volumeMounts:
    - { name: db, mountPath: /etc/db-creds, readOnly: true }
  volumes:
  - name: db
    secret: { secretName: db-creds }
EOF
pod/secret-file created
$ kubectl -n payments exec secret-file -- ls /etc/db-creds
password
username

Takeaway

Secrets are envelopes. Encrypt etcd at rest, restrict get/list, mount as files, and keep RBAC away from blanket secret readers.

Related