Encrypting Kubernetes Secrets at rest in etcd

Kubernetes Secrets are only base64 by default. Turn on an EncryptionConfiguration and rotate the key without downtime.

A Secret is base64-encoded in etcd, and base64 is an encoding. Whoever has the etcd data (a snapshot in an S3 bucket, a backup on a jump host, a control-plane disk image) has every password, token and TLS key in the cluster, without ever calling the API server. Encryption at rest closes exactly that path: the API server encrypts the value before it writes to etcd and decrypts on read, so a copy of etcd is ciphertext plus a key the copy does not contain. It does nothing about a user with RBAC permission to get secrets; that path is governed by RBAC and audited by the audit log.

Choose the provider before writing the config

Encryption providers

The order of providers in the file matters twice over: the first provider encrypts new writes, and every listed provider is tried in turn when reading. Keeping identity last during the migration lets the API server read Secrets that were written before encryption was enabled; removing it is the final step, not the first.

apiVersion: apiserver.config.k8s.io/v1
kind: EncryptionConfiguration
resources:
  - resources: [secrets]
    providers:
      - secretbox:
          keys:
            - name: key1
              secret: <32 random bytes, base64: head -c 32 /dev/urandom | base64>
      - identity: {}      # read-only fallback for objects written before encryption
 
# kube-apiserver flags (static pod manifest): mount the file read-only and pass
#   --encryption-provider-config=/etc/kubernetes/enc/enc.yaml
#   --encryption-provider-config-automatic-reload=true

Rewrite what already exists, then prove it

Once the API server restarts with the file, new Secrets are encrypted. Existing ones are still plaintext in etcd until something writes them again. The documented migration is a replace of every Secret through the API server, which reads each one and writes it back unchanged, now encrypted. The command is safe to repeat; a conflict on a Secret that changed mid-run is an error to retry, not a failure. On a large cluster, run it per namespace so a failure is easy to resume.

The prefix is the proof. k8s:enc:<provider>:v1:<keyname>: at the start of the stored value says which provider and which key were used, and a value with no such prefix is still plaintext. A script that lists Secrets whose stored form lacks the prefix is the check that the migration actually finished, because the API server will happily keep serving the unencrypted ones.

Rotating the key without downtime

Rotation follows the same read-many, write-one rule. Add the new key as the first entry so new writes use it, keep the old key listed so existing objects still decrypt, and make sure every API server has loaded the new file before rewriting: with --encryption-provider-config-automatic-reload=true the file is polled every minute and no restart is needed, and the metric apiserver_encryption_config_controller_automatic_reload_last_timestamp_seconds tells you when each server picked it up. Then run the replace again, back up the new key, and only then remove the old one. Removing a key while any Secret is still encrypted with it makes those Secrets unreadable, and no amount of restarting fixes that.

# edit 1: new key first, old key kept for reads; wait for reload on every apiserver,
#         then: kubectl get secrets -A -o json | kubectl replace -f -
providers:
  - secretbox:
      keys:
        - name: key2
          secret: <new 32-byte key>
        - name: key1
          secret: <old key>
  - identity: {}
 
# edit 2: after the rewrite completed and key2 is backed up
providers:
  - secretbox:
      keys:
        - name: key2
          secret: <new 32-byte key>
  # identity removed too, once no plaintext objects remain

The unchanged column is where the next controls live: least-privilege RBAC for who may read a Secret through the API, and External Secrets or Vault so the authoritative copy is outside the cluster and the object in etcd is a short-lived projection of it.

Related posts

Quick reference