Rotation & multi-recipient

Many keys; rotate without downtime.

One safe, one combination. Rather than reading the combination out loud across the office, you write it on a slip of paper, seal a copy in a separate envelope for each person who needs in, and tape the envelopes to the safe door. Anyone holding the right envelope opens theirs, reads the combination, and works the dial. Everyone else sees sealed paper. A file encrypted with SOPS (Secrets OPerationS, the tool that started at Mozilla in 2015, moved to the community getsops organisation, and is now a CNCF (Cloud Native Computing Foundation) Sandbox project) has exactly that shape. SOPS generates one random data key per file, encrypts every secret value with it, then encrypts a copy of that data key once for each recipient you name and stores those wrapped copies in a plaintext sops: block at the bottom of the file. The values get locked once. The data key gets wrapped many times.

Notice which half of the file gets locked, because it surprises people. SOPS encrypts values and leaves the keys around them alone. In the committed file, password: is still the readable word password:, and only the text after the colon becomes ENC[AES256_GCM,data:...]. A reviewer can see that a Secret gained a field, which environment it belongs to, and that somebody touched the database block, without reading one character of the credential itself. Git diffs stay useful, which is the whole reason to keep secrets in Git at all. The trade is that field names, file layout and the entire recipient list are public to anyone who can read the repository, so never hide anything sensitive in a key name.

Nothing in that design ties a file to one kind of key. An age recipient (age is the small modern encryption tool whose public keys start age1 and whose private keys start AGE-SECRET-KEY-1), an AWS KMS (Key Management Service) key, a Google Cloud KMS key, an Azure Key Vault key, a PGP (Pretty Good Privacy) fingerprint: each one holds its own wrapped copy of the same data key, and any single one of them opens the file. That is how a team shares encrypted files without anybody ever handing over a private key. Every engineer gets their own. The cluster gets its own. The CI (continuous integration) runner gets its own. Adding a person means adding an envelope, and the safe itself is never touched.

Who Is on the File, and Who Says So

Two places record the answer, and they drift apart more often than you would like. Your .sops.yaml holds intent: for files matching this path, encrypt to these recipients. The sops: block inside each encrypted file holds reality: these are the keys that can open this file right now. SOPS reads the config only when it creates a file or re-keys one, so editing .sops.yaml changes nothing about the files already sitting in your repository. No warning, no error. The gap opens quietly and stays open until a person, or some controller in a cluster, cannot decrypt.

creation_rules:
  # Production. Four recipients: the cluster's own identity, plus each
  # platform admin. Any one of them can open the file on its own.
  - path_regex: secrets/prod/.*\.enc\.yaml$
    encrypted_regex: '^(data|stringData)$'
    age: >-
      age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw,
      age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p,
      age1zj4dpv6d0lgh0nfx2rmhn7wgxr9vqcfcs7ep5hkg6f0kchdxu4hqz2xdhr,
      age1u79m4nlxrp8m9zt9tv9r0nctg7wnhcqz9c8sdc2yr7lqvwf4qdvq8h6djc
 
  # Everything else: the shared dev key. First matching rule wins, so this
  # broader rule must stay below the prod one.
  - path_regex: .*\.enc\.yaml$
    age: age1w4rq7sdh0mvxc5tz9pf3n2kljy6ge8ur7ahd4qzv0s5m9x2tcfqsq7dnh3

Two details in that file earn their keep. encrypted_regex matches key names, and anything under a matching key gets encrypted, so ^(data|stringData)$ locks the payload of a Kubernetes Secret and leaves metadata.name and namespace readable. The >- is plain YAML folding: it joins those lines into one long comma separated string, which is the form a creation rule expects for age at the top level. Now, the last recipient on that prod rule belongs to an admin who joined last week. Her key went into the config on Monday. Check what the file on disk actually accepts. You need no key at all to answer that, because the sops: block is ordinary plaintext YAML, and yq (a command line YAML processor, the same idea as jq for JSON) reads the recipient list straight out of the committed file.

# Who can open this file today? No decryption key required to find out.
yq '.sops.age[].recipient' secrets/prod/db.enc.yaml

age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw
age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
age1zj4dpv6d0lgh0nfx2rmhn7wgxr9vqcfcs7ep5hkg6f0kchdxu4hqz2xdhr

Four recipients in the config, three on the file. The new admin can open anything created since Monday and nothing created before it, which is the single most common SOPS support ticket in existence. That yq line cuts both ways. It is a free audit of who can read your production secrets, runnable by anyone on the team with no special access. It is equally free for an attacker who has read the repository and wants to know which laptop to go after.

Adding a Recipient With sops updatekeys

sops updatekeys closes the gap. It opens the file with a key you already hold, works out the recipient set the config now asks for, and re-encrypts the existing data key to that new set. It never reads or rewrites your secret values. By default it is interactive: it prints the change it is about to make and waits for you to approve, which is the right default for a command that edits who can read production. Add -y when you script it. One thing to know before you point it at a whole repository: a file matched by no creation rule is a hard error, not a skip. SOPS stops with The config file .sops.yaml does not contain any creation rule and returns non-zero.

sops updatekeys secrets/prod/db.enc.yaml

2026/07/22 09:14:02 Syncing keys for file /home/dev/platform/secrets/prod/db.enc.yaml
The following changes will be made to the file's groups:
Group 1
    age1yt3tfqlfrwdwx0z0ynwplcr6qxcxfaqycuprpmy89nr83ltx74tqdpszlw
    age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
    age1zj4dpv6d0lgh0nfx2rmhn7wgxr9vqcfcs7ep5hkg6f0kchdxu4hqz2xdhr
+++ age1u79m4nlxrp8m9zt9tv9r0nctg7wnhcqz9c8sdc2yr7lqvwf4qdvq8h6djc
Is this okay? (y/n):y
2026/07/22 09:14:05 File /home/dev/platform/secrets/prod/db.enc.yaml synced with new keys

Lines flush against the left margin are the changes: +++ adds a recipient, --- removes one, and the indented entries are recipients that stay. Group numbering starts at 1 in this diff, which is worth filing away, because the decryption error you will meet in a minute numbers the very same group 0. Do not stop at "the command exited zero" either. Verify the way the new admin will, with her key and nothing else. --extract pulls a single value out of the tree so you can prove access without dumping the whole file to your terminal, and SOPS_AGE_KEY_FILE points SOPS at one specific identity file instead of your usual one.

# Prove the new recipient can really open it, using only her key.
SOPS_AGE_KEY_FILE=~/keys/priya.agekey sops decrypt \
  --extract '["stringData"]["password"]' secrets/prod/db.enc.yaml

pg-prod-4f1c9d0b2e7a

The Overlap Window Buys You Zero Downtime

Because a file can list many recipients, you never have to swap a key. It works like changing the locks on a shared front door while people are still coming and going: you fit the second cylinder first, hand out the new keys, and only then pull the old one. You add the new recipient, confirm it works, then take the old one away. During the window in between, both keys open the file, so nothing that reads the file can break. Retiring the cluster's own identity looks like this. Generate the new keypair. Add its public half to .sops.yaml next to the old one. Run sops updatekeys across the repo and commit. The cluster keeps decrypting happily with its old private key the whole time. Now give the cluster the new private key alongside the old one, watch a reconcile succeed, and only then delete the old public key from the config and run updatekeys again.

# Flux's kustomize-controller imports every entry in this Secret whose name
# ends in .agekey, and an age identity file can hold several private keys
# (blank lines and # comments are ignored). Old and new identities coexist
# for as long as the rollout needs.
apiVersion: v1
kind: Secret
metadata:
  name: sops-age
  namespace: flux-system
type: Opaque
stringData:
  age.agekey: |
    # retiring identity (age1yt3tfql...), removed at the end of the window
    AGE-SECRET-KEY-1EG3M5N9WW6MLTPJ7QK9GVJ9XZL4KXD2ZQ7T5V8CHPRA6QYX4RSQ2DGCLWX
    # incoming identity (age1lggyhqrw...)
    AGE-SECRET-KEY-1V7HXQ5D3TZKM9SFQ2GPWLA8CJRU6YN4XZE5T7HKD2QVMSCX9RGQFT3W2LK

The same trick works everywhere else. Locally, SOPS_AGE_KEY_FILE can point at a file holding several identities and SOPS tries each until one fits. In CI, the runner's secret can carry two keys for a week. Skip the overlap and you get the other outcome. Replace the cluster's key in one commit, and the moment that commit lands, the file lists not a single key the cluster holds. Here is what that looks like when you reproduce it on your laptop with the retired identity.

# Simulate the cluster: try the file with only the retired identity.
SOPS_AGE_KEY_FILE=./cluster-retired.agekey sops decrypt secrets/prod/db.enc.yaml

Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  age1lggyhqrw2nlhcxprm67z43rta597azn8gk38fnfnk9htp5v05m5s34xnw2: FAILED
    - | failed to create reader for decrypting sops data key with
      | age: no identity matched any of the recipients
 
  age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p: FAILED
    - | failed to create reader for decrypting sops data key with
      | age: no identity matched any of the recipients
 
  age1zj4dpv6d0lgh0nfx2rmhn7wgxr9vqcfcs7ep5hkg6f0kchdxu4hqz2xdhr: FAILED
    - | failed to create reader for decrypting sops data key with
      | age: no identity matched any of the recipients
 
  age1u79m4nlxrp8m9zt9tv9r0nctg7wnhcqz9c8sdc2yr7lqvwf4qdvq8h6djc: FAILED
    - | failed to create reader for decrypting sops data key with
      | age: no identity matched any of the recipients
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Every recipient reported FAILED, which is the fingerprint of this specific mistake: the reader holds none of the listed keys. Flux and Argo CD surface that same block of text inside a reconciliation error. Nothing is corrupt and no secret has changed. The envelope addressed to that key is gone, that is all. Running workloads keep the Secret they already have, so the failure is invisible from the outside until the next deploy needs to write a new one, and then everything downstream of that Kustomization stops moving. Putting the retired public key back and re-running updatekeys restores service in one commit, which is why the overlap is worth the extra step.

updatekeys Re-addresses Envelopes, rotate Changes the Combination

sops rotate generates a brand new data key, re-encrypts every value under it, and wraps the new data key for the recipients. One detail trips people badly: rotate does not read .sops.yaml at all. It works from the recipient list already inside the file, plus whatever you pass on the command line: --add-age, --rm-age, --add-kms, --rm-kms, and the matching pairs for GCP KMS, Azure Key Vault, HashiCorp Vault and PGP. So updatekeys is config driven and rotate is file driven. Say that out loud once and you will save yourself an afternoon. A second detail matters only if you use key groups: any recipient you hand to --add-age lands in the first group, never in a group of its own. On version numbers, SOPS 3.9 introduced the sops encrypt, sops decrypt and sops rotate subcommands used throughout this lesson, and the older flag spellings such as sops -r -i still work.

# New data key, every value re-encrypted, one recipient dropped in one pass.
sops rotate --in-place \
  --rm-age age1zj4dpv6d0lgh0nfx2rmhn7wgxr9vqcfcs7ep5hkg6f0kchdxu4hqz2xdhr \
  secrets/prod/db.enc.yaml
 
# A rotate rewrites every ENC[...] blob, so the Git diff is the whole file
# and reviewing it line by line tells you nothing. Review the plaintext:
# last committed version vs the working copy, decrypted on the fly.
diff <(git show HEAD:secrets/prod/db.enc.yaml | sops decrypt --input-type yaml /dev/stdin) \
     <(sops decrypt secrets/prod/db.enc.yaml) && echo 'plaintext identical'

plaintext identical

--input-type yaml is doing real work in that line. SOPS picks its parser from the file extension, and /dev/stdin has no extension, so the fallback is the binary store: without the flag, SOPS treats your YAML as one opaque blob and the comparison fails in a way that reads like corruption. Bake that diff into your review habit for any rotation commit. It is the difference between "the tool said it worked" and "the values are provably the same and the file still opens."

Taking Access Away for Real

When an engineer leaves, run the steps in this order. Sync first so the departing key is off the files, then rotate so the new data key is wrapped only for the people who remain. Do it the other way round and the rotate hands the fresh data key to the person you are removing on its way out.

#!/usr/bin/env bash
set -euo pipefail
LEAVER=age1zj4dpv6d0lgh0nfx2rmhn7wgxr9vqcfcs7ep5hkg6f0kchdxu4hqz2xdhr
 
# 1. Intent: delete the key from every creation rule.
"${EDITOR:-vi}" .sops.yaml
 
# 2. Reality: sync the recipient list on every encrypted file (config-driven).
#    A file matched by no creation rule errors out rather than being skipped,
#    and set -e stops the script, which is what you want here.
find . -name '*.enc.yaml' -print0 | xargs -0 -n1 sops updatekeys -y
 
# 3. Invalidate the data key they may already hold (file-driven).
find . -name '*.enc.yaml' -print0 | xargs -0 -n1 sops rotate --in-place
 
# 4. Prove it: their key must not appear in any encrypted file.
! grep -rq "$LEAVER" --include='*.enc.yaml' .
 
# 5. The part SOPS cannot do: change the credentials themselves in the
#    systems that issue them, then encrypt the new values into these files.
echo 'now rotate the database password, the API tokens, the cloud keys'

Step 5 is the one people skip, and it is the one that matters. Every old version of every file is still in Git history, still encrypted to whoever was a recipient at the time, and still openable by anyone who kept that key. Rewriting history does not save you either, because forks, mirrors, CI caches and laptops already have copies. Treat any secret that was ever encrypted to a key you no longer control as burned, and reissue it at the source. SOPS decides who can open a file. It cannot make anyone un-remember what they read.

Splitting the Combination With Key Groups

Sometimes one envelope is one too many. Two bank managers, two physical keys, one vault door that opens for neither of them alone. For the credentials that would end your week if a single laptop were stolen, SOPS supports key groups with a Shamir threshold, named after Shamir's Secret Sharing (a scheme that splits a secret into shares so that a set number of shares, and no fewer, can rebuild it). SOPS splits the data key into one share per group and needs shamir_threshold groups to put it back together. Inside a group, any single key still opens that group's share.

creation_rules:
  # Root credentials: no single team, and no single stolen laptop, can open
  # this file. Any 2 of the 3 groups together can.
  - path_regex: secrets/break-glass/.*\.enc\.yaml$
    shamir_threshold: 2
    key_groups:
      - age:                      # platform
          - age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
      - age:                      # security
          - age1w4rq7sdh0mvxc5tz9pf3n2kljy6ge8ur7ahd4qzv0s5m9x2tcfqsq7dnh3
      - kms:                      # a KMS key in a separate AWS account
          - arn: arn:aws:kms:eu-west-1:111122223333:key/1a2b3c4d-5e6f-7a8b-9c0d-1e2f3a4b5c6d

Note the shape change. At the top level of a creation rule, age is that one comma separated string you saw earlier. Inside key_groups, every backend takes a proper YAML list instead, kms takes a list of objects each with an arn, and the Azure and Google fields are spelled azure_keyvault and gcp_kms. You can also edit groups on an existing file directly, which is handy when a team is added or dissolved. Both group subcommands print the rewritten file to standard output unless you pass --in-place, and it is easy to run one, see a wall of ciphertext scroll past, and believe the file changed when it did not.

# Add a group holding the security team's key, and require any 2 of 3.
# Without --in-place this prints the new file to stdout and changes nothing.
sops groups add --in-place --file secrets/break-glass/root.enc.yaml \
  --age age1w4rq7sdh0mvxc5tz9pf3n2kljy6ge8ur7ahd4qzv0s5m9x2tcfqsq7dnh3 \
  --shamir-secret-sharing-threshold 2
 
# Drop a group entirely. The index is a positional argument, not a flag,
# and indexes start at 0.
# sops groups delete --in-place --file secrets/break-glass/root.enc.yaml 2
 
yq '.sops.shamir_threshold, (.sops.key_groups | length)' secrets/break-glass/root.enc.yaml

2
3

The honest cost: everything that needs the data key now needs two groups present at the same moment. Your CI runner cannot decrypt a 2-of-3 file on its own, by design, and neither can a single admin doing a routine updatekeys, because updatekeys has to recover the data key before it can re-wrap it. Even sops groups add has to meet the current threshold first. That friction is the whole point for break-glass material and pure obstruction for application config, so keep threshold rules to a small, deliberate set of paths.

Make CI Notice the Drift

Since the config is intent and the files are truth, let a machine compare them on every change. sops updatekeys is idempotent (running it a second time changes nothing the second time), and when a file already matches the config it logs already up to date and does not write to disk at all. So a job that runs it across the repo and then asks Git whether anything changed will pass when everything is in sync and fail loudly when a .sops.yaml edit was merged without the follow-up sweep.

# CI drift check. Needs a key that can decrypt, so pin it to a trusted
# runner: never run this on pull requests from forks.
find . -name '*.enc.yaml' -print0 | xargs -0 -n1 sops updatekeys -y
 
git diff --quiet -- '*.enc.yaml' || {
  echo 'recipients drifted from .sops.yaml; run sops updatekeys and commit'
  git diff --stat -- '*.enc.yaml'
  exit 1
}

2026/07/22 03:11:19 Syncing keys for file /builds/platform/secrets/prod/db.enc.yaml
2026/07/22 03:11:19 File /builds/platform/secrets/prod/db.enc.yaml already up to date
2026/07/22 03:11:20 Syncing keys for file /builds/platform/secrets/prod/api.enc.yaml
The following changes will be made to the file's groups:
Group 1
    age1lggyhqrw2nlhcxprm67z43rta597azn8gk38fnfnk9htp5v05m5s34xnw2
    age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
+++ age1u79m4nlxrp8m9zt9tv9r0nctg7wnhcqz9c8sdc2yr7lqvwf4qdvq8h6djc
2026/07/22 03:11:21 File /builds/platform/secrets/prod/api.enc.yaml synced with new keys
recipients drifted from .sops.yaml; run sops updatekeys and commit
 secrets/prod/api.enc.yaml | 15 +++++++++++++--
 1 file changed, 13 insertions(+), 2 deletions(-)

Read that output as a two file story: db.enc.yaml was already correct and untouched, api.enc.yaml was missing the new admin and got fixed in the runner's working copy, which is what turns the build red. Pair the check with sops filestatus secrets/prod/db.enc.yaml, which prints {"encrypted":true} and costs you nothing as a guard against someone committing a decrypted file by accident. Between those two, a missing recipient becomes a red build on the pull request that caused it, instead of a decryption error a cluster reports to you at 3am.

Try this

Run yq '.sops.age[].recipient' secrets/prod/db.enc.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: updatekeys does not invalidate anything. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related