The secrets-in-Git problem & SOPS

Why plaintext in Git is dangerous.

Someone on your team pastes a production database password into config/prod.yaml, commits it, and pushes. The pipeline goes green. No linter complains, no alarm fires, nobody comments on the pull request. That is what makes it dangerous. Committing a secret costs you nothing on the day you do it, so nothing in your workflow stops you. The bill arrives months later, from a direction nobody was watching.

Git is a ledger, not a folder of files. Picture a filing cabinet that quietly photocopies every page you ever put in it, keeps the copies of the pages you later shredded, and hands a complete duplicate of itself to anyone holding a library card. Every version of every file you have ever committed is kept as a permanent object, and git clone gives the whole pile to whoever asks. Once a secret lands in a commit it is in every clone, every fork, every nightly backup, and every CI (continuous integration, the robot that builds and tests your code on a server somewhere) runner cache that ever fetched the repository.

SOPS (Secrets OPerationS) takes a different route from "keep secrets out of Git". It encrypts the sensitive values inside your config file so the file itself becomes safe to commit, review, and diff. The project started at Mozilla in 2015, went quiet for a while, was donated to the CNCF (Cloud Native Computing Foundation, the group that also houses Kubernetes) as a Sandbox project in 2023, and now lives in the getsops GitHub organization. Old blog posts and Dockerfiles still point at mozilla/sops. The code, the releases, and the issues are at github.com/getsops/sops, and every command below is written for the current 3.13 release line.

Why a deleted secret is still a live secret

Deleting a password from a file and committing the change does not remove the password. It writes a new commit that says "this line is now gone". The old commit still exists, still holds the password, and is still handed out to anyone who fetches the repository. Your working tree looks clean. A grep -r across the checkout finds nothing. History disagrees, and history is what gets cloned.

# grep only sees today's files. Search every commit object instead:
git rev-list --all | xargs -n 100 git grep -n -I -E 'AKIA[0-9A-Z]{16}'

9de2b31c47a0f85d6e29b3140ca7f82d51e6b904:config/prod.yaml:12:  aws_access_key_id: AKIAIOSFODNN7EXAMPLE
9de2b31c47a0f85d6e29b3140ca7f82d51e6b904:config/prod.yaml:13:  aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
7c41f0e9d2a84b6135ce07f9b3a1d85206ef4c73:config/prod.yaml:12:  aws_access_key_id: AKIAIOSFODNN7EXAMPLE
7c41f0e9d2a84b6135ce07f9b3a1d85206ef4c73:config/prod.yaml:13:  aws_secret_access_key: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

git rev-list --all prints the full hash of every commit in the repository, and git grep will take those hashes as places to search, so the pipeline reads every historical version of every file. Four hits, two commits, in a file that is spotless today. The next question is when the key arrived and when somebody believed they had removed it. Git has a tool for exactly that. The -S flag is called the pickaxe, and it behaves like asking a librarian for only the days when the number of copies of a book on the shelf changed. It lists the commits where the count of your string went up or down, which hands you both ends of the story.

git log --all --oneline -S 'AKIAIOSFODNN7EXAMPLE' -- config/prod.yaml

4b8a0c2 chore: move creds to env vars
7c41f0e feat: wire payments service to S3

The key went in at 7c41f0e and came out at 4b8a0c2. Every commit between those two carries it, and 4b8a0c2 is the commit people point at when they say the problem was fixed. By then the ledger has already been copied: onto laptops, into CI caches, into the backup system, into whatever code search tool your company runs. On GitHub, every fork of a repository shares a single object store, so a commit that no branch points at any more can still be pulled up by its SHA (Secure Hash Algorithm, the long hex string that names a commit) through any repository in the fork network. Rewriting history with git filter-repo or BFG changes every commit hash from the edit onward, breaks open branches and pull requests for everyone else, and still cannot reach the copies that already left the building.

Encoding is not encryption

The usual first attempt is a Kubernetes Secret, the small YAML object that hands credentials to a running application. Its values look scrambled. They are not. Base64 is an alphabet swap, in the same way Morse code is an alphabet swap: anyone holding the chart reads it straight back, and the chart is printed in every reference manual on earth. No key is involved at any point, so there is nothing to protect and nothing to steal. Here is kubectl, the command line client for Kubernetes, printing a Secret without sending it anywhere (--dry-run=client means build it locally and stop).

kubectl create secret generic payments-api \
  --from-literal=DB_PASSWORD='pr0d-Pa55!' \
  --dry-run=client -o yaml

apiVersion: v1
data:
  DB_PASSWORD: cHIwZC1QYTU1IQ==
kind: Secret
metadata:
  creationTimestamp: null
  name: payments-api
type: Opaque

echo 'cHIwZC1QYTU1IQ==' | base64 -d

pr0d-Pa55!

One command, no key, no expertise required. Committing that manifest commits the password with one extra step in front of it, and it survives in history exactly as long as the plaintext would. The same goes for a Kustomize secretGenerator (a Kubernetes templating helper that builds Secret objects for you) pointed at a plaintext file sitting next to it in the repo. Encoding hides a secret from a bored reader. Encryption hides it from someone who wants it.

What SOPS actually does to your file

Here is the shape of it. Put every secret value in one strongbox and lock it with a single freshly cut key. Copy that little key, seal each copy in an envelope addressed to one named holder, and tape the envelopes to the outside of the box. The box can now travel anywhere and sit anywhere in the open. Only someone who can open one of the envelopes can open the box. That pattern is called envelope encryption, and it is what SOPS performs every time you encrypt a file.

Concretely: SOPS generates a random 256-bit data key for the file. Every value it has been told to protect is encrypted with AES-256-GCM (Advanced Encryption Standard, 256-bit key, Galois/Counter Mode, a cipher that hides data and detects tampering with it at the same time) under that data key. The data key is then wrapped once per recipient you named. A recipient can be an age public key, an AWS KMS (Amazon Web Services Key Management Service) key, a GCP KMS (Google Cloud Platform) key, an Azure Key Vault key, a PGP (Pretty Good Privacy) key, or several of those at once. For the cloud backends, SOPS sends the data key to the provider and stores whatever comes back, so the wrapping key itself never leaves the provider. All the wrapped copies land in a sops: block at the bottom of the same file. The file always carries its own list of who can open it.

Two details make the result awkward to tamper with. Each value is encrypted with its own path in the document mixed into the calculation as additional authenticated data, meaning context that stays readable but is still covered by the tamper check. That works like a hotel key card cut for one specific door: a ciphertext lifted out of DB_PASSWORD and pasted under API_TOKEN will not open there. SOPS also stores a MAC (Message Authentication Code, a tamper seal computed across all the values in the file), so adding, deleting, or reordering entries gets caught. Decryption fails loudly instead of quietly handing back a document somebody else rearranged. Note what the seal covers: the values, not the key names.

# One-time: create an age identity. age is a modern encryption tool whose
# public keys look like age1... and whose private keys look like
# AGE-SECRET-KEY-1...
mkdir -p ~/.config/sops/age
age-keygen -o ~/.config/sops/age/keys.txt

Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

That path is where SOPS looks by default on Linux: $XDG_CONFIG_HOME/sops/age/keys.txt, where $XDG_CONFIG_HOME is the standard config directory and falls back to ~/.config. On macOS it looks under ~/Library/Application Support/sops/age/. Two environment variables feed it as well. SOPS_AGE_KEY_FILE points at a different file, and SOPS_AGE_KEY carries the key material itself, which is how a CI job gets a key without ever writing one to disk. age-keygen creates the file with 0600 permissions, readable by your user alone. The age1... public key is the address you encrypt to and is safe to paste into a README. The AGE-SECRET-KEY-1... line inside that file is the thing an attacker actually wants, and it is a plain text file on a laptop. Sit with that for a second, because it is the whole security model.

sops encrypt --age age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p \
  --encrypted-regex '^(data|stringData)$' \
  k8s/payments/secret.yaml > k8s/payments/secret.enc.yaml
 
rm k8s/payments/secret.yaml               # the plaintext copy has no business existing
sops filestatus k8s/payments/secret.enc.yaml

{"encrypted":true}

Three things in that command matter. sops encrypt, sops decrypt, and sops edit are the subcommand forms; the older sops -e and sops -d flags still work, and you will meet them in every pipeline written before 3.9. --encrypted-regex '^(data|stringData)$' limits encryption to values sitting under keys that match, because the default is to encrypt every value in the document, and a manifest whose apiVersion reads ENC[...] is useless to Kubernetes. sops filestatus answers one yes-or-no question and prints {"encrypted":true} or {"encrypted":false}, which makes it the right thing to wire into a hook. About that rm: it unlinks the file and nothing more. Overwriting tools such as shred cannot promise anything on an SSD or a copy-on-write filesystem, where the old blocks may still be sitting there under another name. Treat any plaintext secret that touched a disk as spent, and rotate it when you get the chance.

apiVersion: v1
kind: Secret
metadata:
    name: payments-api
    namespace: payments
type: Opaque
stringData:
    DB_PASSWORD: ENC[AES256_GCM,data:k99sn+BtMBPqLuM=,iv:4kx0p8yLIW3UW00GTn3Q6vmV2WpPA4ZTcWafJIKUX14=,tag:r+G2JXLhQo0du9jZQGJk4w==,type:str]
    STRIPE_KEY: ENC[AES256_GCM,data:W3BU7gunyFdXWlg=,iv:yohSrr21sU+1Yt13NsXI7l2mSSNVlxG0Oq2jV+cfy9o=,tag:DcwROnqpeRGs4Vqzp2ffwg==,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
          enc: |
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBmTXhLdlZrOGRIN0xxNXpo
            eEcwSGtQTVo4dlRhQXk1NgpMcm5UOWJ2eEhkSWZzUWNXK2ZqNGtNMlpwTnhFVXpn
            Ci0tLSBlSzJ4V2ZwQXVpTTZ0R3FMc1pkNHZOOGN5UjBiSGpFN21LMXdUdlkzUXNB
            M2ZLdEg5cldiTjRxWFkwdkFqUzZ4RThtUDdjVWlHMXpEbjVMYlI4dEsyaFdlOXBY
            dEs0bVJ6WDhnQ2ZOMHFKcFY3aXlCbDNzVW9BZTVIdE0yeFB3RGtOaV==
    lastmodified: "2026-07-22T09:14:02Z"
    mac: ENC[AES256_GCM,data:Gsq8YMVd4V4w9zsIDuccun74jl5Q4TUSJf8N2tmp7TyDabYCRmfxmAazonesfaQzzM0HX7XpBHLwcSO5wTcF0YwwKbPsVrmhm9E4cRC4tp4kbA7oB7Q7nPZIQM9Y4dQDtNrp9WnMAeVCMuZIbROUkCnQreFLx31ZUamHqZWwayO7,iv:eKlEx+A18jlhP02gAZBZekdvFa/JmPDGUVCNRvn++d4=,tag:VYF27ZaVeKZjJ5Tav2/hlQ==,type:str]
    encrypted_regex: ^(data|stringData)$
    version: 3.13.2

Read that file the way an attacker would. The key names are in the clear. DB_PASSWORD and STRIPE_KEY say precisely what is worth stealing and roughly what it opens. The structure, the namespace, the number of entries, the recipient list, and the time of the last change are all readable. The values are not. SOPS rewrites the document through its own YAML writer, so expect a formatting change on the first encrypt: four-space indent by default, which --indent 2 changes if your linter objects. Comments living inside an encrypted branch are treated as values and encrypted too, so they come back as #ENC[...] lines. Getting the decrypted values into a running cluster is a separate job, handled by Flux's built-in SOPS decryption, an Argo CD plugin, or External Secrets Operator, and that is a later lesson.

Typing --age and --encrypted-regex correctly every single time is a bet you will eventually lose. Write the rules down instead, in a .sops.yaml file at the top of the repository, and SOPS picks them up based on the path of the file you are encrypting.

# First matching rule wins, so put the narrow paths first.
creation_rules:
  - path_regex: k8s/prod/.*\.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
 
  - path_regex: k8s/staging/.*\.enc\.yaml$
    encrypted_regex: ^(data|stringData)$
    age: age1lggyhqrw2nlhcxprm67z43rta597azn8gknawjehu9d9dl0jq3yqqvfafg

With that in place, sops encrypt k8s/prod/payments.enc.yaml picks the right recipient on its own, and staging files can never be encrypted to the production key by accident. One trap comes with it. Creation rules apply when a file is first encrypted and never again. Change the recipient list later and every file already in the repo keeps the old one until you run sops updatekeys on it, which re-wraps the existing data key for the new list. People forget this on the day a colleague leaves.

That readable structure is the reason SOPS survives code review at all. Rotate the database password, re-encrypt, and the change is legible without a single secret appearing on screen.

git diff HEAD~1 -- k8s/payments/secret.enc.yaml | grep -E '^[-+]' | cut -c1-88

--- a/k8s/payments/secret.enc.yaml
+++ b/k8s/payments/secret.enc.yaml
-    DB_PASSWORD: ENC[AES256_GCM,data:k99sn+BtMBPqLuM=,iv:4kx0p8yLIW3UW00GTn3Q6vmV2WpPA4
+    DB_PASSWORD: ENC[AES256_GCM,data:7rh8xQggfk084NY=,iv:mMfvC0t+ZTEKLquIOkuMgWgdBD2d+B
-    lastmodified: "2026-07-20T11:02:44Z"
+    lastmodified: "2026-07-22T09:14:02Z"
-    mac: ENC[AES256_GCM,data:Gsq8YMVd4V4w9zsIDuccun74jl5Q4TUSJf8N2tmp7TyDabYCRmfxmAazon
+    mac: ENC[AES256_GCM,data:Ba1C6zpt5eeKZimMUfKs6nAkYEv+TSRDmO9ONVUY0lYqWX5ZPJZaxSFl9P

A reviewer can see that the database password changed, that STRIPE_KEY did not, and that nobody quietly added a recipient. That readability is deliberate: while it works, SOPS keeps a note of the random starter value (the initialisation vector) used for each value it decrypted, and reuses it for any value whose plaintext you did not touch. Untouched entries come out byte for byte identical. You only get that if you go through SOPS. Run sops decrypt -i and later sops encrypt -i and you have two separate runs with no shared note, so every value gets a fresh vector, every line changes, and your reviewer faces a wall of new ciphertext with nothing to compare against. The lastmodified and mac lines change on every edit by design, since the seal covers the whole document.

Prove the boundary holds

Do not take the word "encrypted" on trust. Reproduce the attacker's position directly: a full clone, no key, one command. The trick is making the machine genuinely keyless for the length of that command, which is harder than it looks, because SOPS collects age identities from several places at once. It reads any SSH key you have, the SOPS_AGE_KEY variable, the file named by SOPS_AGE_KEY_FILE, the output of SOPS_AGE_KEY_CMD, and the default file under your config directory. Pointing one of those at /dev/null proves nothing, because the others still answer. Wipe the environment instead.

git clone --quiet https://git.internal/payments.git /tmp/stolen
cd /tmp/stolen
 
# env -i clears every variable; the empty HOME means no default keys.txt exists.
# PATH is expanded by your shell before env runs, so sops is still findable.
mkdir -p /tmp/nokey
env -i HOME=/tmp/nokey PATH="$PATH" sops decrypt k8s/payments/secret.enc.yaml; echo "exit=$?"

Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p: FAILED
    - | failed to load age identities
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.
exit=128

That failure is the security property, printed by the tool itself. Look at what the message hands over: the exact age public key that would have worked, which tells an attacker whose laptop to target next. It is the same recipient already written in the file, so nothing new leaks here, but it is a reminder that the recipient list is public information and should be read as a target list. Then look at the exit code. SOPS returns 128 for CouldNotRetrieveKey, so a smoke test can assert on a number instead of scraping message text that may be reworded in the next release. If this ever succeeds, a private key is sitting somewhere it should not be, and you found out for free.

Stop the plaintext copy from ever being staged

The failure that undoes all of this is boring and common: the plaintext gets committed anyway, under a filename that claims otherwise. sops decrypt --in-place secret.enc.yaml overwrites the encrypted file with plaintext and keeps the name. Somebody runs it to check a value, gets pulled into a meeting, and git add . finishes the job. Reviewers see .enc. in the filename and skim past. Guard it mechanically rather than relying on anyone's attention at 6pm on a Friday.

#!/usr/bin/env bash
# Refuse to commit an unencrypted *.enc.{yaml,yml,json,env}.
# Checks the STAGED blob, not the working copy: they can differ.
set -euo pipefail
 
status=0
 
while IFS= read -r -d '' f; do
  case "$f" in
    *.enc.yaml|*.enc.yml|*.enc.json|*.enc.env) ;;
    *) continue ;;
  esac
 
  tmp=$(mktemp --suffix=".${f##*.}")   # keep the extension so SOPS picks a parser
  git show ":$f" > "$tmp"
  if [ "$(sops filestatus "$tmp" 2>/dev/null | jq -r '.encrypted')" != "true" ]; then
    echo "BLOCKED: $f is staged in plaintext"
    status=1
  fi
  rm -f "$tmp"
done < <(git diff --cached --name-only --diff-filter=ACM -z)
 
exit $status

chmod +x .git/hooks/pre-commit
 
sops decrypt --in-place k8s/payments/secret.enc.yaml   # the mistake
git add k8s/payments/secret.enc.yaml
git commit -m 'bump replica count'

BLOCKED: k8s/payments/secret.enc.yaml is staged in plaintext

The commit never happens, and the fix is one sops encrypt away. Three details keep that hook honest. It reads the staged blob with git show ":$f" rather than the file on disk, because you can stage one version and then carry on editing another. It loops with a process substitution rather than a pipe, so the status variable survives instead of dying in a subshell. And it fails closed: if sops errors out or jq (a command line JSON reader) finds nothing, the comparison is not true and the commit is refused. Hooks in .git/hooks are local and never travel with a clone. You can point core.hooksPath at a directory inside the repo so everyone gets the same hooks, but nothing forces a teammate to enable it, so run the same sops filestatus sweep as a CI job over every matching file in the tree and point a secret scanner such as gitleaks or trufflehog at the full history on a schedule. The hook catches the mistake in the two seconds where it is still free. CI catches the colleague who never installed the hook.

The honest trade-off

SOPS does not delete the risk. It moves it, from "protect a secret that gets copied into every clone" to "protect one key that never has to be copied anywhere". That is a far better problem to own, and it arrives with two sharp edges worth understanding before your first encrypted file merges.

The first edge is that ciphertext is as permanent as plaintext. Every encrypted version of that file lives in history forever, and all of it opens for whoever eventually holds the key. An age private key that leaks out of a laptop backup in 2028 opens the 2026 commits too, including the passwords you rotated along the way. sops rotate --in-place generates a fresh data key and re-encrypts the current file, which limits what a future leak of that one data key reaches, and it cannot un-publish ciphertext already sitting in your history. Rotating also leaves the recipient list untouched, so if the key holder is the problem, swap the list at the same time with sops rotate --rm-age <old> --add-age <new>, or edit .sops.yaml and run sops updatekeys. After a key compromise, rotate the underlying secrets, not the file.

The second edge is blast radius. Whoever holds a key reads everything that key covers, so one team-wide age key spanning staging and production means a contractor with staging access can read your production database password. Scope keys the way you scope access: separate recipients per environment, and separate keys for humans and for the deploy controller. This is where the cloud backends earn their price. An age private key is a file that leaves no trace when it is used, while an AWS KMS key writes a CloudTrail record (the audit log of every API call in an AWS account) for each Decrypt, with the calling identity attached, and access is withdrawn by editing an IAM (Identity and Access Management, the AWS permissions system) policy rather than by chasing every copy of a file across laptops.

Make the clone-and-fail test a habit after any change to who can decrypt: a recipient added, a key rotated, a contractor offboarded. Thirty seconds, no cost, and it is the only check in your toolkit that behaves exactly the way an attacker will.

Try this

Run git rev-list --all | xargs -n 100 git grep -n -I -E 'AKIA[0-9A-Z]{16}' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: scrubbing history is not revocation. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related