SOPS in GitOps

Flux & Argo decrypt at apply.

A GitOps repository is the master binder for a system. Every deployment, every config file, every firewall rule, written down so that anyone can rebuild the lot starting from an empty cluster. Then you reach the pages with the database passwords on them and the binder falls apart, because everyone who can read the repo can read those pages too. SOPS lets you keep those pages in the binder, written in cipher. One controller inside the cluster holds the codebook, and it does the decoding at the last possible moment, on the way to the cluster.

Two words to pin down before the commands start. GitOps means a controller, a program that runs inside your cluster in a loop and never stops, keeps pulling a Git repository and reshaping the live cluster until it matches what the repo says. The repo is the source of truth, and nobody applies changes from a laptop. SOPS (Secrets OPerationS, born at Mozilla, now maintained in the getsops organisation and donated to the CNCF, the Cloud Native Computing Foundation that also hosts Kubernetes) is a file encryptor that understands structure. Point it at a YAML file (YAML is the indented text format Kubernetes manifests are written in, and a manifest is one file describing one object) and it encrypts the values while leaving the keys, the shape of the document and the comments in plain sight. That second half is what makes the whole pattern work. kustomize, the tool that assembles a directory of manifests into the final set of objects, can still see a Secret called db-creds bound for the apps namespace. It sees noise only where the password should be.

What Flux Does During One Reconcile

Flux splits the work between two controllers. The source-controller fetches the Git commit and stores it as a tarball, like a courier leaving a sealed parcel at the door. The kustomize-controller unpacks that parcel into a temporary directory. If the Flux Kustomization object carries a spec.decryption section, the controller then goes through the files that kustomization actually references (resources, patches, and any file a secretGenerator reads), picks out the ones carrying SOPS metadata, and decrypts them in place. Only after that does it run kustomize build. The rendered output is applied to the API server, and the temporary directory is deleted.

Two useful things follow from that ordering. Plaintext lives only in the controller's own filesystem and memory, for the second or two a build takes, and it never travels back toward Git. And because decryption happens before the build rather than after it, a secretGenerator pointed at an encrypted .env file behaves exactly like an encrypted Secret manifest. The controller does not care which shape you chose. It cares that the file parsed as SOPS-encrypted and that it holds a key which fits.

Give The Controller A Key

age is the backend most GitOps setups start with, and the easiest one to reason about. Picture a padlock you hand out freely: anyone can snap it shut, only your key opens it. The padlock is the recipient, a public string starting age1..., safe to paste into a repo. The key is the identity, starting AGE-SECRET-KEY-1..., and it is the only thing that decrypts. Generate one identity per cluster, never one for the whole fleet. That single choice decides whether a leaked key costs you one environment or all of them.

# One identity per cluster. -o writes the private half to a file
# (age-keygen already creates it mode 0600) and prints only the
# public recipient to the terminal.
age-keygen -o prod.agekey

Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Now hand the private half to Flux, and to nothing else. The kustomize-controller does not try every value in the Secret hoping one works. It picks key material by filename suffix, so the name you give the file inside the Secret matters more than the name of the Secret itself: .agekey for age identities, .asc for PGP (Pretty Good Privacy, the veteran email encryption standard) private keys, and fixed names like sops.aws-kms, sops.gcp-kms, sops.azure-kv or sops.vault-token for cloud and Vault credentials. Get the suffix wrong and the entry is ignored, with nothing in the logs that says so plainly.

# The Secret must live in the same namespace as the Kustomization
# that references it. The .agekey suffix is what the controller matches on.
kubectl create secret generic sops-age \
  --namespace=flux-system \
  --from-file=age.agekey=prod.agekey

secret/sops-age created

That Secret is now the most powerful object in the cluster, and you should feel slightly uncomfortable about it. Keep prod.agekey somewhere durable and offline as well, because Git holds only ciphertext and the cluster holds a single copy of the key. Lose the cluster and the laptop on the same bad afternoon and every secret in that repo is unreadable forever, including the ones sitting in commits from two years ago.

Encrypt To The Right Recipient, Automatically

Nobody should have to remember which key belongs to which environment at 2am. .sops.yaml at the repo root is a mailroom sorting rule: post goes into a bag based on the address printed on it, not on who happens to be on shift. The file holds creation_rules, a list read from top to bottom, and SOPS uses the first rule whose path_regex (a pattern matched against the file's path) fits the file you are encrypting. Give prod and staging different recipients and the path now picks the key. Encrypt something into the prod directory by mistake and the prod cluster is the only thing that can open it, which is the failure mode you want. The commented rule at the bottom swaps age for AWS KMS (Key Management Service, Amazon's hosted key vault), where the private half never leaves Amazon, access is granted through IAM (Identity and Access Management, the cloud's permission system), and every decrypt lands in the CloudTrail audit log.

# Repo root. Read on every `sops encrypt` and `sops updatekeys`.
# First matching rule wins, so order matters.
creation_rules:
  - path_regex: clusters/prod/.*\.enc\.ya?ml$
    encrypted_regex: ^(data|stringData)$
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
 
  - path_regex: clusters/staging/.*\.enc\.ya?ml$
    encrypted_regex: ^(data|stringData)$
    age: age1lggyhqrw2nlhcxprm67z43rta597azn8gknawjehu9d9dl0jq3yqqvfafg
 
  # Same idea with a cloud key: the private half never leaves KMS,
  # access is IAM-scoped, and every decrypt shows up in CloudTrail.
  # - path_regex: clusters/prod/kms/.*\.enc\.ya?ml$
  #   encrypted_regex: ^(data|stringData)$
  #   kms: arn:aws:kms:eu-west-1:111122223333:key/1a2b3c4d-5e6f-7890-abcd-ef1234567890

The encrypted_regex: ^(data|stringData)$ line does quiet, load-bearing work. Leave it out and SOPS encrypts every value in the file, kind and metadata.name along with the password, so your Secret's name becomes a hundred-odd characters of ENC[...]. Kubernetes names have to be short lowercase DNS-style labels (letters, digits and dashes, the same rules as a hostname), so that object stops being valid. Flux itself would survive this, because it decrypts before it builds. Nothing else in the chain would: a kustomize build in CI (continuous integration, the checks that run on every push), a policy scanner, a schema linter, a human reading the pull request. Restricting encryption to data and stringData keeps the file parseable by every tool that does not hold the key.

sops encrypt --in-place clusters/prod/apps/db-creds.enc.yaml
head -14 clusters/prod/apps/db-creds.enc.yaml

apiVersion: v1
kind: Secret
metadata:
    name: db-creds
    namespace: apps
type: Opaque
stringData:
    password: ENC[AES256_GCM,data:Yk9wTHZ4,iv:3rQ0uH1pLd8mA6bXk4S7fJ2vC5nZeT9wYgR1sKuIoPc=,tag:tqB7Xn1yQeV0mCkAr9dZLg==,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnMkxxZ0d5c1E4RGpObVNq

Read that the way a reviewer would. The kind, the name, the namespace and the field name password are all legible. So is the fact of the change: this commit touched the database password. What the value became is not legible. A reviewer can approve the shape of a change without ever being trusted with the secret inside it, and no opaque encrypted blob gives you that. One surprise on your first encryption of an existing file: SOPS re-emits the whole document with its own YAML writer at four-space indentation, so the diff is bigger than the change you actually made. Set stores.yaml.indent in .sops.yaml if that upsets your linter.

Wire It Into The Kustomization

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./clusters/prod/apps
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops          # sops is the only value this field accepts
    secretRef:
      name: sops-age        # namespace-local: flux-system, same as above
 
  # Flux 2.6+ can instead authenticate to AWS KMS / GCP KMS / Azure Key Vault
  # with Kubernetes workload identity, so no key material sits in-cluster.
  # Needs the ObjectLevelWorkloadIdentity feature gate on the controller:
  #   decryption:
  #     provider: sops
  #     serviceAccountName: kustomize-controller-kms

If you drive Flux from the command line instead of hand-writing YAML, the same object falls out of flux create kustomization apps --source=GitRepository/flux-system --path=./clusters/prod/apps --prune=true --interval=10m --decryption-provider=sops --decryption-secret=sops-age --export. Notice that secretRef takes a name and nothing else. There is no namespace field, deliberately, so a Kustomization living in a tenant's namespace physically cannot reach across and grab the platform team's key. That limitation is a gift, and the warning further down leans on it.

Verify It Actually Worked

Three checks, and skipping any one of them leaves you believing something untrue. Did the controller reconcile the commit at all. Does the Secret in the cluster hold the value you meant. And the one everybody forgets: is the file you pushed genuinely encrypted.

git add -A && git commit -m "rotate prod db password" && git push
flux reconcile kustomization apps --with-source
flux get kustomizations

[main 9f2a1c4] rotate prod db password
 1 file changed, 4 insertions(+), 4 deletions(-)
To github.com:acme/fleet.git
   8c1d0ab..9f2a1c4  main -> main
► annotating GitRepository flux-system in flux-system namespace
✔ GitRepository annotated
◎ waiting for GitRepository reconciliation
✔ fetched revision main@sha1:9f2a1c4b3d5e6f708192a3b4c5d6e7f809a1b2c3
► annotating Kustomization apps in flux-system namespace
✔ Kustomization annotated
◎ waiting for Kustomization reconciliation
✔ applied revision main@sha1:9f2a1c4b3d5e6f708192a3b4c5d6e7f809a1b2c3
 
NAME         REVISION                                            SUSPENDED  READY  MESSAGE
apps         main@sha1:9f2a1c4b3d5e6f708192a3b4c5d6e7f809a1b2c3  False      True   Applied revision: main@sha1:9f2a1c4b3d5e6f708192a3b4c5d6e7f809a1b2c3
flux-system  main@sha1:9f2a1c4b3d5e6f708192a3b4c5d6e7f809a1b2c3  False      True   Applied revision: main@sha1:9f2a1c4b3d5e6f708192a3b4c5d6e7f809a1b2c3

kubectl -n apps get secret db-creds -o jsonpath='{.data.password}' | base64 -d; echo

pr0d-P0stgres-Xk92

For the third check, SOPS 3.9 added sops filestatus, which answers one question in machine-readable JSON (JavaScript Object Notation, the shape machines parse most easily): is this file encrypted or not. That turns "did anyone commit a plaintext secret" from a code-review hope into a build step that fails loudly. Name encrypted files by a convention, *.enc.yaml here, so the check knows what to look at.

sops filestatus clusters/prod/apps/db-creds.enc.yaml
 
# CI guard: any *.enc.yaml that is not actually encrypted fails the build.
fail=0
for f in $(git ls-files '*.enc.yaml'); do
  sops filestatus "$f" | grep -q '"encrypted":true' \
    || { echo "PLAINTEXT: $f"; fail=1; }
done
exit "$fail"

{"encrypted":true}
PLAINTEXT: clusters/staging/apps/redis-creds.enc.yaml

Argo CD Decrypts One Step Earlier

Argo CD ships with no SOPS support of its own, and the workarounds all land in the same place. KSOPS is a kustomize generator plugin: kustomize runs the ksops binary, which decrypts the listed files and hands back plain Secret manifests. helm-secrets covers the Helm path. A sidecar CMP (Config Management Plugin, the supported way to extend Argo CD since the old plugin list inside the argocd-cm ConfigMap was deprecated in Argo CD 2.4 and later removed) covers everything else. Whichever you pick, decryption happens inside argocd-repo-server, the component that renders manifests, rather than inside the controller that applies them. Same ciphertext in Git, a different pod holding the key, a meaningfully different blast radius.

# KSOPS runs as a kustomize generator: kustomize execs the `ksops` binary,
# which decrypts the listed files and emits plain Secret manifests.
apiVersion: viaduct.ai/v1
kind: ksops
metadata:
  name: secret-generator
  annotations:
    config.kubernetes.io/function: |
      exec:
        path: ksops
files:
  - ./db-creds.enc.yaml

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization
resources:
  - deployment.yaml
generators:
  - ./secret-generator.yaml     # KSOPS produces the Secret at render time

apiVersion: v1
kind: ConfigMap
metadata:
  name: argocd-cm
  namespace: argocd
data:
  # kustomize refuses to exec a generator binary unless BOTH of these are on
  kustomize.buildOptions: --enable-alpha-plugins --enable-exec

# The age identity and the ksops binary both live in the repo-server pod.
spec:
  template:
    spec:
      volumes:
        - name: sops-age
          secret:
            secretName: sops-age          # the entry inside must be keys.txt here,
                                          # NOT age.agekey: this is the sops CLI, not Flux
      containers:
        - name: argocd-repo-server
          env:
            - name: SOPS_AGE_KEY_FILE
              value: /home/argocd/.config/sops/age/keys.txt
          volumeMounts:
            - name: sops-age
              mountPath: /home/argocd/.config/sops/age
              readOnly: true
# An initContainer must also copy the `ksops` binary onto the repo-server's PATH.

When It Breaks, It Breaks Quietly

The most common failure is a file encrypted to the wrong recipient, usually because somebody moved it between environment directories and did not re-encrypt it. Here is what that looks like on a laptop, using the staging identity against a prod file.

SOPS_AGE_KEY_FILE=./staging.agekey sops decrypt clusters/prod/apps/db-creds.enc.yaml

Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p: FAILED
    - | no identity matched any of the recipients
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

In the cluster the same problem shows up as the Kustomization going READY False with Error getting data key: 0 successful groups required, got 0 in its message. flux events --for Kustomization/apps is the quickest way to see it, and flux logs --kind=Kustomization --name=apps --level=error gives you the controller's side of the story. Now notice what does not happen: nothing crashes. Flux leaves the last successfully applied state running and keeps retrying on its interval. Your app stays up on yesterday's config while every new commit quietly fails to land. Alert on Kustomization readiness, not on pod health, or you find out three days and eleven merged pull requests later. The fix is to correct the matching creation_rule and run sops updatekeys -y clusters/prod/apps/db-creds.enc.yaml, which re-encrypts the data key to whatever recipients that rule now lists without ever exposing the value itself. Leave the -y off and it asks you to confirm, which is fine by hand and a hang in CI.

One more failure looks like corruption and is not. SOPS covers the file with a MAC (message authentication code, a fingerprint that changes the moment anything underneath it changes), computed over every value, encrypted or not. Hand-edit an encrypted file to resolve a merge conflict or bump an image tag sitting next to your Secret, and decryption then fails on a MAC mismatch even though the ciphertext itself is untouched. Edit through sops edit or sops set instead, or add mac_only_encrypted: true to the creation rule so only the encrypted values feed that fingerprint.

What This Protects, And What It Does Not

SOPS protects the repository. It does not protect the cluster. The moment that manifest is applied you have an ordinary Kubernetes Secret: base64 (an encoding, not encryption, reversible by anyone in a second) sitting in etcd, readable by anyone holding get secrets in that namespace, mountable by any pod with the right ServiceAccount. Turn on encryption at rest for etcd and be genuinely strict with RBAC (Role-Based Access Control, the Kubernetes permission system), or the ciphertext in Git is set dressing on an unlocked door. Rotation costs you something real, too. Changing one value means re-encrypting a file, opening a pull request and waiting for a merge, where External Secrets Operator (ESO, a controller that pulls values out of a vault and writes them into Kubernetes Secrets) or Vault would let you change it in one place and have controllers pick it up on their next sync. What you buy for that friction: one repository instead of two systems, no extra secret store to run and patch, no runtime dependency on that store being reachable while a cluster rebuilds itself from nothing, and a full history of who changed which secret and when, written in Git where you already look.

One test is worth running before you trust any of this. Write a throwaway Secret, encrypt it with sops encrypt --in-place (encryption needs only the age1... recipient, never the private half, so you can do this on a machine that holds no identity at all), push it, and reconcile. Then read the value back out of the cluster with kubectl -n apps get secret ... -o jsonpath and decode it. If the cluster produces correct plaintext from a file your own laptop cannot open, the key boundary sits exactly where you drew it, and you have proved it rather than assumed it.

Try this

Run age-keygen -o prod.agekey on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the controller's key decrypts your entire Git history. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related