The secrets-management landscape

SOPS, Sealed Secrets, ESO, Vault.

Four tools show up in almost every argument about secrets in a Git repository: SOPS, Sealed Secrets, External Secrets Operator, and Vault. They get pitched as rivals. They mostly are not. Each one answers a different version of the same question, and the question is where the true copy of the password lives and who holds the thing that opens it.

Say you need to get a spare house key to a friend. You could lock it in a strongbox, keep the only strongbox key on your own keyring, and bolt the strongbox to a public noticeboard. That is SOPS (Secrets OPerationS). It started at Mozilla in 2015, was donated to the CNCF (Cloud Native Computing Foundation, the same group that looks after Kubernetes) as a Sandbox project in 2023, and now lives in the getsops organisation on GitHub. Plenty of blog posts still call it Mozilla SOPS. The name moved on.

Or you could drop the key through the mail slot in your friend's front door. Anyone walking past can post something through that slot. Only the person inside can take it out. That is Sealed Secrets. Or leave the key with the building concierge and pin up a ticket reading "front desk, drawer two." That is ESO (External Secrets Operator, a program that runs inside your cluster and fetches values from a secrets service on your behalf). Or have the concierge cut your friend a fresh key that stops working at midnight. That is Vault.

Two questions separate these four, and everything else is detail. First, does the secret of record, the copy every other copy is derived from, live in Git or somewhere else? Second, who physically holds the private half of the key that opens the ciphertext: you, the cluster, or a service?

Encrypt in Git or Sync From a Store

SOPS and Sealed Secrets are encrypt-in-Git tools. The ciphertext is an ordinary file. You commit it, review it, tag it, revert it. Git is the source of truth. That buys you something people underrate: secrets are versioned right beside the code that uses them, so rolling back a bad release rolls back its configuration and its credentials in the same commit. ESO and Vault work the other way round. Git holds a pointer, a short manifest naming a backend and a path. The real value sits in a dedicated service, and a controller (a program that loops forever, comparing what you asked for against what exists, then closing the gap) pulls it into the cluster.

SOPS belongs in the first camp because of exactly what it encrypts: the values, never the keys. Structure stays readable. A reviewer can see that db.password changed without ever seeing the password, and the diff stays meaningful instead of collapsing into one enormous blob. The metadata block SOPS appends to the bottom of the file is readable too. Treat it as the shipping label on a locked crate. It states in plain text which recipients are able to open the thing.

# the whole committed file, exactly as a teammate sees it
$ git show HEAD:secrets/prod.enc.yaml
 
# sops 3.9 added a machine-readable check. Useful as a CI guard against
# somebody committing the plaintext version by mistake.
$ sops filestatus secrets/prod.enc.yaml

db:
    host: postgres.prod.internal
    username: payments_rw
    password: ENC[AES256_GCM,data:PLWqgLPGsL9v,iv:b4D9OxqRuo4ylyne78eZpPvCXlvXrl1vytwW0gyH/XU=,tag:MPdKzzjlgT0fqKMXiHN4PA==,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
          enc: |
            -----BEGIN AGE ENCRYPTED FILE-----
            YWdlLWVuY3J5cHRpb24ub3JnL3YxCi0+IFgyNTUxOSBnU0dCYTErZVJrTEY2c0V3
            RzhybFBGclBBcmptUWNoMnE0SU82M0RTUUhNCkVMZGp6bm10M2g2WE1MQ2N5L2t6
            VUJIRVA0V29nNGk4bHE3S1RPa1ZoMUkKLS0tIE9kdHF1dWVGQWc3N1BNSUxBaXBt
            NXJ0RkkrWWE1NGxIbmNNMkhISHpHSXMKdw8bXdJB98CKOjH5En2uqEH6huAiPyDc
            /Un33mjeQTL8QHTTUyJ+deYR/eyOQhfSQJsTH9O6E6+VVfiBno7qLA==
            -----END AGE ENCRYPTED FILE-----
    lastmodified: "2026-07-19T08:41:52Z"
    mac: ENC[AES256_GCM,data:GEr11LhDe8MN1TrB71AuxFek0q5fP0ufAlmdEKNiS9iQ3v3Mr0+43ngT6varK34Zx21AbTKwuDYZeatWgfGqfxcnpnb8P46ISGrPpnm2wuqmenyqzrwJWX5BocSr4t5eZbN1Cbkqs2Iibxjiv9itusGbv5ZXDtjtlvX4qwcsrLI=,iv:gvwT8/QuMRlWvtiQlrNjvvHUDriqgE9eVQz4HE9yRwA=,tag:zEgtg0nzlVaxZxKfpHqvKg==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2
 
{"encrypted":true}

Read that file top to bottom and you can see the whole design. host and username are in the clear because SOPS was told to encrypt values, and it left every key name alone so the structure survives review. The password is one line of ciphertext with its own initialisation vector and authentication tag, so editing one field does not disturb any other. The mac at the bottom is a signature over the whole document, which is what stops somebody swapping in the ciphertext from last year's file.

That trailer is the honest price of the model. Anyone who can read the repository learns which key opens it. With age you leak a public key, harmless on its own, but it gives an attacker a name to hunt for. With AWS KMS, GCP KMS, or Azure Key Vault (KMS stands for Key Management Service, a cloud service that holds encryption keys and does the encrypting for you without ever handing the key over) you leak the full key identifier. That one string carries your account number and the exact key to phish for, assume a role against, or probe for a sloppy policy. Encrypt-in-Git publishes your key inventory. Always, by design.

Sealed Secrets: The Cluster Holds the Key

Sealed Secrets is that mail slot, built out of real parts. A controller you install into the cluster generates an RSA key pair on first start (RSA is public-key cryptography: a matched pair where one half locks and only the other half opens, and 4096 bits is the default size here). The public half goes to anyone who asks for it. The private half is written to a Secret in the controller's own namespace and never leaves. The kubeseal command runs on your laptop, needs nothing but that public certificate, and produces a SealedSecret object safe to commit to a public repository. A contributor with zero cluster credentials can still seal a value for production. On a big team, that is a genuinely nice property.

# grab the controller's public certificate once; keep it in the repo if you like
$ kubeseal --fetch-cert \
    --controller-name sealed-secrets --controller-namespace kube-system \
    > pub-cert.pem
 
# seal offline against that certificate: no cluster access needed from here on
$ kubectl create secret generic db-creds -n apps \
    --from-literal=password='pr0d-p@ss' \
    --dry-run=client -o yaml \
  | kubeseal --cert pub-cert.pem --format yaml > db-creds-sealed.yaml
 
$ cat db-creds-sealed.yaml

apiVersion: bitnami.com/v1alpha1
kind: SealedSecret
metadata:
  creationTimestamp: null
  name: db-creds
  namespace: apps
spec:
  encryptedData:
    password: AgA/9L0b3ipFjUrsSiXSyChab8DnhrEU3g78kFM4qNt5QWhXJW2SIr8H0NXkOWDQ[...720 characters in total...]
  template:
    metadata:
      creationTimestamp: null
      name: db-creds
      namespace: apps
    type: Opaque

Under that base64 string sits a two-layer scheme, the same shape your browser uses to open an HTTPS session. kubeseal invents a random 256-bit AES key (AES-GCM is fast symmetric encryption: the same key closes the box and opens it again, and the GCM half also detects tampering), encrypts your value with it, then wraps that single-use key in the controller's RSA public key using OAEP padding (Optimal Asymmetric Encryption Padding, the safe way to encrypt something small with RSA). The finished blob is the length of the RSA ciphertext, then the RSA ciphertext, then the AES ciphertext. That layout is why every sealed value in your repository starts with the letters Ag: with a 4096-bit key, the length prefix is always 512. It is also why sealing nine characters produces 720 characters of base64.

The part worth remembering is what OAEP gets handed as a label. RSA-OAEP takes an optional label that must match byte for byte at decryption time, and Sealed Secrets fills it with the string namespace/name. Change either half and the mathematics stops working. That is what the default strict scope means in practice. Rename the object or move it to another namespace and the controller cannot open it, however good your reason.

kubeseal --scope namespace-wide drops the name from that label, so the ciphertext works under any name in the same namespace. kubeseal --scope cluster-wide empties the label completely, so the object unseals anywhere in the cluster. Both are conveniences with a real bill attached. A cluster-wide SealedSecret can be opened by anyone who can create an object in a namespace they control, which on a shared cluster usually means every team in the company. They paste your ciphertext into their own namespace, the controller decrypts it there, and they read the plaintext out of the Secret it produces. Keep strict scope unless you can say the reason out loud.

$ kubectl apply -f db-creds-sealed.yaml
 
# check BOTH objects: the sealed one synced, and the plain Secret it produced
$ kubectl get sealedsecret,secret db-creds -n apps

sealedsecret.bitnami.com/db-creds created
 
NAME                                STATUS   SYNCED   AGE
sealedsecret.bitnami.com/db-creds            True     4s
 
NAME              TYPE     DATA   AGE
secret/db-creds   Opaque   1      4s

External Secrets Operator: Git Holds a Claim Ticket

ESO turns the problem inside out. Nothing encrypted goes into Git at all. You commit the claim ticket: an ExternalSecret manifest saying which backend to ask and which path to ask for. The operator authenticates to that backend (Vault, AWS Secrets Manager, Google Secret Manager, Azure Key Vault, 1Password, and a long list beyond those), reads the value on a schedule you set, and writes an ordinary Kubernetes Secret. Two objects split the work. A SecretStore describes how to reach one backend. An ExternalSecret describes what to pull out of it.

Check the apiVersion line before you copy anything out of an older tutorial. ESO promoted this API to external-secrets.io/v1 in release 0.16.0, which also deleted v1alpha1 and the conversion webhooks that used to paper over the difference. Current versions no longer serve v1beta1 either: the custom resource definition still lists it, marked served: false. An old manifest will not quietly keep working. It dies at apply time with no matches for kind "ExternalSecret" in version "external-secrets.io/v1beta1".

# 1. how to reach the backend (namespaced; ClusterSecretStore is the cluster-wide twin)
apiVersion: external-secrets.io/v1
kind: SecretStore
metadata:
  name: vault-backend
  namespace: apps
spec:
  provider:
    vault:
      server: https://vault.internal:8200
      path: secret                 # the KV v2 mount (key-value store, version 2)
      version: v2
      auth:
        kubernetes:
          mountPath: kubernetes    # where Vault's Kubernetes auth method is mounted
          role: apps-reader        # Vault role bound to this ServiceAccount
          serviceAccountRef:
            name: eso-sa           # no static token: the SA token gets exchanged
---
# 2. what to pull, and what Secret to write
apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: db-creds
  namespace: apps
spec:
  refreshInterval: 1h              # re-read the backend this often
  secretStoreRef:
    name: vault-backend
    kind: SecretStore
  target:
    name: db-creds                 # the Kubernetes Secret ESO creates
    creationPolicy: Owner          # default: ESO owns it, and deletes it with this object
  data:
    - secretKey: password          # key inside the resulting Secret
      remoteRef:
        key: apps/db               # path in the backend
        property: password         # field within that entry

$ kubectl get externalsecret db-creds -n apps
 
$ kubectl describe externalsecret db-creds -n apps | tail -4

NAME       STORETYPE     STORE           REFRESH INTERVAL   STATUS         READY   LAST SYNC
db-creds   SecretStore   vault-backend   1h                 SecretSynced   True    2m14s
 
Events:
  Type    Reason   Age    From               Message
  ----    ------   ----   ----               -------
  Normal  Updated  2m14s  external-secrets   Updated Secret

READY True is the check that matters, and it is the first place to look when an application cannot log in. SecretSynced means ESO reached the backend and wrote the Secret. SecretSyncedError with READY False means it could not, and kubectl describe names the part that failed: the Vault role rejected the ServiceAccount, the path was wrong, or the property you asked for is not in that entry. Rotation is the payoff for all this machinery. Change the value in the backend and ESO writes the new one within refreshInterval, with no commit, no review, no merge. Doing the same under encrypt-in-Git means re-encrypting the file, opening a pull request, and waiting for a reconcile, all to rotate one password.

One trap catches everybody exactly once. ESO updating the Secret does not update the running pod. If the container reads the value through envFrom or env.valueFrom.secretKeyRef, that environment was baked in when the process started and will not change while the process lives. You need kubectl rollout restart deployment/payments, or a helper such as Reloader that watches the Secret and restarts the workload for you. A Secret mounted as a volume does refresh in place, on the kubelet's sync schedule rather than instantly, so give it about a minute. There is an exception inside that exception: a volume mounted with subPath never refreshes at all.

The costs are real ones. You run another controller and a backend, and both have to be reachable at the moment a workload starts, so an outage in the secret store becomes an outage in your deploys. creationPolicy: Owner means the generated Secret belongs to the ExternalSecret, so deleting the manifest deletes the live Secret with it. And a chicken-and-egg problem hides inside every SecretStore: something has to prove the cluster's identity to the backend before any of this works. The manifest above dodges it with Vault's Kubernetes auth method, which trades a ServiceAccount token (the identity Kubernetes already hands to every pod) for a Vault token, so no static credential exists anywhere. Backends without that option need a bootstrap credential, and getting that one credential into the cluster safely is precisely the job SOPS and Sealed Secrets do well.

Vault: Credentials That Expire on Their Own

A hotel keycard is not a copy of the room key. It gets minted at check-in, works for your stay, and is dead by Tuesday noon whether you hand it back or not. Vault's dynamic secrets behave the same way. Ask its database secrets engine for a credential and Vault logs into the database as an administrator, creates a brand new user for that one request with a time limit attached, and hands it over.

$ vault read database/creds/payments-ro

Key                Value
---                -----
lease_id           database/creds/payments-ro/SDn23CEON8jRGVvGGoe4YkqR
lease_duration     1h
lease_renewable    true
password           iTUlSds6QDwbcpaR7s3c
username           v-kubernet-payments-Qh7bK2mXpR4tLnGh-1784686472

That username is generated, not chosen, and its shape tells you where it came from: who asked, which role they used, twenty random characters, and the unix timestamp of the request. No static password exists anywhere to encrypt, commit, leak, or rotate, because none exists until somebody asks and none survives the hour.

lease_id is the handle that makes this operationally different from everything above. A lease is the front desk's record that your card exists and when it dies. One command, vault lease revoke -prefix database/creds/payments-ro, kills every credential that role ever issued, right now. Try the equivalent with a password living in a SOPS file: rotate it in the database, re-encrypt, commit, then hope every consumer picks up the change before something breaks at 2am. Vault can also skip Kubernetes Secrets altogether, writing values as files into the pod under /vault/secrets/ through its Agent Injector, so the value never lands in the cluster's own database. The bill for all of this is a highly available service with storage, unseal keys, policies, audit devices, and an upgrade cadence somebody owns.

What a Leaked Repo Gives an Attacker

Assume the repository goes public tomorrow by accident, because that is the failure everyone is really buying insurance against. Under SOPS an attacker gets ciphertext, every key name, and the sops trailer naming the age recipients or the cloud key identifier, which tells them your account number and exactly which key to go after. Under Sealed Secrets they get ciphertext bound to a namespace and a name, and nothing that identifies the key beyond the cluster it belongs to. Under ESO they get no ciphertext at all, and instead a precise map of your secret store: mount paths, role names, and which application reads which entry. Every model leaks structure. None of them leak the value.

The leak is the smaller problem. What should decide where a high-value credential lives is what happens afterwards. Decryption with SOPS or Sealed Secrets happens offline. Somebody holding the age identity can decrypt every version of every file in the whole history, on a laptop, on a plane, and no server anywhere records that it happened. Vault and the cloud KMS backends log every read with an identity and a timestamp, and both can cut access off after the fact. If you cannot answer "who read this production database password, and when," offline decryption is the reason, and a cloud KMS key is the cheapest way to buy that answer back without leaving SOPS.

Choosing, and Combining

Reach for SOPS when you want one repository, no extra controller, and full ownership of the key material, and when you can live with decryption being offline and unlogged. Check how your GitOps controller (Flux or Argo CD, the agent that reads manifests out of Git and applies them to the cluster) handles it before you commit to the idea. Flux decrypts SOPS files natively, with spec.decryption.provider: sops on the Kustomization and a Secret holding the age identity. Argo CD has nothing built in and needs a config management plugin such as KSOPS bolted on.

Reach for Sealed Secrets when you want the same commit-the-ciphertext story but would rather run a Kubernetes controller than manage cloud key grants, and when contributors should be able to seal production values without any cluster access. Reach for ESO when a central store already exists and several clusters or teams have to read the same values consistently, and when rotating without a pull request is worth a second controller. Reach for Vault directly when the credential should not exist at all between uses.

Most mature platforms run more than one of these, layered by what each is good at. A single age key held by the platform team encrypts the handful of bootstrap values in the cluster's own repository: the Vault address, the certificate authority bundle, the credential that lets ESO authenticate the first time. After that, application passwords never touch Git again, and the database credential the payments service uses gets minted at pod start and revoked an hour later. To find out which model you are running today, open your cluster repository and search for ENC[, kind: SealedSecret, and kind: ExternalSecret in one pass. Most teams find all three, and nobody remembers deciding on any of them.

Try this

Run git show HEAD:secrets/prod.enc.yaml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: back up the sealing key or the ciphertext turns to scrap. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related