Choosing a secrets manager

Vault, cloud-native, or SOPS — a decision tree.

In short. Choose a secrets manager: Vault, cloud-native stores, or SOPS — a practical decision tree.

Think about two ways an office controls its keys. Option one is a locked drawer: the keys sit in a box, and everyone who needs one gets the drawer's combination. Option two is a staffed key desk: you show your badge, the clerk checks a list, hands you a key, writes the checkout in a log, and can re-cut every lock overnight. A secrets manager is the key desk — a dedicated service that stores secrets encrypted, decides *per identity* who may read each one, records every read, and can rotate (replace) a secret from one central place. SOPS, from the last lesson, is a well-run locked drawer: the encryption is real, but there is no clerk and no logbook.

The failure a manager prevents is not the initial leak — earlier lessons covered logs, ps, and git history — it is what happens *after* secrets sprawl. With copies in files and env vars across twenty repos, three questions become unanswerable: who currently holds this credential, when was it last used, and how do we replace it without an outage? A secrets manager makes all three answerable, because there is exactly one authoritative copy, an audit log (a record of every access, by whom, when), and a rotation mechanism that changes the secret in one place instead of twenty.

What happens inside one

Nearly every manager uses envelope encryption: each secret is encrypted with its own *data key*, and the data keys are themselves encrypted by a *root key* held in a KMS (key management service — a hardened service whose keys never leave it) or an HSM (a physical hardware security module). Nothing touches disk in plaintext, and the only door is an API. A read follows the same path everywhere: the caller *authenticates* (an IAM role, a Kubernetes service account, an OIDC token), a *policy* is checked against the secret's path, the value is decrypted, and an audit event is written. Vault adds one extra concept: on startup its storage is sealed — unreadable — until the root key is reconstructed from key shares or fetched from a cloud KMS (auto-unseal). Vault can also mint dynamic secrets: it holds one admin credential to, say, PostgreSQL, and creates short-lived database accounts on demand, so most credentials simply expire instead of leaking.

The three families

Encrypted files in git — SOPS with age or KMS, or sealed-secrets in-cluster. No server to run, perfect fit for GitOps, and you already know it from the last lesson. The limits are structural: values are static, there is no audit of *reads* (git shows who changed a file, never who decrypted it), and revoking access means rotating keys, re-encrypting, and redeploying.

Cloud-native managers — AWS Secrets Manager, Google Secret Manager, Azure Key Vault. The provider runs them, availability is their problem, and access control rides on the IAM you already have, which is the killer feature: a pod or VM with a platform identity can read its secrets with zero stored credentials. Pricing is per secret plus per API call, and the boundary is the obvious one — they live in one cloud.

# Create a secret (one JSON blob per logical secret is idiomatic)
aws secretsmanager create-secret \
  --name prod/payments/db \
  --secret-string '{"username":"app","password":"S3ttled-Horizon-91"}'
# -> {
# ->   "ARN": "arn:aws:secretsmanager:eu-west-1:111122223333:secret:prod/payments/db-Ab12Cd",
# ->   "Name": "prod/payments/db",
# ->   "VersionId": "9f4c2e1a-77b0-4d2a-9c3e-5f8d1b6a0e42"
# -> }
 
# Read it back — this is the call your app makes at startup
aws secretsmanager get-secret-value \
  --secret-id prod/payments/db \
  --query SecretString --output text
# -> {"username":"app","password":"S3ttled-Horizon-91"}
 
# Rotation is an API, not a wiki page: a Lambda re-issues the credential on schedule
aws secretsmanager rotate-secret --secret-id prod/payments/db \
  --rotation-lambda-arn arn:aws:lambda:eu-west-1:111122223333:function:rotate-db \
  --rotation-rules '{"AutomaticallyAfterDays": 30}'

Vault (and OpenBao, its open-source fork under the Linux Foundation after HashiCorp's 2023 license change — API-compatible, bao instead of vault). This family works across clouds and on-prem, and has the richest feature set: dynamic database credentials, an internal certificate authority, encryption-as-a-service. The price is that *you* run it — clustered storage, unsealing, upgrades, and an on-call rotation for a service that everything else depends on.

# Dev mode: in-memory, auto-unsealed, root token printed. Learning ONLY — never production.
vault server -dev
# -> Root Token: hvs.6fT9wKxq...
 
export VAULT_ADDR=http://127.0.0.1:8200
vault kv put -mount=secret payments/db username=app password=S3ttled-Horizon-91
# -> == Secret Path ==
# -> secret/data/payments/db
# -> version    1
 
vault kv get -mount=secret -field=password payments/db
# -> S3ttled-Horizon-91
 
# The feature the other families lack: credentials that don't exist until requested
vault read database/creds/payments-app
# -> Key                Value
# -> lease_id           database/creds/payments-app/rGx4tYq8zW
# -> lease_duration     1h
# -> username           v-kubern-payments-app-x7Qb2M
# -> password           A1a-9wLpXnE3vRtC
# a fresh DB account, valid one hour, revoked automatically at expiry

The decision tree

Ask the questions in order. *Do you have fewer than a few dozen static secrets and a GitOps workflow?* SOPS or sealed-secrets is enough; you finished this decision last lesson. *Is everything in one cloud, with workloads that already carry IAM identities?* Use that cloud's manager — least operational load, audit and rotation built in, and the IAM policy language you already review. *Are you multi-cloud or on-prem, or do you need dynamic database credentials or an internal CA?* That is the Vault/OpenBao case — and only if you can staff it; an unstaffed Vault is worse than a well-run cloud manager. For Kubernetes, the External Secrets Operator (ESO) makes the choice reversible: it syncs any backend into ordinary Kubernetes Secrets, so your manifests never learn which manager won.

apiVersion: external-secrets.io/v1
kind: ExternalSecret
metadata:
  name: payments-db
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-prod            # a ClusterSecretStore pointing at Secrets Manager
    kind: ClusterSecretStore
  target:
    name: payments-db         # the K8s Secret ESO creates and keeps in sync
  data:
    - secretKey: password
      remoteRef:
        key: prod/payments/db
        property: password
# kubectl get externalsecret payments-db
# -> NAME          STORE      REFRESH INTERVAL   STATUS         READY
# -> payments-db   aws-prod   1h                 SecretSynced   True

What breaks at scale

Cloud managers bill per secret and per call — roughly $0.40 per secret per month plus $0.05 per 10,000 API calls on AWS — so a service mesh of chatty clients fetching on every request turns into real money and API throttling. Fetch at startup and cache (the official client-side caching libraries and the Vault Agent sidecar exist exactly for this). SOPS scales badly in *people*: every joiner and leaver means key list changes and re-encryption across repos. Vault scales badly in *criticality*: it becomes tier-zero infrastructure, and when it is down, nothing can start — plan snapshots, disaster recovery, and capacity before the first production secret lands. And no manager fixes downstream hygiene: an app that reads a secret and then logs it has leaked it, exactly as in lesson one.

Run every candidate through the same scorecard: workloads authenticate with platform identity, not stored tokens; credentials are short-lived wherever the backend allows it; the audit log ships to somewhere people actually look; rotation has been *performed*, not just configured; and there is a documented break-glass path for when the manager itself is down. Then pick the cheapest rung of the ladder that passes. Migrating up later is a data migration, not a rewrite — especially if ESO already sits between your cluster and the backend.

That closes the foundations: you can explain how secrets leak, purge them from git history, say precisely what Kubernetes Secrets do and do not protect, encrypt values for GitOps, and defend a tool choice with a decision tree instead of a vibe. If your tree ended at the third rung, the next course — *Vault in production* — covers what running it well actually takes: HA clusters on integrated Raft storage, auto-unseal, policy design, and dynamic secrets beyond the demo.

Related