The Vault provider & dynamic secrets

Beyond static encrypted files.

A bank's safe deposit room runs on one rule: the master key never leaves the building. You hand your box to the clerk, they lock it behind the counter, and you walk out holding nothing that opens anything. HashiCorp Vault's Transit engine is that clerk, and SOPS (Secrets OPerationS, the file encryptor that moved from Mozilla to the getsops organization and is now a sandbox project at the CNCF, the Cloud Native Computing Foundation) is the customer. Every other backend in this course hands SOPS something it can hold. An age identity, meaning the private half of one of those age1... recipient strings, sits in a file on disk. A PGP (Pretty Good Privacy) private key sits in a keyring. A cloud KMS (key management service) call is signed with credentials already on the machine. Transit hands SOPS nothing at all.

The cryptography is not what improves here. The maths is the same AES-256 (Advanced Encryption Standard with a 256-bit key) either way. What you gain is a chokepoint: one place that writes down every unwrap, one credential you can pull back in seconds, one policy that decides who may open what. And since Vault is already running, this lesson keeps going past static files into the thing Vault does that SOPS structurally cannot, which is mint a password that expires on its own.

Setting Up a Transit Key SOPS Can Use

Transit is Vault's encryption-as-a-service engine. Key material stays inside Vault, and the engine exposes two doors: send plaintext to /encrypt and get ciphertext back, send that ciphertext to /decrypt and get plaintext back. The key is not exportable by default, and there is no endpoint that returns it. SOPS uses this the same way it uses AWS KMS, in a pattern called envelope encryption: you lock the letter in a small box, and the bank locks away the small box's key. For each file, SOPS generates a fresh random 32-byte data key, encrypts the file's values with that data key on your own machine, then asks Transit to wrap the data key and stores the wrapped blob in the file's metadata. Vault never sees your secrets. It only ever handles the key that opens them.

# One-time setup, run by whoever administers Vault
vault secrets enable -path=sops transit
vault write -f sops/keys/firstkey
vault read sops/keys/firstkey

Success! Enabled the transit secrets engine at: sops/
Success! Data written to: sops/keys/firstkey
Key                       Value
---                       -----
allow_plaintext_backup    false
auto_rotate_period        0s
deletion_allowed          false
derived                   false
exportable                false
imported_key              false
keys                      map[1:1784692442]
latest_version            1
min_available_version     0
min_decryption_version    1
min_encryption_version    0
name                      firstkey
supports_decryption       true
supports_derivation       true
supports_encryption       true
supports_signing          false
type                      aes256-gcm96

Four of those lines matter later. exportable and allow_plaintext_backup are both false, so nobody can pull the raw key out, which is the entire reason to use Transit. deletion_allowed is false, so a mistyped vault delete gets refused, and you want that, because deleting this key orphans every file ever wrapped with it. min_decryption_version is the oldest key version Vault will still accept ciphertext from, and raising it is the most common way teams break their own SOPS files. type is aes256-gcm96, which reads as AES-256 in GCM (Galois/Counter Mode, which scrambles the data and checks it for tampering in one pass) with a 96-bit nonce (a number used once, never reused with the same key).

A Vault policy works like a building pass. It lists the doors that open, and every door not listed stays shut. One quirk trips people up on day one: Transit encrypt and decrypt are writes rather than reads, so the capability you grant is update, not read. Split the pass along the line that matters operationally. Whoever creates secrets needs both doors. Everything else needs one. The file below is HCL (HashiCorp Configuration Language, HashiCorp's own config format).

# Transit encrypt/decrypt are writes, so the capability is "update", not "read".
# One key, two paths, nothing else in the mount.
path "sops/encrypt/firstkey" {
  capabilities = ["update"]
}
 
path "sops/decrypt/firstkey" {
  capabilities = ["update"]
}

vault policy write sops-user sops-user.hcl
 
# CI only ever reads secrets, so give it half of that policy
vault policy write sops-ci - <<'EOF'
path "sops/decrypt/firstkey" {
  capabilities = ["update"]
}
EOF
 
vault token create -policy=sops-ci -ttl=20m -field=token

Success! Uploaded policy: sops-user
Success! Uploaded policy: sops-ci
hvs.CAESIC1qZ3RtN2ZmVWpQaTZ2WDlZbmxLeFQwYkxwWmNRRWc2dHc2bGc

Notice the split. Pipelines that consume secrets get decrypt and nothing else. A decrypt-only token cannot create a new encrypted file, cannot re-wrap an existing one, and cannot rotate anything, so a compromised CI (continuous integration) runner reads what it could already read and gains no new power. It also stops an attacker quietly encrypting a poisoned config that your pipeline would happily decrypt and apply. The short TTL (time to live, meaning how long the token stays valid) does similar work. Twenty minutes covers a deploy job and is useless to somebody who scrapes the token out of a log the next day.

Encrypting Through Vault

SOPS finds Vault the same way the vault command does. It reads VAULT_ADDR for the address and VAULT_TOKEN for the token, and if VAULT_TOKEN is empty it falls back to the ~/.vault-token file that vault login writes. That fallback is handy on a laptop and a trap in CI, where the token should arrive through the environment and expire quickly. You name the key by its full URI (uniform resource identifier, the complete address of a thing), including the /v1/ API (application programming interface) prefix and the keys/ path segment.

export VAULT_ADDR="https://vault.example.com:8200"
export VAULT_TOKEN="$(vault token create -policy=sops-user -ttl=1h -field=token)"
 
# secrets.yaml holds plaintext and never gets committed
sops encrypt --hc-vault-transit "$VAULT_ADDR/v1/sops/keys/firstkey" \
  secrets.yaml > secrets.enc.yaml
 
cat secrets.enc.yaml

db:
    host: ENC[AES256_GCM,data:WPyoc6nmW3dLefw=,iv:Bo2BMSAqeSIkxEJ621WyRCat2/ZhldvT7nIJP9s4sC0=,tag:S0QV8b2tgM550ULsT4uJSQ==,type:str]
    password: ENC[AES256_GCM,data:+T68MzWzCoGlHH+C71XN4R8Lkg==,iv:asYtWfxasaTEf0+sTBO8Y3vkAdpUiqUK/3LgxY3IFlE=,tag:Qo9wUKy4OrHPEWJ7WPcA+A==,type:str]
sops:
    hc_vault:
        - vault_address: https://vault.example.com:8200
          engine_path: sops
          key_name: firstkey
          created_at: "2026-07-22T09:14:02Z"
          enc: vault:v1:ooW8bWZxSX4AHdTH4AIxyc/qXwf6flzURQ/4BsrbWz4RiV/W86AumZhgeBXgimEVrozmqykTYoUYBndG
    lastmodified: "2026-07-22T09:14:02Z"
    mac: ENC[AES256_GCM,data:d+ZNdNspXOvVvQCeb9f18iZG8Jf8Hn9z8jcrjcFflYr50UJJZTJ6mQEN2PyK0hZldo1Zf/QrBSvnCcDFuZwF7+0gQ+o1jb/JnaYJPeL7bUDA2oCc2Y5OxdZdmroZMjglPe0GVGu5B86SEYaAA90/nmfgLSDe+37rcgzE1UT1MNk=,iv:3VBpuTUwmbGemdQxMKcrE54cCaRpc/ZFkvjt9pCi3ME=,tag:cg6gOYrTS5JwJTRR4bwS3Q==,type:str]
    unencrypted_suffix: _unencrypted
    version: 3.13.2

Now read that file the way an attacker would. The key names are sitting there in plaintext: db, host, password. SOPS encrypts values and never key names, which is what makes diffs reviewable in a pull request, and which also means your structure is public. The metadata is plaintext too, and it says more than people expect. vault_address publishes your Vault's hostname and port. engine_path and key_name publish the exact mount and key worth attacking. In a public repository that is a free map. The one unreadable part is enc, the data key wrapped by Transit, and its vault:v1: prefix is Vault's own label for which version of the key did the wrapping. That prefix is about to matter a lot.

# Declare the key once so nobody has to remember the flag.
creation_rules:
  - path_regex: \.enc\.yaml$
    hc_vault_transit_uri: "https://vault.example.com:8200/v1/sops/keys/firstkey"

With that rule in place, sops encrypt secrets.yaml picks the key by filename, and sops decrypt secrets.enc.yaml needs no flags at all, because the URI is already recorded inside the file. If you prefer environment configuration, SOPS_VAULT_URIS sets the same thing. Decryption needs only update on sops/decrypt/firstkey, so test that boundary rather than trusting it.

sops decrypt secrets.enc.yaml
 
# Negative test: a token without the sops-ci policy must fail
VAULT_TOKEN="$(vault token create -policy=default -ttl=5m -field=token)" \
  sops decrypt secrets.enc.yaml

db:
    host: pg.internal
    password: 6Qh2-vNr9-Ttk4-Wpa1
 
Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  https://vault.example.com:8200/v1/sops/keys/firstkey: FAILED
    - | failed to decrypt sops data key from Vault transit
      | backend 'sops/decrypt/firstkey': Error making API request.
      |
      | URL: PUT https://vault.example.com:8200/v1/sops/decrypt/firstkey
      | Code: 403. Errors:
      |
      | * 1 error occurred:
      | 	* permission denied
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Five seconds of testing, and now you know the boundary holds. A policy nobody has ever tested is a belief, not a control.

Proving Vault Is in the Loop

Every sops decrypt against Transit is a live network call. That is the cost. The benefit is a receipt. Vault's audit devices write a line for every request, the way a front desk logs every visitor, so "who read the production database password, and when" becomes a query rather than a guess. Audit devices are off by default, so turn one on before you need it. One caveat bites people badly: if the only enabled audit device cannot write, because the disk filled or permissions changed, Vault stops answering requests entirely. The device is a dependency as well as a control. The tool below, jq, is a command-line filter for JSON (JavaScript Object Notation) that pulls named fields out of each log line.

vault audit enable file file_path=/var/log/vault/audit.log
 
# After a CI job decrypts, look at what Vault wrote down
jq -r 'select(.type=="response" and .request.path=="sops/decrypt/firstkey")
       | [.time, .auth.display_name, (.auth.policies|join("+")), .request.remote_address]
       | @tsv' /var/log/vault/audit.log | tail -3

Success! Enabled the file audit device at: file/
2026-07-22T09:21:44.881Z	token	default+sops-ci	10.42.7.19
2026-07-22T10:04:02.117Z	token	default+sops-ci	10.42.7.19
2026-07-22T11:36:55.402Z	token	default+sops-user	10.42.9.4

Two things there that an age key on disk will never give you. The first is the receipt. The second is revocation: vault token revoke -accessor <accessor> cuts one specific token off in the time the API takes to answer. Learn one wrinkle now rather than during an incident. Audit devices run sensitive fields through an HMAC (hash-based message authentication code, a one-way fingerprint computed with a secret key), so the accessor recorded in the log is a fingerprint, not the value you can paste into that revoke command. To get from a log line to a live token you list the real accessors with vault list auth/token/accessors, then fingerprint each candidate with vault write sys/audit-hash/file input=<accessor> until one matches. That same HMAC treatment is why the log proves a decrypt happened without ever recording what was decrypted. Compare all of that to an age private key that leaked six months ago. It still works today, on every copy of every file that ever existed, and nothing anywhere recorded a single use of it.

Rotating the Transit Key Without Breaking Old Files

Transit rotation is versioned, closer to a new edition of a book than a replacement. Rotating adds version 2 and encrypts new things with it, while version 1 stays available for decrypting old ciphertext. Your existing files carry vault:v1: in their metadata, so they keep working, untouched, for as long as you leave them alone. That is a feature right up until somebody decides the old version should stop being usable.

# As the Vault admin: add a new key version
vault write -f sops/keys/firstkey/rotate
vault read -field=latest_version sops/keys/firstkey
 
# Rotation did not touch a single file
git ls-files '*.enc.yaml' | xargs grep -hoE 'vault:v[0-9]+' | sort | uniq -c

Success! Data written to: sops/keys/firstkey/rotate
2
     13 vault:v1

To move files onto version 2 you have to re-encrypt them, and sops rotate is the command for it. It decrypts the file, generates a brand new data key, re-encrypts every value with it, and asks Transit to wrap that new data key, which Vault does at the current version. sops updatekeys is the wrong tool here, because its job is re-wrapping in response to changes in the key list in .sops.yaml, not moving to a newer version of the same key. Do the files first, verify, then tighten Vault.

# sops rotate needs encrypt AND decrypt, so use the sops-user token
for f in $(git ls-files '*.enc.yaml'); do sops rotate -i "$f"; done
git ls-files '*.enc.yaml' | xargs grep -hoE 'vault:v[0-9]+' | sort | uniq -c
 
# Only now is it safe to retire version 1
vault write sops/keys/firstkey/config min_decryption_version=2

     13 vault:v2
Success! Data written to: sops/keys/firstkey/config

Get that order wrong and the failure is loud but misleading, because nothing about the file changed and the token still holds the right capability.

# Someone restored an old branch: this file is still wrapped at v1
sops decrypt legacy/secrets.enc.yaml

Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  https://vault.example.com:8200/v1/sops/keys/firstkey: FAILED
    - | failed to decrypt sops data key from Vault transit
      | backend 'sops/decrypt/firstkey': Error making API request.
      |
      | URL: PUT https://vault.example.com:8200/v1/sops/decrypt/firstkey
      | Code: 400. Errors:
      |
      | * ciphertext or signature version is disallowed by policy (too old)
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Read the status code, not the vibe. This is a 400, not a 403. A 403 means the token lacks the capability, so you go and fix the policy. A 400 carrying "ciphertext or signature version is disallowed by policy (too old)" means Vault is refusing the key version itself, and no amount of extra permission will help. The way back is to put min_decryption_version at 1, run sops rotate -i across every affected file, confirm with that same grep, then raise it again. Note the trap in the ordering: sops rotate has to decrypt the file before it can re-encrypt it, so while min_decryption_version sits at 2 the repair tool fails for exactly the same reason the pipeline did.

A Break-Glass Key You Keep Offline

There is a clean answer to the offline problem, and it falls out of how SOPS key groups work. Think of the fire axe behind glass in a stairwell. Nobody touches it on a normal day, and breaking that glass is a decision somebody has to own. Within one SOPS key group, any listed key can recover the data key on its own, because SOPS wraps the same data key once per recipient. So you list Transit and an age recipient side by side and get both properties at once: day-to-day access goes through Vault and is audited, while one age identity in a sealed envelope in a safe can open the file when Vault is down. Existing files do not pick this up by themselves, so run sops updatekeys over them, which is precisely the job that command exists for.

# Both keys wrap the SAME data key, so either one alone can open the file.
creation_rules:
  - path_regex: \.enc\.yaml$
    key_groups:
      - hc_vault:
          - https://vault.example.com:8200/v1/sops/keys/firstkey
        age:
          # This is the PUBLIC recipient. It is safe to commit.
          # Its matching AGE-SECRET-KEY-1... identity is printed on paper,
          # kept in a safe, and never copied onto a laptop.
          - age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

The trade-off is exactly what it looks like. That age identity decrypts everything, forever, with no audit entry and no revocation short of re-encrypting every file. So it lives offline, taking it out is a declared event with two people present, and you rotate afterwards. What you must not do is dodge the decision and then discover during an outage that your only decryption path depends on the thing that is down.

Where Static Files Stop and Dynamic Secrets Start

Everything so far still produces a static file. Vault guards the key, but the database password inside secrets.enc.yaml is a real, long-lived password, sitting in every clone of that repository and every backup of it, valid until a human changes it. Dynamic secrets invert the whole arrangement. A hotel key card is the closest everyday version: the front desk mints one at check-in, it opens exactly your room, and it dies at checkout whether you hand it back or not. Vault's database engine does that with database users.

vault secrets enable database
 
vault write database/config/app-postgres \
  plugin_name=postgresql-database-plugin \
  allowed_roles="app-readonly" \
  connection_url="postgresql://{{username}}:{{password}}@pg.internal:5432/app?sslmode=require" \
  username="vault-admin" \
  password="$PG_ADMIN_PW"
 
vault write database/roles/app-readonly \
  db_name=app-postgres \
  default_ttl=1h \
  max_ttl=24h \
  creation_statements="CREATE ROLE \"{{name}}\" WITH LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";"

Success! Enabled the database secrets engine at: database/
Success! Data written to: database/config/app-postgres
Success! Data written to: database/roles/app-readonly

# The app asks for this at startup instead of reading a file
vault read database/creds/app-readonly
 
# And if you want it gone before the hour is up
vault lease revoke database/creds/app-readonly/8f2c1b7e-3d94-4a16-b0c5-7e1a92d4fb63

Key                Value
---                -----
lease_id           database/creds/app-readonly/8f2c1b7e-3d94-4a16-b0c5-7e1a92d4fb63
lease_duration     1h
lease_renewable    true
password           A1a-uXyZ9k2QpLm4nRtV
username           v-token-app-read-kL2rN9pQxWv3sT7bYc4d-1784711045
 
All revocation operations queued successfully!

Nothing was stored anywhere. Vault created that PostgreSQL user milliseconds ago and will drop it in an hour. Look at the generated username, because it is doing real work: it encodes the token display name, the role, a random chunk and a timestamp, so "who touched this table" becomes a lookup instead of an investigation. And when something goes wrong you do not rotate a shared password across forty services and hope. You revoke one lease.

Compare the blast radius honestly. A leaked file plus a stolen decrypt token gives an attacker a password that keeps working until a human notices and rotates it everywhere it was used. A leaked dynamic credential gives them a named, read-only database user that dies within the hour and left a lease ID in the audit log on the way in. SOPS cannot do this, and no configuration will make it, because it encrypts files and does not issue credentials. The price is real, though. Your application now has to authenticate to Vault at startup, using Kubernetes auth, AppRole (a machine login made of a role ID and a secret ID, roughly a username and password for software) or a cloud identity, and Vault has to be reachable every single time. That is an availability dependency, not a footnote.

Handing These Secrets to Things That Cannot Speak Vault

Plenty of software cannot ask Vault for anything, so two bridges are standard. The first covers SOPS files already sitting in Git. Flux's kustomize-controller decrypts them at apply time using a Vault token it reads from a Kubernetes Secret, and it looks for one exact data key name inside that Secret, sops.vault-token. Get that name wrong and the symptom is a decryption failure rather than a missing-secret error, which sends people hunting in the wrong place. Be clear about the limit here: for SOPS decryption, Flux accepts a static token and nothing else. There is no Kubernetes auth or AppRole path for this particular integration, so a long-lived token has to live in the cluster and something has to keep it alive.

apiVersion: kustomize.toolkit.fluxcd.io/v1
kind: Kustomization
metadata:
  name: apps
  namespace: flux-system
spec:
  interval: 10m
  path: ./apps/production
  prune: true
  sourceRef:
    kind: GitRepository
    name: flux-system
  decryption:
    provider: sops
    secretRef:
      name: sops-vault      # the data key inside MUST be named sops.vault-token

kubectl -n flux-system create secret generic sops-vault \
  --from-literal=sops.vault-token="$(vault token create -policy=sops-ci -period=768h -field=token)"

secret/sops-vault created

That token can decrypt everything Flux reconciles, which is why decrypt-only capability and a working audit device stop being optional. It also has a lifetime, and nothing in the cluster renews it for you. A periodic token is the usual choice, meaning one that can be renewed indefinitely as long as somebody renews it inside each period, and creating one needs a token holding the sudo capability on auth/token/create. Decide up front what runs vault token renew, because the failure mode is a Kustomization stuck reporting a decryption error while your commits quietly pile up unapplied.

The second bridge is the External Secrets Operator (ESO), which reads from Vault and writes an ordinary Kubernetes Secret that any pod can mount. Its VaultDynamicSecret generator can pull a fresh database credential rather than a static value, which is as close to dynamic secrets as software that only knows how to read an environment variable will ever get. Be honest about what happens at that boundary: the moment a leased credential lands in a Kubernetes Secret it is stored again, and it is only as short-lived as your refresh loop. So check the two numbers against each other. If the lease is one hour and refreshInterval is 1h, you have guaranteed yourself a window where pods hold a credential Vault already revoked. Set the refresh to a comfortable fraction of the lease TTL, and make sure the application reconnects on an authentication failure instead of logging one error and wedging until somebody restarts it.

Try this

Run vault secrets enable -path=sops transit on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the file decides which Vault gets your token. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related