Dynamic secrets at scale
Database and cloud engines, leases, max_ttl, and revocation trees.
You search the code repository for a database password and find the same one pasted into twelve different deployment config files. That is a static secret: stock you have to guard, hunt down, and rotate forever. A dynamic secret works the other way round. Vault generates it on demand, scopes it to one consumer, and destroys it on a timer. The database admin password never leaves Vault. Each app gets a brand-new database user of its own, good for a few minutes. When the lease runs out (the lease is the countdown Vault attaches to that credential), Vault deletes the user. Nothing long-lived is left lying around for anyone to steal.
Leases, TTL, and the max_ttl ceiling
Every dynamic secret arrives with a lease, and a lease carries two numbers. The TTL (time to live) is a clock; when it reaches zero Vault revokes the credential without being asked. The max_ttl is a ceiling; past that moment no amount of renewing keeps the credential breathing. A workload renews while it still has work to do, stops renewing when it finishes, and the credential dies quietly on its own. Turn that around and you can see the whole point. In the static world a credential defaults to valid forever. Here it defaults to about to die.
TTL length is the dial you will actually turn. Short enough that a leaked credential is worthless within minutes. Long enough that renewal traffic and revocation work do not swamp Vault or the database behind it. The max_ttl is your emergency brake: without it, whoever holds a stolen lease ID (the handle Vault uses to name one specific credential) can keep renewing that credential for as long as they please. Set default_ttl for ordinary Tuesdays and max_ttl for the day you need every outstanding credential dead by a known deadline.
Wiring up a real database role
You configure this once. Vault holds the privileged database connection plus a role template that says how to create a user and how to drop that user again. Every read of database/creds/<role> mints a fresh user with narrow grants and a lease of its own. Finished early? Revoke the lease and that user disappears everywhere at once. No emails to six teams asking them to please rotate the shared password by Friday.
The creation_statements template holds placeholders that Vault fills in per request: {{name}}, {{password}}, and {{expiration}}. The revocation_statements are the SQL (structured query language, the language databases speak) that drops the user when the lease expires or when you revoke it by hand. The postgres plugin falls back to a default drop if you leave them out, but write them yourself anyway, because a plugin with no such fallback leaves orphaned users piling up inside the database itself, standing privilege that nobody declared and nobody is watching.
vault secrets enable database# username/password are Vault's own privileged database account;# the {{username}}/{{password}} templates in connection_url are filled in from themvault write database/config/app-postgres \plugin_name=postgresql-database-plugin \allowed_roles="payments-ro" \connection_url="postgresql://{{username}}:{{password}}@db.internal:5432/app" \username="vault-root" \password="$PG_VAULT_ROOT_PW"vault write database/roles/payments-ro \db_name=app-postgres \creation_statements="CREATE ROLE \"{{name}}\" LOGIN PASSWORD '{{password}}' VALID UNTIL '{{expiration}}'; GRANT SELECT ON ALL TABLES IN SCHEMA public TO \"{{name}}\";" \revocation_statements="REVOKE ALL PRIVILEGES ON ALL TABLES IN SCHEMA public FROM \"{{name}}\"; REVOKE USAGE ON SCHEMA public FROM \"{{name}}\"; DROP ROLE IF EXISTS \"{{name}}\";" \default_ttl=20m max_ttl=1h
Success! Enabled the database secrets engineSuccess! Data written to: database/config/app-postgresSuccess! Data written to: database/roles/payments-ro
vault read database/creds/payments-rovault lease renew database/creds/payments-ro/9f2..
username v-kubernetes-payments-ro-x7Qb...password A1b2C3...lease_id database/creds/payments-ro/9f2..lease_duration 20mlease_id database/creds/payments-ro/9f2..lease_duration 1200 # renewed while workload still active
vault lease revoke database/creds/payments-ro/9f2..vault list sys/leases/lookup/database/creds/payments-ro
Revoked lease database/creds/payments-ro/9f2..# DB user dropped; other consumers on different leases unaffectedKeys----(no entries) # lease gone from the tree
Revocation trees: pull one thread, the branch unravels
Leases hang off one another like a family tree. The token a workload logs in with has a lease, and that lease is the parent of every secret the token leases afterwards. Revoke the parent and Vault walks the branch below it, revoking every child on the way down. That single property is what makes incident response survivable: kill the token an attacker stole, and Vault revokes every database user, cloud key, and certificate it ever minted, from one command instead of twenty cleanup tickets. It is a sweep rather than a switch, though, and revoked does not always mean dead. The database user really is dropped. A temporary cloud key keeps working until it expires on its own, and a revoked certificate only goes onto Vault's revocation list, the published roster of certificates nobody should trust any more, so anything that does not check that roster will carry on accepting it. Vault also retries the revocations that fail and marks the ones it cannot finish as irrevocable leases, so check what is left behind instead of assuming the branch came away clean.
It is also the reason each workload gets a token of its own. Share one token across ten services and you share one revocation blast radius, so you can no longer cut off the compromised service without cutting off the other nine. During an incident, vault lease revoke -prefix database/creds/payments-ro/ is a scalpel. Revoking the token everybody shares is an outage.
The same trick, everywhere else
Databases are where most teams start, and the engine model repeats with almost nothing changed. Cloud IAM (identity and access management, the system that decides which account may call which API), where Vault assumes a role and hands back short-lived AWS, Google Cloud, or Azure credentials. Message queues. PKI (public key infrastructure, the machinery that issues TLS certificates). SSH (secure shell) access to hosts. One rule holds across all of them: Vault keeps a single privileged root credential under heavy audit, consumers never lay eyes on it, and what they receive instead are disposable children with narrow scope.
Notice what happened to the trust problem. It did not vanish. It moved, and it shrank. You stop chasing thousands of copies of a password scattered across repos and .env files, and you start guarding a handful of engine root credentials on a rotation schedule. Smaller target, better defended, still a target.
Tuning it once it is live
Watch two numbers above all: renewal rate and revocation failures. A jump in renewals usually means your TTL is shorter than your connection pool's patience. Failed revocations mean orphaned database users, so schedule a reconciliation query that lists every role matching Vault's naming prefix and alerts you on the stragglers. And be honest about outages. Dynamic secrets degrade gracefully while Vault is unreachable only if the application knows how to refresh a credential. Write clients that retry and re-fetch, not clients that grab a password at startup and cling to it until the process dies.
Then rehearse the revocation path in staging, under load. Mint credentials, revoke the parent token, and watch what the database actually does. In Postgres, dropping a role does not hang up the sessions that role already has open, so pooled connections carry on working until something recycles them; it is the next login that gets refused. Time that gap, and if it is longer than you can stomach, terminate the leftover backends yourself as part of the revocation rather than trusting the drop to do it. An untested dynamic setup finds its gaps during the first real incident, at the worst hour of the worst night.
Cloud credentials and SSH follow the same shape
The AWS secrets engine assumes an IAM role on your behalf and returns access_key, secret_key, and security_token, with a lease attached like any other dynamic secret. The Google Cloud and Azure engines are built the same way. One difference is worth knowing before an incident: temporary cloud keys like these cannot be called back early, so revoking the lease tidies up Vault's books while the keys themselves keep working until they expire on their own. That is the argument for keeping the time to live short on cloud roles. No access keys frozen into Terraform state files. No permanent IAM user per microservice.
The SSH engine does the same job for host access, handing out one-time OTP (one-time password) credentials or short-lived signed certificates, so standing access to a server is measured in minutes instead of years. Nobody has to walk the fleet afterwards pulling stale public keys out of authorized_keys files that somebody edited by hand two years ago.
vault write aws/roles/deploy \credential_type=assumed_role \role_arns=arn:aws:iam::111122223333:role/deploy-scoped \default_sts_ttl=30m max_sts_ttl=1hvault read aws/creds/deploy
Success! Data written to: aws/roles/deploylease_id aws/creds/deploy/a8b9...lease_duration 30mlease_renewable falseaccess_key ASIA...secret_key ...security_token IQoJb3JpZ2luX2VjE...
kubectl exec -n prod payments-0 -- env | grep -i AWS_ACCESS || echo "no static AWS keys in pod"vault list sys/leases/lookup/aws/creds/deploy | head -5
no static AWS keys in pod# dynamic creds fetched at runtime, nothing in env at restKeys----a8b9...c7d2...
Put lease creation and revocation rates on a dashboard. A sudden spike in database/creds reads with no deployment behind it deserves a look; it might be a retry storm, or it might be a leaked token being replayed. Pair those metrics with Vault audit queries so whoever is on call can take a username out of a slow query log and trace it back to the token that minted it.
Connection pools need to be part of this conversation. A pool holds a credential for as long as it holds the connection, so the pool's maximum connection lifetime has to sit below default_ttl. Run it the other way round and the pool goes on serving connections built from a user Vault has already dropped, and the next connection it opens comes back as an authentication failure. Write that pool timer into the runbook right beside the TTL values. Whoever tunes one has to tune the other.
curl -s -H "X-Vault-Token: $VAULT_TOKEN" \"$VAULT_ADDR/v1/sys/metrics?format=prometheus" | grep expire_numvault audit list
vault_expire_num_leases 1428vault_expire_num_irrevocable_leases 3# graph those two; vault_database_CreateUser tracks how long a mint takesPath Type Description---- ---- -----------file/ file n/a
Rolling it out without a bad week
The payoff lands hardest where every service used to share one password. Changing that password meant coordinating a dozen deploys, half the fleet broke anyway, so in practice nobody ever changed it. With a user per lease, you revoke one lease or wait out the TTL, and exactly one consumer loses access. Start with a service that will not page anyone when it breaks. Pick a default_ttl short enough to make a stolen credential worthless and long enough for your connection pool. Then widen the rollout.
Watch the user count as you go. You wrote one role; every read of it mints another user, so users and leases are what pile up. Each revocation that quietly fails leaves its user behind, and some engines start creaking once tens of thousands have collected, which is what that reconciliation query is really for. And if an application caches a password past max_ttl, expect authentication errors that everyone in the channel will describe as the database flapping. The database is fine. The lease discipline is not.
For cloud roles, prefer STS-style temporary credentials (STS is the AWS Security Token Service, the API that vends time-limited keys) over having Vault create real IAM users with long-lived access keys. Temporary credentials expire on their own with no delete call required, which shrinks your exposure window when a revoke is slow or the cloud API is throttling you.
Try this
Three commands tell the whole story. Mint a database credential, look at the lease behind it, revoke it, and then check the database to confirm that user really is gone.
vault read database/creds/payments-rovault lease lookup <lease_id>vault lease revoke <lease_id># optional: psql check that the role vanishedpsql -c "\du" | grep payments-ro || echo "dynamic user gone"
Key Value--- -----lease_id database/creds/payments-ro/abcdlease_duration 20musername v-kubernetes-payments-ro-x7k2password A1b2C3...lease_id database/creds/payments-ro/abcdexpire_time 2026-07-28T08:10:00ZSuccess! Revoked lease: database/creds/payments-ro/abcddynamic user gone
Takeaway
Dynamic secrets trade "guard this password forever" for "mint it, use it, let it expire." Leases buy you revocation trees. A username per consumer buys you a database log that names names. And max_ttl is what stops a stolen lease from renewing its way into next year.
Next up: rotate the engine root credential, line your connection pool lifetimes up with default_ttl, and move one shared static password off the critical path this sprint.