Zero-downtime rotation & the dual-secret pattern

Rotate roots, leases, and app credentials without an outage.

You change the database password on a Tuesday morning, right on schedule. Forty minutes later three services are down, because one of them read the old password at boot and never looked again. Everyone agrees credentials should rotate. Then the first rotation causes an outage, rotation gets switched off "temporarily", and two years go by. The fix here is a pattern rather than a product: dual-secret rotation, where two versions of the credential are valid at the same time, for a window longer than anything that could still be holding the old one. Size that window right and rotation becomes boring, which is exactly what you want it to be.

The dual-secret pattern: two slots, both live

You keep two versions of the credential alive at once, one in the current slot and one in the previous slot. Rotating means minting a new value, promoting it to current, and pushing the old current down into previous. Both still open the door. Each consumer picks up the new value whenever it next refreshes, on its own schedule, with nobody coordinating a synchronised restart. You retire the previous version only once the overlap window has fully run out, and that window has to be longer than your longest cache TTL (time to live, how long a copy is kept before it is fetched again), your longest lease, and your slowest deploy, added together.

AWS Secrets Manager (Amazon Web Services' hosted store for secrets) ships this pattern with names already attached: the AWSCURRENT and AWSPREVIOUS staging labels do the promote-and-demote step for you. Anywhere else you build the same thing by hand, with a version table, a symlink swap, or two config keys called password_current and password_previous. The pattern is the same everywhere. Only the labels change.

The credentials Vault itself has to hold

Dynamic secrets mostly rotate themselves, because their leases expire and Vault mints fresh ones. The privileged root credentials the engines log in with do not expire, and those are the crown jewels: the database admin account, the cloud account allowed to create users. Vault can rotate those in place and keep the new value to itself. After that command runs, nobody knows the password. Not the database administrator, not you, not the person who first typed it in.

Vault's barrier keys, the encryption keys wrapped around everything in its storage, rotate too. Running vault operator rotate adds a new keyring version for future writes and leaves older data readable under the version that encrypted it. The rule of thumb: the higher the privilege, the more often it should rotate on a schedule, and the fewer humans should ever see the value. Put engine root rotation on the calendar the day you switch on dynamic secrets, not six months later when an auditor asks you about it.

vault write -f database/rotate-root/app-postgres
vault read database/config/app-postgres | grep -v password
vault write -f aws/config/rotate-root

Success! Rotated root credentials for app-postgres
# connection_url shows {{username}}/{{password}} — actual password not returned
Success! Rotated AWS root credentials
# even operators no longer know the engine root password

vault operator rotate
vault read sys/key-status

Key Term  3   Install Time  2026-07-24T09:00:00Z
 
Term  1  Active  false
Term  2  Active  false
Term  3  Active  true   # new barrier key for future writes; old data still readable

Scheduled rotation versus the emergency kind

Keep the two apart in your head. Scheduled rotation is housekeeping. It caps how long any credential can live, and it keeps the machinery greased so that the emergency run is never the first real one. Emergency rotation is the answer to a compromise: change it now and accept that something might blink, because a live attacker outranks a tidy deploy.

The dual-secret pattern is what makes scheduled rotation invisible to everyone downstream. A rehearsed runbook is what makes the compromise case fast. If you can only build one of them, build the scheduled path first. An unrehearsed emergency rotation in the middle of an incident is how you end up with a breach and an outage stacked on top of each other.

How long should the overlap be?

The number that takes production down is the overlap window. It has to be longer than the longest cache TTL, the longest connection-pool lifetime, the longest lease, and the slowest rollout across every consumer of that secret. Take the biggest of those numbers, then add margin, because somebody always forgets a consumer. Then go and test the retirement step in staging. The classic failure is a nightly batch job that reads the secret once a day and is still holding the previous value at the moment you delete it.

Rotation is not finished when the new value works. It is finished when the old value can be removed and nothing notices. Write down every consumer of a shared secret before the first rotation: connection pools, sidecars, cron jobs, Terraform state, the reporting job nobody has touched since 2023. Do that inventory beforehand, not while the outage is running.

A runbook per type of secret

Different secrets break in different ways. Database passwords want either two database users or dual-secret overlap inside the application. API keys (application programming interface keys, the tokens one service uses to call another) want versioned keys at the provider so both work for a while. TLS certificates (the files that prove a server's identity on an encrypted connection) want renewBefore set longer than your slowest reload. Vault dynamic credentials want a lease TTL shorter than the app's refresh interval. No single template covers all four. Every one of them shares the same spine: overlap, then retire.

Track the last-run and next-run dates for rotation in whatever system already tracks certificate expiry, so one screen shows both. A secret documented as rotating annually that has not actually moved in three years is standing privilege wearing a compliance badge.

aws secretsmanager update-secret-version-stage \
  --secret-id prod/db --version-stage AWSCURRENT --move-to-version-id v2
aws secretsmanager get-secret-value --secret-id prod/db --version-stage AWSPREVIOUS
# after overlap > max(cache TTL, deploy time):
aws secretsmanager update-secret-version-stage \
  --secret-id prod/db --remove-from-version-id v1 --version-stage AWSPREVIOUS

VersionStage moved: v2 is AWSCURRENT; v1 is AWSPREVIOUS
{"SecretString":"{...previous password still valid...}"}
Removed AWSPREVIOUS from v1 — only after all consumers confirmed on v2

Static secrets, and getting apps to notice

KV secrets (key-value, a plain stored string such as a password or a token) do not rotate themselves. You write a new version into Vault or the cloud manager, and every consumer has to poll for it or subscribe to the change. Pair that versioning with the dual-secret pattern: write v2, point consumers at the latest version, leave v1 valid until the overlap has elapsed, then destroy v1. In Kubernetes, the refreshInterval on External Secrets Operator is that poll loop with a name on it.

Any application that loads a secret into memory once needs an explicit way to be told the value changed. That might be a SIGHUP (the hang-up signal, which most daemons treat as "reload your config"), a nudge from a sidecar, or a timer that re-reads the file on a loop. Your runbook has to list every cache layer between the store and the running process: JVM (Java Virtual Machine) system properties, an nginx reload, a connection pool recycle. Miss one and a rotation that looked successful has quietly left the old password working somewhere.

vault kv put secret/prod/db password=v2-new-value
vault kv get -version=1 secret/prod/db
vault kv get -version=2 secret/prod/db
vault kv destroy -versions=1 secret/prod/db

===== Data =====
password    v1-old-value
 
===== Data =====
password    v2-new-value
 
Success! Data deleted for secret/prod/db version 1

A rotation calendar with no inventory of consumers is theatre. Schedule the retirement only once automated checks confirm that every known client picked up the new value during the overlap. A spreadsheet of owners goes stale inside a quarter. A metric showing how old each service's copy of the secret is does not go stale.

Certificate rotation has exactly the same shape. renewBefore has to beat your slowest reload path, in the same way the AWSPREVIOUS overlap has to beat your slowest cache. Teams with database password rotation wired perfectly still take outages on TLS, because nobody ever connected the nginx reload to cert-manager updating the Secret.

Publish the rotation calendar to application owners with the overlap window written out in real dates. When a team knows AWSPREVIOUS disappears on Tuesday, somebody checks their caches on Monday. When they do not know, they find out on Tuesday, from an alert.

vault lease renew database/creds/payments-ro/9f2..
vault list sys/leases/lookup/database/creds/payments-ro | wc -l

lease renewed — lease_duration 1200
 
7
# active leases — know count before emergency mass-revoke

Rotation is a program with owners and dates, not a button somebody presses. Static KV secrets need an owner, a schedule, and either a dual-write or a bounce window so applications actually pick up the new value. Dynamic secrets rotate by design, so prefer them anywhere the system can mint credentials on demand. The database root and cloud root credentials Vault holds need scheduled rotate-root jobs, so that no human ever needs to know them at all.

Rehearse the emergency version. Assume the value is already public, mint or set a new one, revoke the old, flush the caches. Time it with a stopwatch, from the moment someone decides to rotate to the moment the old credential fails everywhere. Those minutes are your real blast-radius number, and it is almost always bigger than anyone guessed.

Write down which applications need a restart and which can hot reload. Updating a Secret in Kubernetes changes nothing for a process that read its environment variables once at startup. Pair rotation with rollout mechanics, or with a file watcher that reloads when the mounted file changes.

Try this

Rotate a database root through Vault, bump a KV secret to a new version, and confirm that an old lease can still be revoked on demand. Watch what the metadata reports about version numbers as you go.

vault write -f database/rotate-root/payments
vault kv put secret/payments/db password=new-$(date +%s)
vault kv metadata get secret/payments/db
vault lease revoke -prefix database/creds/payments-readonly/

Success! Data written to: database/rotate-root/payments
 
======= Secret Path =======
secret/data/payments/db
 
======= Metadata =======
Current Version      4
Oldest Version       1
...
 
Success! Revoked prefix: database/creds/payments-readonly/
# apps on dynamic creds reconnect with new leases; static consumers need reload

Takeaway

Rotation goes smoothly when three things have already been rehearsed: who owns the secret, how each application reloads it, and how fast you can revoke. Dynamic credentials handle their own turnover through leases. Static ones need versions, an overlap window sized to the slowest consumer, and one drill where you pretend the value has leaked.

Next: pick the static secret with the widest blast radius, put a name and a date against it, and run an emergency rotation in a non-production environment with a stopwatch running.

Related