BlogSecrets

Automating secret rotation without downtime

Rotate database and API credentials on a schedule using overlapping versions so nothing breaks mid-rotation.

May 6, 2025·10 min readAdvanced

Rotation breaks things when the old credential dies before every client has the new one. The fix is an overlap window: create version N+1, roll it out to all readers, and only then disable N. Nothing ever sees a gap.

Zero-downtime rotation
1
Create v2
both valid
2
Deploy readers
move to v2
3
Verify
no v1 usage
4
Disable v1
window closes
Two ways to rotate
Naive (breaks)
change password
old dies instantly
in-flight clients fail
a mini outage
Overlapping
two valid versions
roll readers over
confirm, then revoke
no gap

Let the manager do it

Dynamic secrets sidestep rotation entirely — every consumer gets its own short-lived credential. Where you must rotate a shared one, use the platform's staged rotation.

bash
# AWS Secrets Manager: staged rotation with a Lambda
aws secretsmanager rotate-secret \
--secret-id prod/db \
--rotation-lambda-arn arn:aws:lambda:...:rotate-db \
--rotation-rules AutomaticallyAfterDays=30
Give the window room
The overlap must be longer than your longest cache/lease TTL and deploy time. Rotate faster than that and you recreate the outage you were avoiding.
Go deeper in a courseVault from dev to productionDynamic secrets, leases, and staged rotation without downtime.View course

Related posts