Automating secret rotation without downtime

Rotate database and API credentials on a schedule using overlapping versions so nothing breaks mid-rotation.

Rotation breaks production when version N dies before every client has version N+1. The naive playbook — change the password in the database, update the config, hope nothing cached the old value — guarantees a mini-outage every quarter. The fix is an overlap window: two valid credentials coexist while readers migrate, verification confirms zero traffic on the old version, and only then is N revoked. The window must be longer than your longest cache TTL, pod restart cycle, and deploy pipeline combined.

Better still: skip shared static credentials entirely. Vault's database secrets engine and cloud IAM roles mint dynamic credentials — unique per consumer, auto-revoked at lease end — so rotation becomes something the platform does continuously instead of a calendar event. Where static secrets are unavoidable, use staged rotation with explicit version tracking. The Advanced secrets track compares delivery patterns — injector, CSI, ESO — and rotation without downtime.

Overlapping versions vs the naive cutover

Map every reader before you rotate: application pods, CI jobs, cron tasks, third-party integrations, and backup tools. The one you forget is the one that pages you at 2am when v1 is revoked.

Let the platform rotate with staging

AWS Secrets Manager supports rotation Lambda functions that create a new secret version, update the database, test connectivity, and mark the new version AWSCURRENT only after the test passes — all while the previous version remains valid during the staging window. Schedule rotation with AutomaticallyAfterDays and monitor CloudWatch for rotation failures instead of discovering a broken credential at deploy time.

aws secretsmanager rotate-secret \
  --secret-id prod/db/master \
  --rotation-lambda-arn arn:aws:lambda:eu-west-1:111:function:rotate-rds \
  --rotation-rules AutomaticallyAfterDays=30
 
# verify both versions existed during staging
aws secretsmanager list-secret-version-ids --secret-id prod/db/master

Dynamic secrets skip the rotation calendar

When every pod gets its own database user with a 20-minute lease, there is no shared password to rotate — Vault creates and drops users continuously. Pair with Stakater Reloader or an app that reloads on SIGHUP for the cases where you must rotate a shared API key in KV. External Secrets Operator refreshes Kubernetes Secrets on an interval; rotation in Vault flows through automatically, but pods that read env vars at startup still need a restart strategy.

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: api-key
  namespace: shop
spec:
  refreshInterval: 15m
  secretStoreRef:
    name: vault-prod
    kind: SecretStore
  target:
    name: api-key
    creationPolicy: Owner
  data:
    - secretKey: key
      remoteRef:
        key: api/production
        property: key

Where this goes next

Rotation is one piece of secret lifecycle management. Next: dynamic credentials per workload, short-lived tokens via OIDC in CI so pipelines never store cloud keys, and admission policy that rejects Deployments mounting Secrets older than your rotation SLA. The Advanced secrets path covers injector vs CSI vs ESO trade-offs and building rotation that survives real deploy cadences.

Related posts