Courses Vault from dev to productionRotation and break-glass

Rotation and break-glass

Root credential rotation and the sealed-glass drill.

In short. Vault rotation and break-glass: root credential rotation plus the sealed-glass recovery drill.

Advanced25 min · lesson 5 of 5

Break-glass is the fire axe behind glass on the wall: you hope never to touch it, but the day the building is burning is the wrong day to discover the glass is bulletproof. In Vault, the axe is the root token and the Shamir key shares that can regenerate it. Everything else in this course hands out short-lived, self-expiring credentials; this lesson is about the handful of secrets that can't expire, because they sit above the system that expires everything else. You rotate those on a schedule, and you rehearse pulling them under pressure. Get it wrong and a rotation locks you out of your own vault; get it right and a compromised operator is a shrug, not an outage.

There is no root token at rest

A live root token is a single string that bypasses every policy, every lease, and every expectation your audit log has. It should not exist while you sleep. The pattern is to generate a root token only for the minutes you actually need it, then revoke it. When it's gone, you get it back through generate-root, which forces a quorum of unseal-key holders (or recovery-key holders, under the auto-unseal you saw in lesson 0) to cooperate — no single operator can resurrect root alone. The drill issues a one-time password and a nonce; each holder submits a share against that nonce, and the final output is a root token encrypted to the OTP, so it never travels in cleartext. Store the shares split across people and locations, so assembling them is a deliberate, witnessed act.

regenerate root, then revoke it

# Kick off the drill: prints a Nonce and a one-time password (OTP)
vault operator generate-root -init
 
# Each key holder runs this and pastes their unseal/recovery key share
vault operator generate-root -nonce=<nonce>
 
# After enough shares, you get an Encoded Token — decode it with the OTP
vault operator generate-root \
  -decode=<encoded-token> \
  -otp=<otp>
 
# Do the one privileged task, then destroy the token immediately
vault token revoke -self
 
# Started a drill by mistake? Abort it
vault operator generate-root -cancel

Rotate the keyring; reshuffle the shares

Two different rotations get confused. vault operator rotate advances Vault's internal encryption keyring so new writes use a fresh key — it's cheap, needs no quorum, and you can cron it; old key terms stick around only to decrypt old data. vault operator rekey is the heavy one: it mints a brand-new set of Shamir unseal shares (or, under auto-unseal, recovery shares) and requires a quorum of the current shares to authorize. Reach for rekey when a key holder leaves the company, when a share may have leaked, or when you want to change the share count or threshold. Rotating the keyring protects data-at-rest going forward; rekeying protects the ability to unseal at all. Confuse them and you'll cron the wrong one and wonder why nothing changed.

rotate the keyring vs rekey the shares

# Cheap, quorum-free: advance the encryption keyring
vault operator rotate
vault read sys/key-status          # current key term + install time
 
# Heavy: mint new unseal shares. Needs a quorum of the CURRENT shares.
vault operator rekey -init -key-shares=5 -key-threshold=3
vault operator rekey -nonce=<nonce>   # each holder submits a current share
 
# Under auto-unseal, the Shamir shares are recovery keys — target them:
vault operator rekey -target=recovery -init -key-shares=5 -key-threshold=3

Rotate the credentials engines hold

The dynamic-secrets and PKI engines from earlier in this course each hold one privileged credential of their own: the database superuser Vault logs in as, or the cloud key it uses to mint short-lived keys. That bootstrap credential is the single long-lived secret in the whole chain, so you hand it to Vault once and then have Vault rotate it to a value no human ever sees. After rotate-root, only Vault knows the password and you can never read it back out — which is exactly the point. Schedule it: a quarterly rotate-root means a leaked bootstrap password has a short shelf life.

rotate the root credential an engine uses

# The Postgres/MySQL superuser configured on the database engine (lesson 2)
vault write -force database/rotate-root/my-postgres
 
# Cloud engines expose the same pattern for their root key
vault write -force aws/config/rotate-root
vault write -force gcp/config/rotate-root

The sealed-glass drill

1Seal the glass

Shamir shares split to N holders; no root token at rest

2Break on incident

Quorum meets, submits shares to generate-root

3Act with root

Decode the token with the OTP; do the one task

4Reseal

Revoke the token, rotate the keyring, rekey if a holder left

The drill is only real if you rehearse it before the incident, not during it.

rotate-root can lock you out for good

rotate-root immediately changes the credential to a random value and never returns it. If that database superuser or cloud key is shared with anything outside Vault — a monitoring job, a second admin, Terraform state, your break-glass DBA login — every one of those breaks the instant you run it, and there is no way to recover the old password. Before rotating, give Vault its own dedicated account with no other consumers, and keep a separate emergency admin account that Vault does not manage. Same discipline as the key shares: the whole point of break-glass is that exactly one sealed path exists, and you have already tested that it opens.

PKI and short-lived certificatesFinish · self-test

How secrets leakSecrets management foundations The secrets threat model & secret-zeroAdvanced secrets management HashiCorp Vault cheat sheetCheat sheet Vault & secrets management interview questionsInterview prep

Rotation and break-glass

There is no root token at rest

Rotate the keyring; reshuffle the shares

Rotate the credentials engines hold

Related