Audit, tamper-evidence, break-glass & DR

Audit devices, HMAC log integrity, recovery keys, and the DR drill.

After a breach, the questions come fast and they are always the same. Who could read production secrets? Who actually did? And can you prove nobody edited the record afterwards? The people running the incident add one more: if the whole cluster is gone, can we get back to a known-good state? Every one of those answers rests on three things. Audit logs you can trust, a break-glass path (emergency access that is time-boxed and recorded) you have rehearsed, and a disaster-recovery drill you have actually run. A secrets manager you cannot audit or recover is a liability wearing the costume of a control.

What Vault writes down, and what it never writes

Turn on an audit device and Vault records every request and every response that passes through it. It does that without printing a single secret. Sensitive values go through an HMAC (hash-based message authentication code, a one-way fingerprint computed with a key that only this Vault cluster holds), so the line proves a secret was read, by whom, at which path, while the value itself never appears. That is what makes the stream safe to ship into a SIEM (security information and event management, the central system your security team searches logs in). Vault also fails closed here. If it cannot write to at least one audit device, it stops answering requests rather than hand out secrets blind.

That is the right default for a system of record, and it comes with a bill you have to plan for: a broken log pipeline can take Vault down with it. So run more than one sink. Enable at least two devices, file plus syslog, or file plus a socket to your log aggregator. Then treat disk space and pipeline health as first-class SLOs (service level objectives, the numbers you actually page someone about), because a full disk on the audit host is now a production outage.

vault audit list
vault audit enable file file_path=/var/log/vault/audit.log
vault audit enable syslog tag="vault" facility="AUTH"
vault read database/creds/payments-ro

No audit devices enabled
Success! Enabled the file audit device
Success! Enabled syslog audit device
# log records: read database/creds/payments-ro; password=hmac-sha256:5f2e...

The log has to survive your own admins

An audit log an insider can quietly edit proves nothing. A camera pointed at the vault door is worthless if the guard keeps the only tape. So push audit events off the box in real time, into storage that appends and refuses deletes: a SIEM index with locked retention, or an S3 bucket with object lock (the Amazon setting that rejects deletion until a timer expires). Then watch for the tells. Gaps in the sequence numbers. An audit device that got disabled. A log file that suddenly got shorter.

The threat model here names privileged insiders on purpose, so the control that proves integrity has to sit outside the trust boundary of the people it watches. The practical version is a split of hands. Anyone who can turn off an audit device should not hold delete rights on the immutable SIEM index or on the object-lock bucket receiving the stream. Two roles, two accounts, two sets of credentials.

tail -1 /var/log/vault/audit.log | jq ".type,.auth.display_name,.request.path,.response.data"
# forwarder ships same line to SIEM — compare sequence numbers for gaps

"response"
"oidc-alice"
"database/creds/payments-ro"
{"password":"hmac-sha256:5f2e8a...","username":"hmac-sha256:9b3c..."}
# HMAC proves access; cleartext never in log

Two emergencies worth rehearsing

Break-glass is the "we need root right now" emergency, and the path is deliberately awkward. A quorum of recovery-key holders (enough of the people holding key shards to reach the required count) come together to generate a temporary root token with vault operator generate-root. One person uses it while a second person watches. It gets revoked the minute the emergency ends. Root is the fire axe behind glass, not a tool that lives on the desk.

The second emergency is "the primary is gone." You promote the DR (disaster recovery) replica cluster, repoint clients at the new address, and confirm secrets actually flow. Neither path works if its first real run happens during the incident. Put both on a calendar and repeat them until the steps live in people's hands instead of a Google Doc nobody has opened since onboarding.

Snapshots, recovery keys, and where you keep them

Auto-unseal takes the daily unseal ceremony off your plate, and it does not make the keys vanish. Recovery keys still exist, and anyone holding a quorum of them can mint root or unseal a restored backup. Treat them like the Shamir shares they replaced (a key cut into pieces so that some minimum number must be brought together to rebuild it): custody split across people who do not report to each other, tamper-evident storage, a written ceremony with witnesses. Take Raft snapshots on a schedule, since Raft is the built-in storage that keeps Vault's data in sync across its nodes, encrypt them, and park them somewhere that shares no blast radius with the seal and recovery keys.

Then restore one. Pick a date, load a snapshot into a scratch cluster, and find out whether it comes back. A backup nobody has restored is a hope, and a backup sitting beside the keys that unseal it is one break-in away from total compromise. The same care applies to DR secondaries. They hold the same secrets as the primary, so their audit trail and network path deserve identical guarding.

What the auditor wants, what the responder wants

An auditor wants three things from you: access proof with HMAC'd values instead of cleartext, two audit sinks instead of one, and an off-box copy the Vault team cannot delete. A responder wants a one-page break-glass card and a DR runbook somebody exercised this quarter. Build both before the week you need them. Your secrets platform is production-grade when you can prove who read what last March and still serve credentials an hour after losing the primary region.

Keep the artifacts as you go: sample audit lines, drill reports, ticket links. "We have audit enabled" is a claim. A dashboard showing six months of unbroken ingest, plus an alert you have tested that fires when an audit device is disabled, is evidence.

vault operator generate-root -init
# recovery key holders submit shards interactively
vault operator generate-root -decode=$OTP -otp=$OTP
vault token revoke $GENERATED_ROOT_TOKEN

Nonce         abc123...
OTP           def456...
Encoded token hvs.root...
# use root under observation for emergency only
Success! Revoked token hvs.root...
# root destroyed — return to normal scoped tokens

vault operator raft snapshot save /tmp/vault-$(date +%F).snap
vault operator raft snapshot restore /tmp/vault-2026-07-24.snap
vault status

-rw------- 1 vault vault 48M /tmp/vault-2026-07-24.snap
Restored snapshot
Sealed          true
# expected — unseal before serving; drill this quarterly

Mapping the trail to the framework, then keeping it honest

Do the mapping once, in writing, field by field. The display_name and policies values in an audit line are your SOC 2 evidence (SOC 2 is the security audit report most US vendors are asked for) showing who reached what under which permissions. The HMAC'd response bodies are your PCI evidence (PCI is the card-industry rulebook for handling payment data) that cleartext never landed in a log. Fail-closed behavior answers the ISO 27001 control about keeping the audit trail available. Hand an auditor that table with a few sample queries and the ad-hoc questions mostly stop.

Continuous assurance is the boring half. Automated checks that both devices are still enabled, that both sinks are still receiving, and that the list of recovery-key holders has not changed without a ticket behind it. A weekly script that lists the audit devices and compares them against the expected set is how you catch the "temporary" disable from a Tuesday debugging session that quietly became permanent.

vault audit list -detailed
vault read sys/health | jq ".sealed,.standby,.performance_standby"

File Audit Device:
  Path: /var/log/vault/audit.log
Syslog Audit Device:
  Tag: vault
 
false
false
false
# SIEM should show steady event rate — alert on gaps or disable events

Audit logs and snapshots need different retention, so give them separate buckets. Audit lines pile up for years, because an investigator may need last spring. Snapshots roll weekly under an encrypted lifecycle rule, because a nine-month-old copy of every secret you hold is mostly a liability. Point one S3 lifecycle policy at both and you will eventually delete the history of an incident you have not discovered yet.

Make every drill produce paper. Who ran the promotion, the timestamp, which secrets got smoke-tested, and how long it took to issue the first working dynamic credential afterwards. That last number is the one leadership asks about. An auditor accepts "we have DR" when you show last quarter's drill report with names on it, and not when you point at a standby cluster nobody has promoted in two years.

Wire break-glass root usage to the same alert severity as a honeytoken (a fake credential planted so that any use of it means someone is poking where they should not). Both mean a human reached for extraordinary access. If break-glass fires silently, you learn about the abuse months later, in a log review nobody scheduled.

vault read sys/replication/dr/secondary
vault operator raft list-peers

mode         dr-secondary
cluster_id   dr-cluster-1
 
Node     Address              State
vault-1  dr-1.internal:8201   leader
# rehearse promotion on this cluster — not just read config

Decide up front which paths earn an alert rather than a row on a dashboard. Encrypt and decrypt calls on the transit engine, policy writes, auth method tuning, anything under a production secret path. Then practice reading the trail during a game day, well before the night it matters, because query syntax you have never typed is painfully slow at 2 a.m. An audit log nobody queries is expensive decoration.

One caveat to hold onto. The HMAC hides the secret value, and it does not hide the metadata wrapped around it. Path, entity, timestamp, frequency: all of it stays readable. Someone with only the log can tell that the payments service started pulling a fresh database credential every four minutes at 1 a.m. Guard the log sink with the seriousness you give the secrets engine itself. Two audit devices cover you when one sink dies, and Vault can also refuse requests when an audit write fails, depending on how you configured it. Learn which mode your cluster runs in on a calm afternoon, not during the outage.

Leave the compliance language for last. Get the queries working, prove the trail holds, then translate it into the framework's words. What an auditor is really asking for is evidence of access review and rotation, and your saved SIEM searches plus the tickets they link to are that evidence.

Try this

On a lab Vault, switch on a file audit device, read a secret, and go find yourself in the log. Then run something that should be refused and watch the deny line show up.

vault audit enable file file_path=/vault/audit/audit.log
vault kv get secret/payments/db
vault policy read payments-read >/dev/null
jq -r 'select(.request.path|contains("payments")) | [.time,.type,.auth.display_name,.request.path,.error] | @tsv' /vault/audit/audit.log | tail -5

path                    file/
description             n/a
options                 map[file_path:/vault/audit/audit.log]
...
 
2026-07-24T01:12:01Z  response  alice@acme  secret/data/payments/db  null
2026-07-24T01:12:44Z  response  ci-runner   secret/data/payments/db  permission denied
# second line is the deny you want to alert on when path is prod

Takeaway

Audit is how you prove least privilege held, and how you catch the moment it did not. Two sinks, a copy of the log your own operators cannot delete, and alerts on the handful of high-risk paths turn Vault from a box that hands out passwords into a system that can account for every password it handed out.

Next: check that production has at least one audit device actually writing, that your SIEM parses those lines into fields instead of storing them as blobs, and that a named person reviews policy writes and decrypt spikes every week.

Related