Snapshots, upgrades, and DR

Raft snapshots and rehearsed recovery.

In short. A Vault cluster that has never restored a snapshot is a rumor of durability. Raft integrated storage gives you replication across nodes, not a substitute for backups.

A Vault cluster that has never restored a snapshot is a rumor of durability. Raft integrated storage gives you replication across nodes, not a substitute for backups. Upgrades, bad policies, and regional loss still need a rehearsed path: snapshot, restore, verify, and only then celebrate.

This lesson closes the production loop: take consistent raft snapshots, store them like recovery keys, practice restore into a scratch cluster, plan upgrades without quorum loss, and know what DR means when auto-unseal depends on a KMS in one region.

Raft snapshots

Use vault operator raft snapshot save against a healthy active node. Do not cold-copy disk files from a live data directory and trust them. Schedule snapshots, encrypt them at rest in the backup store, and restrict who can read them — a snapshot is a clone of your secret plane.

vault operator raft snapshot save /var/backups/vault/raft-$(date +%F).snap
ls -lh /var/backups/vault/*.snap
vault operator raft list-peers

-rw------- 1 vault vault 42M Jul 24 02:00 raft-2026-07-24.snap
Node     State     Voter
vault-1  leader    true
vault-2  follower  true
vault-3  follower  true

Restore drill

Restore into an isolated lab cluster with the same seal configuration. vault operator raft snapshot restore -force is destructive — never point it at production casually. After restore, verify seal type, policies, auth methods, and a sample secret read. Time the drill; write down surprises.

Upgrades without drama

Upgrade standbys first, then the leader (or follow current HashiCorp guidance for your version). Maintain quorum throughout. Read changelog for storage/autopilot notes. Never leap across multiple major versions in one weekend without intermediate checks. Pair upgrades with a fresh snapshot taken immediately before.

Going deeper

Performance replication and disaster recovery replication (Enterprise) are different from raft snapshots. Open-source teams still need snapshots plus a documented rebuild. Know which license features you have before promising RPO numbers to leadership.

Seal locality bites during regional DR: if auto-unseal depends on a KMS key only in region A, region B restore stays sealed until you can reach that KMS or migrate seal. Design KMS multi-region keys or a Transit seal strategy before the outage.

Upgrade rehearsals belong next to restore drills. Snapshot, upgrade a scratch cluster through the same version jump, run smoke tests (OIDC login, KV read, database cred mint, PKI issue). Only then touch production waves.

Try this

On a disposable three-node lab, snapshot, write a canary secret, restore an older snapshot to a fourth scratch node (or rebuilt lab), and confirm the canary is absent while an older key returns.

vault operator raft snapshot save /tmp/before.snap
vault kv put secret/canary v=1
# rebuild scratch from /tmp/before.snap per docs, then:
vault kv get secret/canary || true

Error: ... path not found
# SUCCESS — restore point predates canary

Takeaway

Raft HA is not a backup strategy. Snapshot, encrypt, restrict, rehearse restore, and plan upgrades around quorum and seal locality. That is production Vault operations.

You now have the baseline path from -dev to a defendable production cluster — continue into advanced secrets management for namespaces, replication, and deeper identity.

Related