Vault internals: storage, seal, and replication

Raft, the seal/unseal barrier, auto-unseal, and HA/DR/performance replication.

It is 2 a.m., Vault is down, and someone on the bridge call asks the two questions that actually matter: is our data encrypted on disk, and how do we get it back? If you cannot answer both from memory, you are running a secrets platform you do not understand. We use HashiCorp Vault as the worked example because its shape repeats in every serious secrets system: storage sitting behind an encryption barrier (everything locked at rest until someone unlocks it), a master key ceremony that does the unlocking, and replication for HA (high availability, surviving a dead node) and DR (disaster recovery, surviving a dead region). Learn the seal and the replication modes once and the rest transfers.

What "sealed" actually means

Every byte Vault writes down goes through an encryption barrier first. Data is encrypted with a barrier key before it touches storage, so whatever you store on (Raft, Consul, S3) only ever holds ciphertext. The barrier key is itself encrypted by the master key, and the master key is the thing that unsealing rebuilds. A Vault that has just started is sealed: it holds all the ciphertext and none of the key, so it can decrypt nothing and answers no requests.

Plaintext lives in exactly one place, the leader's memory, and only while that leader is unsealed. Restart the process and you are back to sealed. That is why either auto-unseal or a rehearsed Shamir ceremony has to exist before production traffic arrives, not after.

Who holds the combination: Shamir or a KMS

Out of the box, the master key is split by Shamir's Secret Sharing (a scheme that cuts one secret into pieces, where any agreed number of pieces rebuilds it) and the pieces go to different people. Three of five holders show up, hand over their shares, Vault unseals. For a break-glass root ceremony that is exactly right. As a daily routine it is miserable, because every restart needs humans awake. Auto-unseal hands the wrapping of the master key to a cloud KMS (key management service, a hosted box that holds keys and performs encrypt and decrypt calls for you) or a hardware security module. Vault asks the KMS to decrypt its stored master key at boot and unseals itself.

You have traded a human ceremony for a hard dependency on one KMS key and the IAM (identity and access management) policy guarding it. That policy is now part of your threat model, sitting right next to Vault itself. Keep the Shamir recovery keys for break-glass anyway, even once auto-unseal handles the ordinary restarts.

storage "raft" {
  path    = "/vault/data"
  node_id = "vault-1"
}
 
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-..."
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault/tls/tls.crt"
  tls_key_file  = "/etc/vault/tls/tls.key"
}

Raft storage and the arithmetic of quorum

Modern Vault clusters keep their own state with integrated Raft storage: an odd number of nodes, three or five, agreeing on every write through a consensus protocol. One node is leader and takes the writes. The others replicate the log behind it. Quorum, the number of nodes that must agree, is (n/2)+1. Three nodes tolerate one failure. Five tolerate two. Fall below quorum and there is no leader and no writes, until enough nodes come back or you run a manual recovery.

Never run an even voter count. Keep api_addr and cluster_addr pointed at addresses the peers can genuinely reach. Watch autopilot health so you replace a sick voter before it costs you your failure tolerance.

vault status
vault operator raft list-peers
vault operator raft autopilot state

Sealed          false
HA Enabled      true
HA Mode         active
 
Node     Address                  State     Voter
vault-1  vault-1.internal:8201    leader    true
vault-2  vault-2.internal:8201    follower  true
vault-3  vault-3.internal:8201    follower  true
 
Healthy         true
Failure Tolerance  1

One cluster, two clusters, many clusters

High availability happens inside a single cluster. Raft elects a leader, and losing a minority of nodes is a shrug. Replication happens between clusters, and it comes in two shapes that people mix up constantly. Disaster recovery replication keeps a warm standby cluster in step with the primary, byte for byte. It serves nothing at all until you promote it, and it exists for one job: failover. Performance replication copies policies and backend configuration to other clusters that then mint their own tokens and leases locally. That one exists for scale and for keeping secrets near the workloads, not for failover.

Confuse the two and you get one of two bad days. Either somebody promotes a performance secondary expecting a clean failover, or the team hardens the primary and leaves the DR cluster's network rules and audit trail as a chore for later. A DR secondary holds every secret, token and lease the primary holds. Guard it exactly as hard.

Snapshots, and the blast radius they carry

A Raft snapshot is a complete encrypted copy of Vault's state. Safe enough sitting in a bucket, and a disaster the moment it sits beside access to the unseal path. Keep snapshots and seal keys in separate blast radii, always. Storage sees only ciphertext, true, but the people who run your backups and the bucket that holds them are now part of your threat model.

Restore a snapshot into a scratch cluster every quarter. A backup nobody has restored is a wish, not a control. Run the restore drill next to a DR promotion rehearsal so the on-call rotation learns which of the two runbooks applies to which kind of outage.

vault operator raft snapshot save /tmp/vault-$(date +%F).snap
ls -lh /tmp/vault-*.snap

-rw------- 1 vault vault 48M Jul 24 09:14 /tmp/vault-2026-07-24.snap
# encrypted — useless without unseal path; store off-box, never beside recovery keys

The failure modes you will actually meet

Raft quorum keeps split-brain off the table, but a network partition still hurts: the minority side quietly stops accepting writes. A misconfigured cluster_addr throws join failures that look for all the world like a TLS problem, and people burn an hour on certificates before they find it. A performance secondary running drifted policy can mint a token the primary would have refused. Secondaries are security boundaries in their own right, not read replicas you can stop thinking about.

Write down what "Vault is down" means in your shop, because it means three different things: sealed, quorum lost, or audit devices failing closed. Each has its own fix and its own blast radius for the workloads sitting there trying to renew a lease.

vault operator raft snapshot restore /tmp/vault-2026-07-24.snap
vault status

Restored snapshot
Sealed          true
# expected after restore — unseal or auto-unseal before serving traffic

Listeners, TLS, and traffic between clusters

Vault listens on TLS for clients and on a separate cluster port for the Raft traffic between nodes. Bind those listeners to internal networks, terminate TLS with real certificates rather than self-signed leftovers, and write security groups so only the clients you expect can reach :8200. Storage never sees plaintext, but anyone tapping the network still sees request metadata when you skip TLS, and sees tokens outright if something leaks them into headers.

Performance replication puts more traffic on the wire between clusters. Encrypt it, authenticate the peers at both ends, and graph the replication lag. A secondary running hours behind is a failover candidate that will come up holding stale policy and an out-of-date view of every lease.

Record which cluster is primary for each region, which DR secondary pairs with it, and which performance clusters are cleared to serve production traffic. A runbook written during the outage is fiction. Write it while everything is healthy and you can check each address and each token policy path by hand.

Put the seal and recovery procedures in that same bundle so whoever is on call is not hunting through three wikis at 3 a.m.

vault read sys/config/state/sanitized | jq ".listeners,.storage"

{
  "listeners": [{"tcp": {"address": "0.0.0.0:8200", "tls_disable": false}}],
  "storage": {"type": "raft"}
}
# verify TLS enabled and storage type matches your runbook

Treat seal status and Raft peer health as first-class checks, the way you already watch database replication lag. A sealed node is offline for secret reads even though the process is up and the port answers. A cluster that lost quorum is offline for writes even though two nodes still return HTTP 200. Get vault status and vault operator raft list-peers onto the on-call dashboard before anyone declares that Vault is running.

Promote a DR secondary for real, in a game day, at least twice a year. A standby nobody has ever promoted is a wish. Write down who holds the unseal shares or the KMS permissions, how many minutes a promotion takes end to end, and which applications have to pause while lease state catches up.

Try this

Prove seal state, storage type and peer health from live output instead of trusting a dashboard. Use a lab Raft cluster or your non-prod HA pair. Never point this at the only production primary you have.

vault status
vault operator raft list-peers
vault read sys/storage/raft/autopilot/state -format=json | jq ".data.failure_tolerance,.data.healthy"

Seal Type: shamir
Sealed: false
HA Enabled: true
HA Cluster: https://vault-0.vault-internal:8201
HA Mode: active
 
Node           Address                State       Voter
vault-0        vault-0:8201           leader      true
vault-1        vault-1:8201           follower    true
vault-2        vault-2:8201           follower    true
 
2
true
# failure_tolerance 2 on a 5-voter set; 1 on a 3-voter set

Takeaway

Vault's security story comes down to the seal barrier plus honest arithmetic: ciphertext on disk, the master key in memory only while unsealed, and a quorum count that decides whether writes survive losing a node. Auto-unseal moves that trust into a KMS key policy rather than removing it, so defend that key the way you defend root.

Next: pick Shamir or auto-unseal per environment and be able to say why, count your Raft voters and make the number odd, then write the DR promotion runbook while nothing is on fire.

Related