Dev server to HA raft

Storage, unseal, and what breaks when a node dies.

In short. HashiCorp Vault HA with Raft storage: unseal flow, quorum, and what breaks when a Vault node dies.

A vault server -dev is a driving-school car: it starts unsealed, keeps everything in memory, prints your root token straight to the terminal, and forgets all of it the moment you close the door. Perfect for learning, useless for anything you would page someone over. Production Vault is the opposite on every axis — durable storage on disk, a sealed vault you must deliberately open, and multiple nodes that vote before they trust a write. This lesson takes you from that toy to a three-node cluster on integrated (raft) storage, and shows exactly what happens when one of those nodes dies.

Trade -dev for a real storage backend

The dev server hides three decisions you now have to make yourself: where data lives, how the node is reached, and how it gets unsealed. Integrated storage — Vault's built-in raft — is the modern default; it replicates data across the cluster itself, so you no longer run a separate Consul. Two address fields matter more than people expect. api_addr is the address clients and peers are redirected to for this node; cluster_addr (port 8201) is the private server-to-server channel raft uses for replication and request forwarding to the leader. Get these wrong — pointing at localhost, or the wrong hostname behind a load balancer — and nodes will initialize fine but silently fail to join or to forward requests to the leader.

# /etc/vault.d/vault.hcl
storage "raft" {
  path    = "/opt/vault/data"
  node_id = "vault-1"            # unique per node
 
  # add leader_ca_cert_file below if the leader's TLS cert
  # is signed by a private/internal CA (not in the system trust store)
  retry_join { leader_api_addr = "https://vault-2.internal:8200" }
  retry_join { leader_api_addr = "https://vault-3.internal:8200" }
}
 
listener "tcp" {
  address       = "0.0.0.0:8200"
  tls_cert_file = "/etc/vault.d/tls/vault.crt"
  tls_key_file  = "/etc/vault.d/tls/vault.key"
}
 
# auto-unseal so a reboot doesn't page a human (see next section)
seal "awskms" {
  region     = "us-east-1"
  kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/abcd-1234"
}
 
api_addr     = "https://vault-1.internal:8200"  # client / standby reachability
cluster_addr = "https://vault-1.internal:8201"  # raft replication channel

Initialize and unseal — once, and deliberately

Initialization happens exactly once for the whole cluster, against a single node. vault operator init generates the root key (older docs call it the master key), splits it with Shamir's algorithm into unseal keys, and hands back the initial root token. Nothing is readable until enough key holders unseal: with -key-threshold=3 you feed three of the five shares before the vault opens. Doing that by hand on every reboot does not scale, so production adds a seal stanza — AWS KMS, GCP KMS, or a Transit mount on another Vault — that unseals automatically at startup. With an auto-unseal seal the Shamir key flags no longer apply; you size the key set with -recovery-shares/-recovery-threshold, and init returns recovery keys (used later for operator generate-root and seal migration) instead of unseal keys.

# run ONCE, against a single node
 
# Shamir seal (no seal stanza): split the root key into 5 unseal keys
vault operator init -key-shares=5 -key-threshold=3
#   prints 5 unseal keys + initial root token
#   supply 3 of 5 keys to open the vault:
vault operator unseal   # run 3x, one key per holder
 
# Auto-unseal (awskms seal stanza): use RECOVERY flags, not key flags
vault operator init -recovery-shares=5 -recovery-threshold=3
#   prints 5 recovery keys + root token; unseal is automatic on boot
 
vault status
# Sealed        false
# HA Enabled    true
# HA Mode       active        # or 'standby' on the other nodes

Add nodes: raft join and quorum

Extra nodes start empty and sealed, then join the existing cluster. The clean way is the retry_join blocks in each node's config (shown above), so a rebooted node re-finds its peers on its own; you can also join imperatively with vault operator raft join. Once joined and unsealed, a node becomes a voter and takes part in electing the leader and committing writes. raft list-peers shows who is in the cluster and who can vote; autopilot handles voter promotion and dead-server cleanup automatically. Quorum is the whole point of running more than one node: raft commits a write only when a majority — (n/2)+1 — of voters acknowledge it, which is also what protects you from a split brain.

# if you didn't use retry_join, join explicitly against any live node:
vault operator raft join https://vault-1.internal:8200
 
# confirm membership and who can vote
vault operator raft list-peers
# Node     Address                  State     Voter
# vault-1  vault-1.internal:8201    leader    true
# vault-2  vault-2.internal:8201    follower  true
# vault-3  vault-3.internal:8201    follower  true
 
# autopilot: quorum, healthy voters, dead-server cleanup
vault operator raft autopilot state

Related