ansible-vault workflows

Encrypting files and single values, key rotation.

A bank does not hand you a safe deposit box and then tape the key to the lid. Ansible Vault is that box for your repository. You seal a secret inside, the sealed box goes into git where everyone can see that it exists, and only someone holding the key can open it. ansible-vault is the command that ships with ansible-core to build those boxes and open them again. It encrypts whole YAML files (YAML is the indented text format Ansible uses for its data files) or single values inside them, using AES-256 (Advanced Encryption Standard with a 256-bit key). The encryption is symmetric, meaning the same password both seals the box and opens it. The result is ordinary printable text, so git treats a vaulted file like any other file. It clones. It diffs. It merges, badly. What it never does is tell anyone what is inside. Every command and message below comes from ansible-core 2.16.

The mistake this prevents is the one you cannot take back. A plaintext password committed to git is permanent. It survives in every clone, every fork, every CI (continuous integration) checkout cache, every laptop backup, and deleting it in a later commit changes nothing at all. The usual breach is boring. A repository gets mirrored somewhere too permissive, somebody searches it for the word password, and that is the whole story. Vault changes what you have to defend. Instead of controlling who can ever read an entire repository, forever, you control one short string.

Sealing a File

Start with ansible-vault create. It opens your $EDITOR (the environment variable naming your preferred text editor) on an empty buffer and writes ciphertext when you save, so no plaintext file ever exists on disk for you to forget about. Use --vault-id from the first day. It stamps a label into the file header and tells Ansible which password this particular file belongs to. The syntax is label@source. Here the source is the word prompt, which means ask me at the terminal. You will swap that for something automated later in this lesson.

ansible-vault create --vault-id prod@prompt group_vars/prod/vault.yml

New vault password (prod):
Confirm new vault password (prod):
 
# $EDITOR opens on an empty buffer. You type two lines, save, and quit:
#   vault_db_password: "hK7-tR2wQ9fLmZ4xB6vN"
#   vault_pg_replication_password: "qX3nD8sV1yTgW5jR7cM2"

cat group_vars/prod/vault.yml
wc -c group_vars/prod/vault.yml

$ANSIBLE_VAULT;1.2;AES256;prod
63636464363364643963616333663035643063326136396134646365646537346261326265333261
3831656132636630306533666633396566353037303864330a346439383565636633666664633531
33626466366565623133643962373434336339346231346532323835323737653132343733386461
6236643366373965650a393530643031383938623834663165346132653866383966386362343037
63353862356633383235336231323262366261303238613361393162646466306436363339663066
35313839336130326663346538343561383534393937363231303939666437366636353834643763
37663462363030396162616334333662633238326533353764343638333966663662386561396538
37633339386334333639386631646365313933636134396330663330393565373263643531346434
31346364363533323662613130353930636432386665353535373038366138333537
748 group_vars/prod/vault.yml

What Is Actually in That Blob

The first line is the map to everything else. Four fields separated by semicolons: the marker $ANSIBLE_VAULT, the format version (1.2 when a vault id label is present, 1.1 when it is not), the cipher name, and the label. Everything below that line is one very long hexadecimal string, chopped into rows of 80 characters.

Your password is not the key. A key-cutting machine takes a pattern and a blank and cuts one specific key; the same pattern against a different blank cuts a different key. Ansible feeds your password plus a fresh 32-byte random salt (the blank) into PBKDF2-HMAC-SHA256 (Password-Based Key Derivation Function 2, a deliberately slow hash whose whole job is to make guessing passwords expensive) and runs it 10,000 times. Out come 80 bytes, sliced into three pieces: a 32-byte AES key, a 32-byte key for the integrity check, and a 16-byte counter block. The content is encrypted with AES-256 in CTR (counter) mode, and then an HMAC-SHA256 (Hash-based Message Authentication Code, a checksum only someone holding the key can produce) is computed over the resulting ciphertext. Encrypt first, authenticate second, which is the order you want. Coming back the other way, Ansible verifies that checksum before it decrypts a single byte, so a truncated file or a mangled git merge fails loudly instead of quietly handing your playbook garbage.

Then the whole thing is written out in hexadecimal twice over. Salt, checksum and ciphertext are each turned into hex, joined with newlines, and that text is turned into hex again. Every byte of ciphertext therefore costs four bytes of file. That is why 96 bytes of YAML became 748 bytes on disk above: the plaintext is first padded out to a multiple of 16 bytes (112 here), then four file bytes per ciphertext byte, plus roughly 300 bytes of fixed overhead for the salt, the checksum, the header line and the line breaks. It is also why changing one character rewrites every single line. A fresh salt is drawn on every encryption, so every derived key changes, so the entire blob changes. Whole-file vaults and git diff are natural enemies.

There is a setting that looks like it fixes the diff churn, and you should leave it alone. vault_encrypt_salt in ansible.cfg pins the salt to a fixed value, and with a fixed salt the same input always produces the same ciphertext, so diffs go quiet. A fixed salt also means a fixed derived key and a fixed counter block for that password. Counter mode turns the key and the counter into a keystream that gets combined with your data, and reusing one keystream across two files is the oldest break in the book: combine the two ciphertexts and the encryption cancels out, leaving the two plaintexts mixed with each other. Vaulted vars files start with predictable text, so that is usually enough to read both. Noisy diffs are the price of the salt doing its job.

Editing Without Spilling Plaintext

Four subcommands cover the daily work. create makes a new sealed file. edit decrypts, opens your editor, and re-encrypts when you save. view pages it read-only. encrypt seals a file that already exists, in place, or somewhere else if you pass --output. One detail worth knowing: the vault command sets a restrictive umask for its own run, so files it creates land at mode 0600, readable by you and nobody else. There is a fifth subcommand, decrypt, and you should treat it as radioactive.

Checking that a vault file is really protected by the key you think is protecting it takes one command. Try to open it with the wrong identity and read what comes back. Nothing has been wired into ansible.cfg yet, so the only identities in play are the ones typed on the command line.

# with the prod identity
ansible-vault view --vault-id prod@prompt group_vars/prod/vault.yml
 
# with only the dev identity loaded
ansible-vault view --vault-id dev@~/.vault/dev-pass group_vars/prod/vault.yml

Vault password (prod):
vault_db_password: "hK7-tR2wQ9fLmZ4xB6vN"
vault_pg_replication_password: "qX3nD8sV1yTgW5jR7cM2"
 
ERROR! Decryption failed (no vault secrets were found that could decrypt) on group_vars/prod/vault.yml for group_vars/prod/vault.yml

Sealing One Value Instead of a Whole File

A whole-file vault is a sealed envelope with no window. Nobody reviewing the change can tell whether it holds two variables or twenty, and approving that diff is approving noise. encrypt_string cuts a window into the envelope. It encrypts one value and prints a YAML snippet you paste into an otherwise readable vars file.

Type the secret into standard input, never onto the command line. --stdin-name NAME names the variable and reads its value from stdin. Hand the same secret over as a plain argument and it lands in your shell history file and shows up in ps output, the list of running processes, which every other user on that machine can read for as long as the command runs. There is one catch with stdin: whatever you type gets encrypted exactly as typed. Press Enter and then Ctrl-D, and that newline becomes part of your API token. Press Ctrl-D twice with no newline, and it does not. The prompt says so out loud, and people still get caught by it.

ansible-vault encrypt_string --vault-id prod@prompt --stdin-name vault_api_token

New vault password (prod):
Confirm new vault password (prod):
Reading plaintext input from stdin. (ctrl-d to end input, twice if your content does not already have a newline)
ghp_Zt4qXf81LmQwB7nR3vKd0aYs
vault_api_token: !vault |
          $ANSIBLE_VAULT;1.2;AES256;prod
          66653561376365656534646638663331373130313034326536336437633534613530616334643761
          3232626537643432653638353663353335316634356332340a383439356661663839666562353631
          36383366633065663432336135666631353135376134656666336432353235633337336234663631
          6137643131313031340a626333353330643462393236623962333130376262613836333130336431
          62386266316237643833363331663438306162326434663033343439313563643065
Encryption successful

# Plain values stay diffable, greppable and reviewable.
db_host: db01.prod.internal
db_user: appsvc
db_password: "{{ vault_db_password }}"    # lives in the vaulted file next door
 
# One value sealed in place. Note what is NOT hidden: the variable name,
# the file name and the vault id label are all plaintext. Vault encrypts
# values, not metadata.
vault_api_token: !vault |
          $ANSIBLE_VAULT;1.2;AES256;prod
          66653561376365656534646638663331373130313034326536336437633534613530616334643761
          3232626537643432653638353663353335316634356332340a383439356661663839666562353631
          36383366633065663432336135666631353135376134656666336432353235633337336234663631
          6137643131313031340a626333353330643462393236623962333130376262613836333130336431
          62386266316237643833363331663438306162326434663033343439313563643065
 
api_token: "{{ vault_api_token }}"

Vault IDs, and Why the Label Is Not a Lock

One password for everything means one blast radius: whoever can read dev can read production. Vault ids cut that up. Every blob carries a label, and you hand Ansible a set of label@source pairs for each run. Wire them into ansible.cfg so nobody has to remember flags, and so a forgotten flag cannot quietly pick the wrong key.

[defaults]
# label@source pairs, tried in order.
# A source is: prompt, a path to a password file, or a path to an executable.
vault_identity_list = dev@~/.vault/dev-pass, prod@./bin/prod-vault-client.sh
 
# Which identity to encrypt with when several are loaded.
# --encrypt-vault-id on the command line overrides this.
vault_encrypt_identity = prod
 
# Only try the password whose label matches the file header.
# ansible-core defaults this to False. Turn it on.
vault_id_match = True

That last setting matters far more than it looks. vault_id_match is off by default, which makes the label in the header a hint about ordering rather than a rule. Ansible tries the matching secret first, and when that fails it works through every other secret it happens to be holding. Two things follow. A file stamped prod opens with the dev password if somebody encrypted it with the dev password, because nothing binds the label to the key: the label sits outside the encrypted payload and outside the checksum, and you can rewrite it in a text editor. And an engineer running a playbook with both identities loaded is one wrong inventory path away from decrypting production data on a laptop. Setting vault_id_match = True makes the label authoritative. Wrong label, hard failure, no quiet fallback.

You can watch the difference. Take a file whose header says prod but which was actually encrypted with the dev password, which is exactly what happens when somebody copies a vars file between environments and re-seals it with whatever key was to hand.

head -1 group_vars/prod/oops.yml
 
# ansible-core default: the label is advisory
ANSIBLE_VAULT_ID_MATCH=False ansible-vault view group_vars/prod/oops.yml
 
# with matching enforced
ANSIBLE_VAULT_ID_MATCH=True ansible-vault view group_vars/prod/oops.yml

$ANSIBLE_VAULT;1.2;AES256;prod
 
vault_db_password: "hK7-tR2wQ9fLmZ4xB6vN"
 
ERROR! Decryption failed (no vault secrets were found that could decrypt) on group_vars/prod/oops.yml for group_vars/prod/oops.yml

A Password No Human Ever Types

The third kind of source is the one that makes this work in production. Point a vault id at an executable file and Ansible runs it, taking whatever it prints on standard output as the password. That script can fetch the password from AWS Secrets Manager, from HashiCorp Vault, or from a cloud KMS (Key Management Service, a managed service that holds keys on your behalf). Nobody on the team ever learns the production vault password. Access is governed by IAM (Identity and Access Management, the cloud's permission system) roles you can revoke one person at a time, every fetch lands in the provider's audit log, and rotation happens in one place.

Name the file so the part before any extension ends in -client, and ansible-core treats it as a vault password client script rather than a plain one. The difference is that it runs the script with --vault-id <label> appended, so a single script serves every environment. Exiting with code 2 is a defined signal meaning I have no secret for that id, and Ansible reports that as its own distinct error instead of a generic failure. The file has to be executable, and it should be mode 700 so no other local user can read the fetch logic or run it.

#!/usr/bin/env bash
# chmod 700. The name ends in -client, so ansible-core appends --vault-id <label>.
set -euo pipefail
 
vault_id="default"
while [ $# -gt 0 ]; do
  case "$1" in
    --vault-id)   vault_id="$2"; shift 2 ;;
    --vault-id=*) vault_id="${1#*=}"; shift ;;
    *)            shift ;;
  esac
done
 
case "$vault_id" in
  dev|staging|prod) ;;
  *) echo "no secret configured for vault-id=$vault_id" >&2; exit 2 ;;
esac
 
aws secretsmanager get-secret-value \
  --secret-id "ansible/vault/${vault_id}" \
  --query SecretString --output text

To confirm the plumbing rather than assume it, run with -vvvv and look for the line where ansible-core announces the script it is about to execute. The same verbosity level prints a line for every secret it tried and failed with, which turns a bare decryption failure into something you can act on.

ansible-playbook -i inventories/prod site.yml -vvvv | grep -i 'vault password client'

Executing vault password client script: ./bin/prod-vault-client.sh --vault-id prod

Rotation Is Two Jobs, Not One

ansible-vault rekey re-encrypts files under a new password. You need it when somebody with password access leaves, or when the password has been somewhere it should not have been. Find the files first. Every whole-file vault starts with the same marker at column zero, so grep finds them whatever they are named. Feed that list in with command substitution rather than a pipe into xargs. xargs hands its children an empty standard input, and the new-password prompt needs a real terminal to read from.

grep -rl '^\$ANSIBLE_VAULT' inventories/ group_vars/ host_vars/
 
ansible-vault rekey \
  --vault-id prod@./bin/prod-vault-client.sh \
  --new-vault-id prod@prompt \
  $(grep -rl '^\$ANSIBLE_VAULT' inventories/ group_vars/ host_vars/)

group_vars/prod/vault.yml
host_vars/db01.prod.internal/vault.yml
New vault password (prod):
Confirm new vault password (prod):
Rekey successful

Now drop the ^ anchor and search again. Inline !vault blocks are indented, so the anchored pattern walks straight past them, and rekey cannot touch them anyway. It only accepts a file whose entire content is one vault envelope. Point it at a plaintext vars file holding an inline value and it refuses. Every one of those values has to be regenerated with encrypt_string under the new password, by hand. This is where stale passwords survive rotations.

grep -rlF '$ANSIBLE_VAULT' group_vars/ | grep -v '/vault\.yml$'
ansible-vault rekey --new-vault-id prod@prompt group_vars/prod/vars.yml

group_vars/prod/vars.yml
New vault password (prod):
Confirm new vault password (prod):
ERROR! input is not vault encrypted data.  for /home/ops/infra/group_vars/prod/vars.yml

Understand what rekeying does not do. The old ciphertext is still sitting in git history, and the old password still opens it. Anyone who cloned before the rekey holds a permanent copy. Rekeying answers a leaked vault password only if you also rotate the credentials stored inside: change the database password, revoke the API token, reissue the TLS (Transport Layer Security) key. Treat those as one operation under one ticket. Rekeying on its own changes the lock on a door whose key has already been photocopied.

Where Vault Stops

Vault is encryption at rest and nothing else. There is no record of who decrypted what, no expiry, no leases, no short-lived credentials, and no access control finer than holding a password. Teams running this at scale keep Vault for bootstrap secrets, the ones Ansible needs before anything else exists, and pull runtime secrets from a real secret manager through lookup plugins such as community.hashi_vault.hashi_vault, which is the next lesson. Remember where a lookup runs: on the control node, inside the templating engine, before anything is sent anywhere. The managed host never talks to the secret manager and never decrypts a thing. If what you want is the contents of a vaulted file at task time rather than as variables, ansible.builtin.unvault is the lookup for that, and it reads from the control node's filesystem too.

The second boundary is runtime. Once decrypted, the value is ordinary data. It flows into module arguments, into registered results, into -vvv output, and into whatever your callback plugins ship to a log aggregator. no_log: true on every task that handles a secret is not optional. Be precise about what it buys you, though. It replaces that task's arguments and its result with the text the output has been hidden due to the fact that 'no_log: true' was specified for this result, and it stops the module writing its Invoked with ... line to syslog on the managed host. It does not stop the secret being sent to that host, and it does nothing about a later task that prints the value some other way. In CI, write the vault password to a file with mode 0600 from a protected variable and delete it in a shell trap, or point a vault id at a script that queries the runner's own secret store. Ansible will not police that file the way SSH (Secure Shell) refuses to use a world-readable private key. A 0644 password file works fine and nothing warns you.

Add one check to your pre-commit hooks today. For every path matching *vault*.yml, fail the commit unless the first line is $ANSIBLE_VAULT. Five lines of shell, and it catches the one mistake that no amount of rekeying will ever undo. Then look at the other half of the problem: that decrypted password is about to become a login on a real machine, and become decides how much authority it carries when it lands.

Related