Ansible Vault & secrets

Encrypt secrets in your repo.

A Git repository is a filing cabinet that nobody ever empties. Delete a password from a file today, commit the change, and the old value is still sitting there: in the history, in every clone on every laptop, in every fork, in last night's backup, and in the cache of whatever CI (continuous integration, the robot that rebuilds and retests your code every time somebody pushes) runner touched the branch. Ansible Vault gets around that by never letting the plaintext in. It turns a secret into a block of hex (a long run of the characters 0 to 9 and a to f) that means nothing without a password, and you commit that block right next to the playbook that uses it.

Be clear about what you are buying. Vault is a padlock on a file, not a secrets manager. There is no rotation schedule, no per-secret access policy, no audit log telling you which engineer read the database password last Tuesday. You get one job done well: credentials that travel with the code, survive code review, and stay unreadable to anyone who clones the repo. For a small team that is often enough. For a bigger one, it is the sensible thing to run until you have somewhere better to keep the secrets.

What Vault Does to a File

Encrypting a file works like sealing a letter inside a thick, unmarked envelope. You cannot read what is inside, and you cannot tell what sort of thing it was. Start with an ordinary variables file and lock it.

cat group_vars/all/vault.yml
 
# encrypt in place: the plaintext is overwritten
ansible-vault encrypt group_vars/all/vault.yml
 
head -3 group_vars/all/vault.yml
wc -l group_vars/all/vault.yml

---
vault_db_password: "s3cr3t-pw"
vault_api_token: "glpat-9f3a1c7e2b4dK"
New Vault password:
Confirm New Vault password:
Encryption successful
$ANSIBLE_VAULT;1.1;AES256
65373830396363323237623739626132376661393033613266626264313438616566323263353937
3662303163663335336439336462613135336433356662310a323363316532356435353938653538
9 group_vars/all/vault.yml

The first line is the only part that is not ciphertext (scrambled output). $ANSIBLE_VAULT is the marker Ansible looks for when it loads a file, 1.1 is the envelope format version, and AES256 names the cipher. Everything below that line is hex, wrapped at 80 characters. Nine lines to hold two short secrets, and none of it tells a reader anything.

Underneath, your password goes through PBKDF2 (Password-Based Key Derivation Function 2, a deliberately slow way of turning a typed password into a cryptographic key). Here that means 10,000 rounds of HMAC-SHA-256 (a keyed fingerprinting function built on the SHA-256 hash), mixed with a salt: 32 random bytes drawn fresh every time and stored in the file itself. Three things come out. A 32-byte key for AES-256 (Advanced Encryption Standard at 256 bits) running in CTR (counter) mode, a second 32-byte key used only for authentication, and the 16-byte block that CTR mode starts counting from. Ansible encrypts first, then computes an HMAC over the finished ciphertext with that second key and stores it alongside. Change one hex character in a pull request and decryption fails loudly instead of quietly handing back garbage.

The random salt has a consequence you will meet on your first commit. Encrypt the same content twice and you get two completely different blobs. Every change to a vaulted file therefore lands as a diff in which every single line changed. A reviewer cannot tell whether you fixed a typo or slipped in five new credentials. That one fact drives most of the conventions below.

The Commands You Will Actually Type

ansible-vault has seven subcommands and you will meet all of them. create opens a new file in $EDITOR (the environment variable naming your text editor, usually vim or nano) and writes ciphertext when you save. encrypt converts an existing plaintext file in place. view prints the plaintext to your terminal without writing it anywhere. edit decrypts to a temporary file, hands that to your editor, then re-encrypts and shreds the temporary copy when you quit. rekey changes the password on one or more files in a single command, which is your rotation story. decrypt turns a file permanently back into plaintext, which you want when retiring a secret and almost never otherwise. encrypt_string is the odd one out and gets its own section.

# read the plaintext without writing it to disk
ansible-vault view group_vars/all/vault.yml
 
# rotate the password across several files at once
ansible-vault rekey group_vars/all/vault.yml roles/db/vars/secrets.yml
 
# encrypting an already-encrypted file is refused, not doubled
ansible-vault encrypt group_vars/all/vault.yml

Vault password:
---
vault_db_password: "s3cr3t-pw"
vault_api_token: "glpat-9f3a1c7e2b4dK"
Vault password:
New Vault password:
Confirm New Vault password:
Rekey successful
New Vault password:
Confirm New Vault password:
ERROR! input is already encrypted

The vault_ Prefix, and the Plaintext File Beside It

Because a vaulted file gives nothing away, the trick is to make it as boring as possible. One file holds nothing but secrets, every variable prefixed vault_. A second, plaintext file sits next to it and maps ordinary names onto those. Picture a locked drawer with a typed index taped to the front: the index tells you what is in there, the drawer still needs a key. Your playbooks, templates and roles only ever mention the names on the index.

---
# Nothing lives here except secrets. One obvious, encrypted place.
vault_db_password: "s3cr3t-pw"
vault_api_token: "glpat-9f3a1c7e2b4dK"

---
# Readable names, encrypted values. Roles and templates use these.
db_password: "{{ vault_db_password }}"
api_token:   "{{ vault_api_token }}"

Two things get better. A reviewer reading vars.yml can see which secrets exist and which one you added, even though the values stay locked away, and that recovers some of what the all-lines-changed diff took from you. And grep -rn db_password roles/ still finds every place the secret is used, because the name is in plaintext everywhere except the one file holding the value. Precedence does not change. Both files sit in group_vars/all/, Ansible loads them exactly as it would if neither were encrypted, and Jinja (the templating language behind the {{ ... }} markers) resolves vault_db_password at the moment the value is needed, so the order the two files load in does not matter.

Encrypting One Value Instead of a Whole File

For two or three secrets scattered through otherwise readable configuration, locking a whole file is heavy-handed. ansible-vault encrypt_string encrypts a single value and prints a YAML (YAML Ain't Markup Language, the indented text format Ansible files are written in) snippet you paste into a plain file. Names and structure stay readable. Only the value is ciphertext.

Do not pass the secret as a command-line argument. It lands in your shell history, and while the command runs it is visible in the process list to every user on the box who types ps. Use --stdin-name instead. It reads the value from standard input (the stream you pipe into a command) and names the variable for you.

# the secret arrives on stdin, so it misses shell history and ps output
printf 's3cr3t-pw' | ansible-vault encrypt_string --stdin-name 'db_password'

New Vault password:
Confirm New Vault password:
Reading plaintext input from stdin. (ctrl-d to end input, twice if your content does not already have a newline)
 
db_password: !vault |
          $ANSIBLE_VAULT;1.1;AES256
          33663261333263653763613037643738363739313532323666353961363063326230383639366333
          3633386139386562323462353561306562633236386262310a613239333863373432373339303132
          61643634643835393037323638613465326361346561613234623035366536653766666536636236
          3535393661336337370a336633343439323135343761616634353032386263376166343130646661
          3030
Encryption successful

!vault is a YAML tag telling Ansible that this string needs decrypting before use, and it works anywhere variables are defined: group_vars, host_vars, vars: inside a play, role defaults. One behaviour catches people out. view and edit only understand whole-file vaults. Point either at a plaintext file that happens to contain inline !vault strings and it refuses, after asking you for the password first. It also prints the filename twice, which is Ansible's doing and not a mistake on your side.

ansible-vault view group_vars/all/vars.yml

Vault password:
ERROR! input is not vault encrypted data. group_vars/all/vars.yml is not a vault encrypted file for group_vars/all/vars.yml

Handing the Password Over at Run Time

At play time Ansible needs that password, and there are three ways to hand it over. --ask-vault-pass prompts you, which is fine at a keyboard and useless in a pipeline. --vault-password-file reads it from a file. The part people miss is what happens when that file is executable: Ansible runs it and takes whatever it prints on standard output as the password. Your CI runner then never stores the vault password on disk at all, because the file is a short script that calls your secrets manager and prints the answer. Get any of this wrong and the run stops with exit code 4, after the play banner and before anything touches a managed host.

# no password supplied at all
ansible-playbook site.yml; echo "rc=$?"
 
# a password, but the wrong one
ansible-playbook site.yml --vault-password-file ~/.dev_pass; echo "rc=$?"
 
# type it yourself
ansible-playbook site.yml --ask-vault-pass

PLAY [Ship app config] *********************************************************
ERROR! Attempting to decrypt but no vault secrets found
rc=4
 
PLAY [Ship app config] *********************************************************
ERROR! Decryption failed (no vault secrets were found that could decrypt) on /srv/deploy/group_vars/all/vault.yml
rc=4
 
Vault password:
 
PLAY [Ship app config] *********************************************************
 
TASK [Render the app config] ***************************************************
changed: [web01]
 
PLAY RECAP *********************************************************************
web01                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

#!/bin/sh
# Executable password "file": Ansible runs it and reads stdout.
# The vault password never lands on the runner's disk.
set -eu
aws secretsmanager get-secret-value \
  --secret-id ansible/vault-password \
  --query SecretString --output text

[defaults]
inventory = inventory.ini
 
# Ansible finds the password by itself, so nobody is tempted to
# paste it into a shell one-liner that lands in history.
vault_password_file = ~/bin/vault-pass.sh

Vault IDs, When One Key Is Not Enough

One password for the whole repository means the contractor who needs the staging database password is holding the production one too. Vault IDs put a label on each encrypted blob, the way a hotel writes a room number on a key fob, so a single run can carry several keys and reach for the right one per file. Encrypt with --vault-id prod@prompt and the header changes: envelope version 1.2, with the label appended in the clear.

printf 'prod-pw' | ansible-vault encrypt_string --vault-id prod@prompt --stdin-name 'db_password'

New vault password (prod):
Confirm new vault password (prod):
Reading plaintext input from stdin. (ctrl-d to end input, twice if your content does not already have a newline)
 
db_password: !vault |
          $ANSIBLE_VAULT;1.2;AES256;prod
          35653465386539636666646564363764333535343039613134346439633237373164616163616161
          6538353839386630353465316635313537333738633832620a333635373830356162626239303363
          63313135663439623664333732353466393331326536396364353562613530353435353437653538
          3432616335376633630a636661656161323336393364323939643734396634376364323735306233
          3234
Encryption successful

At run time you pass one --vault-id per key. label@path reads a file, label@prompt asks you. When more than one identity is loaded and you are encrypting, --encrypt-vault-id says which of them to use. Put the everyday set in ansible.cfg under vault_identity_list and the flags disappear from your muscle memory. Now try opening a production blob with a key that carries a different label.

# this blob was encrypted under the label "prod"
head -1 group_vars/all/vault.yml
 
# open it with a key labelled "staging" that holds the same password
ansible-playbook site.yml --vault-id staging@~/.prod_pass

$ANSIBLE_VAULT;1.2;AES256;prod
 
PLAY [Ship app config] *********************************************************
 
TASK [Render the app config] ***************************************************
changed: [web01]
 
PLAY RECAP *********************************************************************
web01                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Decrypted Is Decrypted

Vault protects a secret while it sits in the repo. The moment a play runs, that secret is a plain string in memory, in module arguments on the target, in task results, and in whatever a callback plugin (the component deciding what Ansible prints and what it ships to a log collector) chooses to show. The most common way a vaulted password escapes is not a broken cipher. It is --diff.

ansible-playbook site.yml --diff

PLAY [Ship app config] *********************************************************
 
TASK [Render the app config] ***************************************************
--- before
+++ after: /root/.ansible/tmp/ansible-local-199jw3udj23/tmp78i0nhzb/app.conf.j2
@@ -0,0 +1,2 @@
+[database]
+password = s3cr3t-pw
 
changed: [web01]
 
PLAY RECAP *********************************************************************
web01                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

There is the production database password, sitting in a pipeline log, in whatever aggregator ships that job's output, readable by everyone with access to the build. Nothing was misconfigured. --diff did exactly what it exists to do. The rendered file also lands on the managed host as plaintext, which is why the mode: on that task matters as much as the encryption did.

no_log: true is the fix, and it works one task at a time. Ansible throws the entire result away, arguments included, and returns a censored placeholder instead.

- name: Ship app config
  hosts: web
  become: true
  gather_facts: false
  tasks:
    - name: Render the app config
      ansible.builtin.template:
        src: app.conf.j2
        dest: /etc/app/app.conf
        owner: root
        group: root
        mode: "0640"        # the rendered file holds the secret in the clear
      no_log: true          # nothing from this task reaches stdout or a callback

# same play, same --diff flag, only no_log added to the task
ansible-playbook site.yml --diff
 
# later, after someone deleted /etc/app on the target: the task now fails
ansible-playbook site.yml --diff

PLAY [Ship app config] *********************************************************
 
TASK [Render the app config] ***************************************************
changed: [web01]
 
PLAY RECAP *********************************************************************
web01                      : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 
PLAY [Ship app config] *********************************************************
 
TASK [Render the app config] ***************************************************
fatal: [web01]: FAILED! => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}
 
PLAY RECAP *********************************************************************
web01                      : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

That censored line is the price, and it is a real one. When a no_log task fails you get no message, no stderr, no module output, only the word censored and a red line. Debugging means reproducing the failure with a fake secret and no_log switched off, on a host whose logs you do not mind. Take the deal for anything holding a credential, but set it on the two tasks that touch the password rather than across the whole play, or you will be blind everywhere at once.

Do not lean on Ansible's built-in scrubbing to save you, because it is narrower than its reputation. Two things happen on their own. When a module declares a parameter as secret in its own argument spec, the way ansible.builtin.user declares password, that value comes back as the literal string VALUE_SPECIFIED_IN_NO_LOG_PARAMETER. And when a module shells out, any argument beginning pass, -pass, --password or --passwd gets masked in the line written to the managed host's system log, and in the cmd field handed back if the command fails to start. Read that last condition twice. On a run that succeeds, the registered cmd is the raw argument list with the password sitting in it, and stdout is never touched at all.

- name: Leak demo
  hosts: web
  gather_facts: false
  tasks:
    - name: Register the runner
      ansible.builtin.command: /usr/local/bin/register --token={{ api_token }}
      ignore_errors: true

ansible-playbook leak.yml

PLAY [Leak demo] ***************************************************************
 
TASK [Register the runner] *****************************************************
fatal: [web01]: FAILED! => {"changed": false, "cmd": "/usr/local/bin/register --token=glpat-9f3a1c7e2b4dK", "msg": "[Errno 2] No such file or directory: b'/usr/local/bin/register'", "rc": 2, "stderr": "", "stderr_lines": [], "stdout": "", "stdout_lines": []}
...ignoring
 
PLAY RECAP *********************************************************************
web01                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=1

--token= is not --password=, so the whole GitLab token is in the log. Spell the same flag --password= and it would have come back as '--password=********', which is worth seeing once so that you stop trusting it: the masking follows the spelling of the flag, never the sensitivity of the value. The same gap covers -p, --api-key, a token handed over in an environment variable, and anything the command prints on its way out. no_log is the only control that covers all of them.

Prove You Actually Encrypted It

Do not trust the word successful. Check the repository the way somebody who stole it would, by searching for the value rather than for the file.

# every vaulted file, found by its marker
grep -rl '^\$ANSIBLE_VAULT' group_vars/ roles/
 
# now hunt for the plaintext among tracked files
git grep -n 'glpat-'; echo "rc=$?"

group_vars/all/vault.yml
roles/db/vars/secrets.yml
rc=1

An exit code of 1 from git grep means no match, which is the answer you want. A clean working tree is only half the check. Git keeps everything, so put the same question to the history with -S, which finds commits where the number of occurrences of a string changed.

git log --oneline -S 'glpat-9f3a1c7e2b4dK'

8caee1c encrypt secrets
8b6fdb5 add config

Two commits. One added the token in plaintext, the other encrypted it, and git show 8b6fdb5 prints the value in full to anyone who can clone the repo. Encrypting a secret that has already been committed hides today's copy and leaves the working one in history forever. So when that search comes back with results, the vault work is the second job. The first is rotating the credential at the system that issued it, so the copy sitting in your history stops being a key and becomes a string.

Try this

Run cat group_vars/all/vault.yml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: edit leaves a decrypted copy on disk while you type. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related