Runtime secrets from a real secret manager

Lookups against Vault, AWS, and Azure.

The last lesson left you with a sealed envelope taped inside the repository. ansible-vault encrypts a secret so the ciphertext can sit in git without being readable, and that genuinely works. It also means the secret lives in git forever: in every clone, on every laptop, in every CI (continuous integration, the robot that builds and deploys your code) cache, in every branch somebody abandoned two years ago, waiting on one password. Nobody gets an alert when somebody opens it. There is a stronger shape available, and it is where most teams end up. Keep the secret out of the repository entirely, fetch it from a real secret manager at the moment the play needs it, and let it disappear when the run ends.

The mechanism is the **lookup plugin**. A lookup is a small piece of code that goes and fetches a value from somewhere outside your playbook while Ansible is filling in a template. You have used one already, probably without noticing. {{ lookup('ansible.builtin.file', 'motd.txt') }} reads a file off disk. Point that same machinery at HashiCorp Vault, AWS Secrets Manager (Amazon's hosted secret store) or Azure Key Vault (Microsoft's) and your repository ends up holding a *reference* to a secret instead of the secret.

The front desk instead of the envelope

An envelope taped under a desk is offline access. Whoever finds it and knows the password reads it at three in the morning, silently, and you find out never. A hotel front desk works the other way round. You walk up, show your badge, the clerk checks a list to see whether that badge is allowed that key, hands it over, writes the loan in a ledger, and the key stops working on Friday.

Translated back into infrastructure, that is four things an encrypted file cannot give you. **Access control per identity.** The deploy pipeline may read the database password and a developer laptop may not, and that decision lives in a written policy rather than in who happens to know a passphrase. **An audit trail.** Vault's audit device, AWS CloudTrail (Amazon's log of every API call made in the account) and Azure's diagnostic logs each record who read which secret and when, so "was this credential touched during the incident window" becomes a query instead of a shrug. **Revocation.** Delete the caller's policy and the next run fails immediately, everywhere, with no chasing of clones. **Rotation without a commit.** Change the password in the secret manager and the next play picks up the new one. No ciphertext churn in git, no rekey ceremony, no pull request.

The bill for all that is a hard runtime dependency. If the secret manager is unreachable, your play does not degrade politely. It stops. Hold that thought, it comes back at the end.

A lookup runs on the control node

Here is the fact people get wrong more often than any other, and it shapes the whole design. **Modules run on the managed host. Lookups do not.** A lookup is you checking your own notebook before you pick up the phone. The person on the other end of the call never hears about the notebook, never sees it, and does not need a copy of it.

Mechanically, it goes like this. Jinja templating happens inside the ansible-playbook process on the control node, meaning the machine you typed the command on. When Ansible assembles the arguments for ansible.builtin.template or community.mysql.mysql_user, it renders every {{ ... }} first. So by the time a single byte crosses SSH (secure shell, the encrypted remote-login channel Ansible uses to reach Linux hosts), the lookup has already opened a connection to Vault, proved who it is, and got a plain string back. The target receives a finished value in its module arguments and has no idea where that value came from.

Three consequences fall straight out of that. The control node holds the credentials, so that is the machine you harden. The control node needs the network path, so the firewall rule is controller-to-Vault, not fleet-to-Vault. And the good one: four hundred web servers need no Vault token, no IAM (identity and access management, the cloud's permission system) role and no route to the secret manager at all. Trust concentrates in one place you can watch instead of smearing across every box you manage.

# Can the managed host itself reach Vault? By design, no.
ansible app01 -i inventories/prod -m ansible.builtin.uri \
  -a 'url=https://vault.internal:8200/v1/sys/health timeout=10'

app01 | FAILED! => {
    "changed": false,
    "content": "",
    "elapsed": 10,
    "msg": "Status code was -1 and not [200]: Request failed: <urlopen error timed out>",
    "redirected": false,
    "status": -1,
    "url": "https://vault.internal:8200/v1/sys/health"
}

# Same host, same minute. The controller has the address and the credentials.
# app01 has neither.
ansible app01 -i inventories/prod -m ansible.builtin.debug \
  -a "msg={{ lookup('community.hashi_vault.vault_kv2_get', 'prod/app/db', engine_mount_point='kv').secret.username }}"

app01 | SUCCESS => {
    "msg": "payments_rw"
}

The target could not open a socket to Vault, and the lookup still returned the value. That is the whole model in two commands. If you ever catch yourself putting a Vault token on a managed host so that "the playbook can read secrets", stop and read this section again. You would be handing a live credential to every machine in the group, which is precisely the spread you set out to avoid.

Wiring up Vault, AWS and Azure

None of these ship with ansible-core. Each one lives in a collection you install yourself, and each one calls a Python library that has to be present in the same interpreter that runs Ansible. Get that second half wrong and you get a baffling "plugin not found" or "boto3 required" message that has nothing whatsoever to do with your target hosts.

ansible-galaxy collection install community.hashi_vault amazon.aws azure.azcollection

Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/community-hashi_vault-7.1.0.tar.gz to /home/deploy/.ansible/tmp/ansible-local-9241k8p/tmpq7v1c0zn
Installing 'community.hashi_vault:7.1.0' to '/home/deploy/.ansible/collections/ansible_collections/community/hashi_vault'
community.hashi_vault:7.1.0 was installed successfully
Installing 'amazon.aws:11.4.0' to '/home/deploy/.ansible/collections/ansible_collections/amazon/aws'
amazon.aws:11.4.0 was installed successfully
Installing 'azure.azcollection:3.20.0' to '/home/deploy/.ansible/collections/ansible_collections/azure/azcollection'
azure.azcollection:3.20.0 was installed successfully

# The Python libraries the plugins actually call, installed into the same
# interpreter Ansible runs under. hvac talks to Vault, boto3 talks to AWS.
python3 -m pip install hvac boto3
 
# Azure ships its own pinned list inside the collection.
python3 -m pip install -r \
  ~/.ansible/collections/ansible_collections/azure/azcollection/requirements.txt

Successfully installed boto3-1.43.52 botocore-1.43.52 hvac-2.4.0

Vault's key/value store version 2 (KV v2, the versioned flavour of Vault's simplest secret engine) keeps a rolling history of a secret, ten versions deep by default, and it pays for that with an odd path. The API (application programming interface, the HTTP endpoint the client talks to) path is not kv/prod/app/db. It is kv/data/prod/app/db, with a literal data wedged between the mount point and your path. The original community.hashi_vault.hashi_vault lookup passes your string straight through to the API, so you write that data segment yourself, and forgetting it is the single most common Ansible-plus-Vault support question there is. The newer community.hashi_vault.vault_kv2_get lookup understands version 2 natively: give it the plain path plus engine_mount_point and it hands back a dictionary whose .secret key holds the actual key/value pairs. The older lookup is not formally deprecated, but the collection's own documentation now points you at the newer plugins, so use vault_kv2_get for anything you write today.

- name: Configure the payments application
  hosts: appservers
 
  pre_tasks:
    # One controller-side fetch for the whole play. run_once fires the lookup
    # a single time, and the resulting fact is applied to every host in the play.
    - name: Read the database credentials from Vault
      ansible.builtin.set_fact:
        db: >-
          {{ lookup('community.hashi_vault.vault_kv2_get',
                    'prod/app/db', engine_mount_point='kv').secret }}
      run_once: true
      no_log: true
 
    # The older string-terms form does the same job. Note the literal "data"
    # segment that KV v2 requires, and the :field suffix that picks one key:
    #   lookup('community.hashi_vault.hashi_vault',
    #          'secret=kv/data/prod/app/db:password')
 
  tasks:
    - name: Write /etc/myapp/app.conf
      ansible.builtin.template:
        src: app.conf.j2          # uses {{ db.username }} and {{ db.password }}
        dest: /etc/myapp/app.conf
        owner: myapp
        group: myapp
        mode: "0400"
      become: true
      no_log: true

Same shape, two other doors. The AWS lookup is amazon.aws.secretsmanager_secret, which you will also see written amazon.aws.aws_secret, its older name kept alive as a working alias. The Azure one is azure.azcollection.azure_keyvault_secret, and it wants the vault's own DNS (domain name system) address rather than a subscription or a resource group.

# AWS Secrets Manager. A secret holding JSON comes back as a JSON string,
# so pipe it through from_json and address the keys yourself.
- name: Read the payments API credentials from AWS Secrets Manager
  ansible.builtin.set_fact:
    api: >-
      {{ lookup('amazon.aws.secretsmanager_secret',
                'prod/payments/api', region='eu-west-1') | from_json }}
  run_once: true
  no_log: true
 
# Same secret, one key at a time. nested=true walks into the JSON using the
# dotted term. on_missing, on_denied and on_deleted each take error (the
# default), skip or warn.
- name: Read a single key out of that JSON secret
  ansible.builtin.set_fact:
    api_token: >-
      {{ lookup('amazon.aws.secretsmanager_secret',
                'prod/payments/api.token',
                nested=true, region='eu-west-1', on_missing='error') }}
  run_once: true
  no_log: true
 
# Azure Key Vault. vault_url is the vault's DNS address; the term is the
# secret name, or "name/version" if you want to pin one version.
- name: Read the TLS bundle passphrase from Azure Key Vault
  ansible.builtin.set_fact:
    tls_passphrase: >-
      {{ lookup('azure.azcollection.azure_keyvault_secret',
                'tls-bundle-passphrase',
                vault_url='https://kv-payments-prod.vault.azure.net') }}
  run_once: true
  no_log: true

Giving the control node an identity

You have moved the secret out of git. Now answer the obvious follow-up: what proves to Vault that this controller is allowed to read prod/app/db? If the answer is "a Vault token sitting in group_vars/all.yml", you have rebuilt the original problem with extra steps and a longer README.

Vault's AppRole method splits the credential in two, the way a bank card splits into the card and the PIN. The **role ID** is the card: stable, boring, fine to keep in configuration management. The **secret ID** is the PIN half, and it should be short-lived and delivered by whatever launches the playbook, never committed. A CI job pulls it from its own protected variable store. A human operator gets one issued for the session. The collection reads both from environment variables, so neither half has to appear in a playbook file at all.

# Control node only. $VAULT_SECRET_ID comes from the CI runner's protected
# variable store, or from a response-wrapped token issued for this session.
export ANSIBLE_HASHI_VAULT_ADDR=https://vault.internal:8200
export ANSIBLE_HASHI_VAULT_AUTH_METHOD=approle
export ANSIBLE_HASHI_VAULT_ROLE_ID=0f9b2c41-7a3e-4d55-9c18-2b7e6a4f1d02
export ANSIBLE_HASHI_VAULT_SECRET_ID="$VAULT_SECRET_ID"
 
ansible-playbook -i inventories/prod deploy-app.yml

PLAY [Configure the payments application] **************************************
 
TASK [Gathering Facts] *********************************************************
ok: [app01]
ok: [app02]
ok: [app03]
 
TASK [Read the database credentials from Vault] ********************************
ok: [app01]
 
TASK [Write /etc/myapp/app.conf] ***********************************************
changed: [app01]
changed: [app02]
changed: [app03]
 
PLAY RECAP *********************************************************************
app01                      : ok=3    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
app02                      : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
app03                      : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Read the recap carefully, because it tells you something real. The Vault task printed ok: [app01] once, and app01 finished with ok=3 while the other two show ok=2. Ansible only counts a run_once task against the host that actually executed it. The fact still reached all three, which is why the config landed everywhere. An uneven ok= column under a play that uses run_once is normal, not a symptom.

Better still, get rid of the standing secret entirely. If the controller runs on EC2 (Elastic Compute Cloud, an Amazon virtual machine), Vault's aws_iam auth method lets the instance prove who it is using its own instance profile, so there is no secret ID left to rotate. On the AWS side you get this free: amazon.aws lookups use the ordinary boto3 credential chain, so an instance profile on the controller, or the AWS_ROLE_ARN and AWS_WEB_IDENTITY_TOKEN_FILE pair that OIDC-based (OpenID Connect, a standard for proving an identity with a short-lived signed token) pipelines and Kubernetes service accounts export, gets picked up with zero Ansible configuration. The Azure lookup does the same trick: use_msi (managed service identity, a login the Azure platform attaches to the virtual machine itself) is on by default, and oidc_token_file_path covers workload identity federation from a pipeline. Vault has a matching jwt auth method for CI systems that mint those signed tokens. In every one of these, a long-lived credential is replaced by something the platform issues fresh per run and expires on its own.

One warning about the shape of all this. The audit log records the *controller*, not the engineer who typed ansible-playbook. Anybody who can start a play on that box, or get a shell on it, inherits every path its role is allowed to read. Give each pipeline its own role with a policy scoped to the exact paths that pipeline needs, keep humans off the controller, and keep the automation identity separate from the one you debug with. A single ansible-prod role that can read kv/data/* is a master key with a tidy audit log attached.

One expression, many API calls

A lookup is a function call, not a value, and Jinja re-runs it every single time the expression is rendered. It is the difference between collecting your room key once and walking back to the front desk before every door. Define db_password as a lookup under vars: and reference it in four tasks across sixty hosts, and you have not made one call to Vault. You have made two hundred and forty. With AppRole, each of those logs in first, so you also mint two hundred and forty tokens that then sit in Vault until their TTL (time to live, the span after which a token stops working) runs out.

You do not have to guess whether this is happening to you. Vault's file audit device writes one line of JSON (JavaScript Object Notation, a plain-text data format you can read in any editor) per request, and while the values in it are hashed, the paths are in the clear. So count them.

# On the Vault server, for a single playbook run.
sudo jq -r 'select(.type=="request") | .request.path' /var/log/vault/audit.log \
  | sort | uniq -c

    240 auth/approle/login
    240 kv/data/prod/app/db

That is the lazy vars: version. Move the fetch into one set_fact with run_once: true, the way deploy-app.yml above does it, and the identical run looks like this.

sudo jq -r 'select(.type=="request") | .request.path' /var/log/vault/audit.log \
  | sort | uniq -c

      1 auth/approle/login
      1 kv/data/prod/app/db

Two hundred and forty requests down to one, and this matters for more than politeness. Vault enforces rate-limit quotas, and a play that hammers it starts collecting HTTP 429 responses (the server saying "you are asking too fast") at exactly the wrong moment. Your audit log stays short enough that an anomalous read is visible in it. A play that reads once also stops failing halfway through when the secret manager has a bad minute. If you need several secrets, vault_kv2_get takes multiple paths in one call, which still costs one read per path but only one login for the lot. And the community.hashi_vault.vault_login module paired with the community.hashi_vault.vault_login_token filter lets you authenticate once and hand the resulting token to every later task.

What still goes wrong

A lookup failure is not a warning. It is a fatal error raised at templating time, and it lands wherever in the play that expression happened to be rendered. Revoke the role's policy mid-rollout and you get this.

ansible-playbook -i inventories/prod deploy-app.yml

PLAY [Configure the payments application] **************************************
 
TASK [Gathering Facts] *********************************************************
ok: [app01]
ok: [app02]
ok: [app03]
 
TASK [Read the database credentials from Vault] ********************************
fatal: [app01]: FAILED! => {"msg": "An error occurred while running the lookup plugin 'community.hashi_vault.vault_kv2_get'. Forbidden: Permission Denied to path ['prod/app/db']."}
 
PLAY RECAP *********************************************************************
app01                      : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
app02                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
app03                      : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Only app01 shows failed=1, because only app01 ran the task, but nothing was written on any of the three. When a run_once task fails, Ansible marks every host in the play as failed and the run stops there. That is correct behaviour and also a design constraint you should lean on. Do the fetching in pre_tasks, before the first change, so a credential problem stops the run while the fleet is still untouched, rather than at host forty of eighty with half your servers on the new config and half on the old. Where a missing secret is genuinely survivable, the AWS lookup gives you on_missing='skip' and on_denied='warn'. For Vault, wrap the set_fact in a block with a rescue that sets a safe default and prints a message somebody will understand at 2am.

The second honest caveat: fetching at run time keeps the secret out of git and does nothing at all to keep it out of everything else. The moment the lookup returns, that value is an ordinary Ansible variable sitting in the controller's memory. Left alone, it shows up in -vvv output, in a registered result you later debug, in the diff that --diff prints for a changed template, and in whatever your callback plugin ships to the log aggregator. no_log: true on every task that touches it is not optional, and that is the whole subject of the next lesson.

The third one catches people who did everything else right, so go and check it on your own controller. If your ansible.cfg turns on fact caching and you wrote cacheable: true on that set_fact, the value is now on the controller's disk as plaintext JSON.

[defaults]
fact_caching            = jsonfile
fact_caching_connection = ~/.ansible/cache
fact_caching_timeout    = 86400

ls -l ~/.ansible/cache/
jq -r '.db.password' ~/.ansible/cache/app01

total 12
-rw-r--r-- 1 deploy deploy 2317 Jul 21 09:14 app01
-rw-r--r-- 1 deploy deploy 2298 Jul 21 09:14 app02
-rw-r--r-- 1 deploy deploy 2301 Jul 21 09:14 app03
pr0d-payments-9f2c41ae7b

There it is. A secret you were careful never to commit, sitting unencrypted in a home directory for the next twenty-four hours, swept into any backup of the control node. Look at the mode column: -rw-r--r--. Ansible's file-based cache plugins chmod every cache file to 0644, so that password is readable by every local account on the controller, not only by the one that ran the play. That jq command takes ten seconds to run against your own cache directory, and it is the check nobody does.

When the sealed envelope still wins

Runtime lookups are the better default. They are also not always available. Three cases where ansible-vault stays the correct answer rather than a compromise. **Air-gapped estates**, where the control node has no route to any secret manager by deliberate design, and an encrypted file travelling with the repository is the only thing that can work. **Small operations with no secret manager to point at**, where running Vault properly (high availability, backups, unseal keys held by real humans who answer the phone) is a bigger operational risk than one encrypted variables file with a well-managed password. And **bootstrapping**, which is the interesting one, because something has to hold the credential that opens the secret manager. Vault unseal keys, the first AppRole secret ID, the break-glass root password: none of them can live inside the thing they open. Those go in a vaulted file, on a hardware token, or in an envelope in an actual safe.

The split most teams settle on is small enough to audit in one command. A handful of bootstrap credentials in ansible-vault, reviewed and rotated on a schedule, and everything else fetched at run time. Run grep -rl '\$ANSIBLE_VAULT' . across the repository. You should get back a short, boring list you could have named from memory. If it returns forty files, the secret manager is not doing the job you installed it for.

Related