Hardening playbooks & the controller

become, SSH, and control-node risk.

Your control node is a locksmith's van. It carries a key to every door in the estate, a list of which key opens which door, and enough paperwork to walk in without knocking. Ansible's agentless design really is a win on the managed hosts: nothing extra listens, nothing runs between playbook runs, so there is no daemon (a background service) of Ansible's own sitting there waiting to be attacked. The trust has to live somewhere, though. It ends up parked in that one van.

Hardening Ansible is therefore two jobs stacked together. On the targets, escalate only as far as each task genuinely needs. On the controller, protect the single machine that can reach all of them. This lesson is about that blast radius. Encrypting the values themselves belongs to an-vault, and gating runs behind a pipeline belongs to an-cicd.

Count Your Blast Radius First

You cannot size a risk you have never measured, and two commands measure this one. The first asks the inventory (Ansible's list of hosts, grouped by role) how many machines this controller is allowed to touch. The second asks ssh-agent, the small background program that holds your decrypted SSH (secure shell) keys in memory, what it is currently willing to sign with.

# every host this controller is allowed to reach
ansible all -i inventory/hosts.yml --list-hosts | head -3
 
# every key the agent will sign with right now, no passphrase needed
ssh-add -l
 
ls -l ~/.ssh/

  hosts (37):
    api01.internal
    api02.internal
256 SHA256:5tYr0jK8xQ2mLpW7vBnZ1cD4eF6gH9iJ3kM0oPqRsTu ansible@ctl-01 (ED25519)
total 16
-rw------- 1 deploy deploy  419 Jun  2 11:14 ansible_ed25519
-rw-r--r-- 1 deploy deploy   98 Jun  2 11:14 ansible_ed25519.pub
-rw-r--r-- 1 deploy deploy 4238 Jul 21 08:12 known_hosts

Thirty-seven hosts. One key. No passphrase standing between that key and the network, because the agent is already holding it open. Read it as a sentence about people rather than files: anyone who can run a process as deploy on this box can reach thirty-seven machines this second, without copying anything, because the agent signs on request. Every setting below is a way of making that sentence smaller.

Escalate Per Task, Not Per Play

A restaurant kitchen keeps the walk-in freezer key on a hook by the door. Cooks take it down for a delivery and hang it back after. Nobody carries it in an apron pocket all shift, because aprons end up on the floor. become is Ansible's privilege escalation: run this one task as somebody else, through sudo (substitute user do) unless you name another method, and as root unless you name another user. Writing become: true at the top of a play is the apron pocket.

The cost of that is quiet. A task that only reads a version string now runs as root. A dest: path built from a Jinja2 expression (Ansible's templating language) that expanded into something you did not intend now writes successfully instead of bouncing off a permission error. Permission denied is what saves you in that moment. So default the play to no escalation and switch it on task by task.

- name: Web tier
  hosts: web
  become: false                       # the play default: connect as an ordinary user
 
  tasks:
    - name: Read the deployed app version
      ansible.builtin.command: /opt/app/bin/app --version
      register: app_version
      changed_when: false             # reading is never a change
 
    - name: Install nginx
      ansible.builtin.apt:
        name: nginx
        state: present
        update_cache: true
      become: true                    # escalation starts here
      become_user: root               # and ends with this task
 
    - name: Ship the sudoers drop-in for the automation account
      ansible.builtin.template:
        src: sudoers-ansible.j2
        dest: /etc/sudoers.d/ansible
        owner: root
        group: root
        mode: "0440"                  # sudo skips any file group- or world-writable
        validate: /usr/sbin/visudo -cf %s
      become: true

Two of those tasks escalate and one does not. The validate line is worth stealing wholesale: Ansible renders the template to a temporary file on the target, runs visudo -cf against that file, and only moves it into place if it parses. A broken sudoers file does not fail loudly. It makes sudo refuse to work at all, and you learn about it at the moment you have lost root on every host that play reached.

ansible web -i inventory/hosts.yml -m ansible.builtin.command -a whoami
 
# -K is --ask-become-pass
ansible web -i inventory/hosts.yml -m ansible.builtin.command -a whoami --become -K

web01.internal | CHANGED | rc=0 >>
ansible
 
web02.internal | CHANGED | rc=0 >>
ansible
 
BECOME password:
web01.internal | CHANGED | rc=0 >>
root
 
web02.internal | CHANGED | rc=0 >>
root

Same module, same hosts, different account. Drop the -K against a host whose sudoers demands a password and the run stops with Missing sudo password instead of guessing or hanging. That failure is the control working, not a bug to route around.

# Rendered by site.yml, validated by visudo -cf before it is moved into place.
# Owner root:root, mode 0440, or sudo skips the file entirely.
 
ansible ALL=(ALL) PASSWD: ALL
 
# Only needed where the host still ships "Defaults requiretty", which most
# modern distributions no longer do. Check before you add it:
#   sudo grep -r requiretty /etc/sudoers /etc/sudoers.d
Defaults:ansible !requiretty

# does it parse? a broken sudoers file locks root out of the whole fleet
visudo -cf /etc/sudoers.d/ansible
 
# what can the automation account do, as sudo itself sees it?
sudo -l -U ansible

/etc/sudoers.d/ansible: parsed OK
Matching Defaults entries for ansible on web01:
    env_reset, mail_badpass,
    secure_path=/usr/local/sbin\:/usr/local/bin\:/usr/sbin\:/usr/bin\:/sbin\:/bin,
    use_pty, !requiretty
 
User ansible may run the following commands on web01:
    (ALL) PASSWD: ALL

PASSWD: is the interesting choice there. Most fleets ship NOPASSWD: ALL, which means a stolen private key is root on every host in one hop with no second step. Requiring the password adds something the key does not carry. Be honest about what it moves rather than removes: unattended runs need that password on the controller, so it lives in Vault, and the vault password becomes the thing worth stealing. It buys you a great deal against a leaked key and nothing at all against a compromised controller.

Do not expect sudoers to narrow this any further, either. (ALL) PASSWD: ALL looks lazy next to a tidy list of permitted commands, but every Ansible module arrives at sudo as a Python interpreter reading a generated payload, and with pipelining on that payload is not even a file on disk. There is no stable command string to pin down. Your real granularity lives in the playbook: which tasks carry become: true, and who is allowed to merge changes to them.

Prove the Escalation Landed Where You Think

Three levels of verbosity (-vvv) make Ansible print the exact command it hands to ssh, sudo wrapper and all. That turns "I think this play is least-privilege" into something you can grep.

ansible-playbook -i inventory/hosts.yml site.yml -vvv > run.log 2>&1
 
# how many remote commands ran, and how many of them escalated?
grep -c 'SSH: EXEC'  run.log
grep -c 'sudo -H -S' run.log
 
# what the escalation actually looks like on the wire
grep -oE 'sudo -H -S -p "[^"]+" -u [a-z]+' run.log | head -1

16
6
sudo -H -S -p "[sudo via ansible, key=ntgvxlqbrmhydwsjpcfkzeaouivbxqmt] password:" -u root

Sixteen remote commands, six of them as root. Two hosts, three root commands on each: the two tasks that carry become: true, plus the extra temp-directory step the file-shipping template task needs on top of them. The apt task adds nothing extra because pipelining lets it run in a single command. Move become: true up to the play and all sixteen escalate. Inside the sudo line, -H sets HOME to the target user's, -S reads the password from standard input rather than a terminal, -p sets the prompt string Ansible waits for, and -u root names the destination account. Sudo's usual -n (fail instead of prompting) is dropped the moment a password is configured, which is why the prompt appears at all. The random BECOME-SUCCESS-... marker that follows the sudo call is how Ansible distinguishes "escalation worked" from "the module printed something".

# a play whose task escalates to a service account rather than to root:
#   become: true
#   become_user: appuser
ansible-playbook -i inventory/hosts.yml render-config.yml

TASK [Render the app config as the service account] ***************************
fatal: [web01.internal]: FAILED! => {"changed": false, "msg": "Failed to set permissions on the temporary files Ansible needs to create when becoming an unprivileged user (rc: 1, err: chmod: invalid mode: 'A+user:appuser:rx:allow'\nTry 'chmod --help' for more information.\n}). For information on working around this, see https://docs.ansible.com/ansible-core/2.16/playbook_guide/playbooks_privilege_escalation.html#risks-of-becoming-an-unprivileged-user"}

The Hop Is Where Identity Gets Checked

A courier arrives at reception with a parcel. You were given a photograph when the contract was signed, and you check the face against the photograph every single visit, not only the first one. That is host key checking. Ansible leaves it on by default, and the SSH connection plugin only appends -o StrictHostKeyChecking=no when you deliberately turn it off.

ansible web03.internal -i inventory/hosts.yml -m ansible.builtin.ping

web03.internal | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: @@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\n@    WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED!     @\n@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@\nIT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!\nSomeone could be eavesdropping on you right now (man-in-the-middle attack)!\nIt is also possible that a host key has just been changed.\nThe fingerprint for the ED25519 key sent by the remote host is\nSHA256:0hM7bnP2tQ9vK4yXcR1sLdW6uZaG5jF8oEiT3nBqYxU.\nPlease contact your system administrator.\nAdd correct host key in /home/deploy/.ssh/known_hosts to get rid of this message.\nOffending ED25519 key in /home/deploy/.ssh/known_hosts:12\nHost key for web03.internal has changed and you have requested strict checking.\nHost key verification failed.",
    "unreachable": true
}

That is what a rebuilt machine looks like. It is also exactly what somebody sitting on the path between you and the host looks like, and from the controller the two are indistinguishable, which is the whole point of the check. Run the same moment with ANSIBLE_HOST_KEY_CHECKING=False exported in a CI (continuous integration) job because the pipeline kept going red, and it plays out differently: the run succeeds, your automation account authenticates to the impostor, your task arguments are pushed there, and the sudo password you were so careful to require gets typed into the impostor's prompt. Silent, green, complete.

For a genuine rebuild, re-pin the key from a source that does not depend on the network you currently distrust: the cloud console, the instance's serial output, or the image build itself. ssh-keyscan against the suspect address only launders the same doubt. At fleet scale, stop pinning individual keys and run an SSH certificate authority instead. Sign each host key at build time, tell sshd to present the resulting certificate, put one line in known_hosts, and every host you ever build afterwards verifies with no new entry at all.

# One line covers every host the CA signs, today and next year.
# ansible.builtin.known_hosts manages individual pins as data if you still need them.
@cert-authority *.internal ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIHc1pQ0oXqM2y7bK9vLZ8dRt4wJnF6sYgEuT0iBxq3Vk host-ca@corp

[defaults]
inventory         = inventory/hosts.yml
host_key_checking = True
private_key_file  = ~/.ssh/ansible_ed25519
; never set allow_world_readable_tmpfiles here
 
[privilege_escalation]
become          = False        ; opt in per play or task, never globally
become_method   = sudo
become_ask_pass = False        ; the password comes from Vault, not a prompt
 
[ssh_connection]
pipelining       = True
ssh_args         = -C -o ControlMaster=auto -o ControlPersist=60s
control_path_dir = ~/.ansible/cp

Two details in that file catch people out. Comments that follow a value must start with ;, because Ansible's parser strips inline ; comments and leaves inline # in place, so host_key_checking = True # on sets the value to the string True # on. And pipelining = True is the rare setting that is both faster and safer: without it, Ansible copies each module to a temporary file on the target and runs it from there; with it, the module is fed straight into the target's Python over the connection you already have, so it never touches the target's disk. It is off by default only because it clashes with the ancient requiretty sudoers setting, and file-shipping modules like template still need a temp directory.

ansible-config dump --type connection ssh --only-changed

 
ssh:
___
control_path_dir(/home/deploy/fleet/ansible.cfg) = ~/.ansible/cp
host_key_checking(/home/deploy/fleet/ansible.cfg) = True
pipelining(/home/deploy/fleet/ansible.cfg) = True
private_key_file(/home/deploy/fleet/ansible.cfg) = ~/.ssh/ansible_ed25519
ssh_args(/home/deploy/fleet/ansible.cfg) = -C -o ControlMaster=auto -o ControlPersist=60s

The path in parentheses is the payoff. It tells you the value came from your file rather than from a default that a future release could change under you. --only-changed filters on where a value came from, not on whether it differs from the default, which is why host_key_checking = True appears even though True was already the default. Run this from the same directory your pipeline runs from, because a config file in a world-writable directory is ignored outright, with a warning, so nobody can drop an ansible.cfg into a shared /tmp and redirect your plugin paths.

Never Forward the Agent

Agent forwarding is handing your whole keyring to the person who opened the door, so they can lock up behind you. -o ForwardAgent=yes places a socket on the managed host, and anything running as root there can talk to that socket while your session lives and ask your agent to sign an authentication challenge. They never see the key file. They do not need to: SSH_AUTH_SOCK=/tmp/ssh-Xk29aB/agent.4471 ssh [email protected] puts them on the second of your thirty-seven hosts, authenticated as you, with nothing to crack.

The real need behind forwarding is usually a bastion (a jump host you must pass through to reach a private network), and ProxyJump solves that without the exposure. Authentication still happens on your controller; the bastion only shuttles bytes.

ansible_user: ansible
ansible_ssh_private_key_file: ~/.ssh/ansible_ed25519
 
# hop through the bastion; never hand it the agent
ansible_ssh_common_args: -o ProxyJump=bastion.example.net -o ForwardAgent=no
 
# decrypted from vault.yml at run time (see an-vault)
ansible_become_password: "{{ vault_become_password }}"

Treat the Controller as Disposable

Do not run playbooks as root on the controller itself. become handles escalation on the targets, so a privileged local account buys nothing and gives up the separation you were building. Keep the private key and the vault password off shared filesystems and out of shell history. Prefer a runner rebuilt for every job over a long-lived box that everybody logs into, because a shared controller collects credentials, cached sudo timestamps, and live connection sockets. Those sockets are not a footnote: ~/.ansible/cp is created mode 0700 for a reason, and for the sixty seconds ControlPersist keeps one open, it is an already-authenticated session to a production host that needs no key at all.

One playbook habit belongs in the same breath. Any task that handles a secret gets no_log: true, which stops Ansible echoing that task's arguments and result into the terminal and into any configured log file.

- name: Write the API token the app reads at boot
  ansible.builtin.copy:
    content: "{{ vault_api_token }}"
    dest: /etc/app/token
    owner: app
    group: app
    mode: "0400"
  become: true
  no_log: true

TASK [Write the API token the app reads at boot] ******************************
changed: [web01.internal] => {"censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result"}

Know its edges. no_log hides the result, not the mechanics. A secret passed to ansible.builtin.command still lands in the target's process list while the command runs, where any local user with ps can read it. Pass secrets as module parameters, as the content: above does, and keep them out of command lines and shell strings.

The last measurement is the one that usually settles everything else.

# who has landed a change in the automation repo in the last year?
git log --since='1 year ago' --format='%ae' -- playbooks/ roles/ | sort -u | wc -l

19

Nineteen people can write a task that runs as root on thirty-seven machines. That number, not ansible.cfg, is the privilege model you actually operate. Branch protection and review on that repository are worth more than every setting in this lesson put together, because a merged malicious task does not have to defeat a single one of them.

Try this

Run ansible all -i inventory/hosts.yml --list-hosts | head -3 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: become_user to a non-root account can go world-readable. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related