Ansible: playbooks & inventory

Agentless config over SSH.

Terraform builds the empty building. Ansible decides what goes on inside it: which packages are installed, what the config files say, which services are running, who is allowed to log in. Two tools, two jobs. Ansible's pitch fits in one sentence. It logs into your servers over SSH (Secure Shell, the encrypted remote-login protocol you already use), does the work, and logs out. Nothing stays behind on the machine afterwards. That is what people mean by agentless: no background program sits on every host waiting for orders.

The everyday version is a locksmith who carries their own tools to your house instead of leaving a spare toolbox in every house on the street. Fewer things to maintain, fewer things to steal. But the van becomes the thing worth guarding. Ansible works the same way. The machine you run it from (the control node) holds the SSH keys and the sudo rights (sudo is the standard Unix way to run one command as another user, normally root) for every host you manage. Take over that one laptop, or that one CI runner (the build machine that executes your pipeline), and an attacker needs no exploit at all. They inherit a supported, documented way to run commands as root across your whole fleet.

What Actually Crosses The Wire

Most tutorials skip this part, and it is the part that shows up in your logs. It works like a courier who brings a sealed envelope, does the job on your doorstep, and takes the envelope away again. Ansible does not send your YAML (a plain-text format for structured data, the language playbooks are written in) to the target. The YAML never leaves the control node. For each task, Ansible takes the module you named, bundles it with your arguments into one self-contained Python script, copies that script to the host, runs it, and reads a single line of JSON (JavaScript Object Notation, a machine-readable text format) back from standard output. Then it deletes the script. The host needs an SSH daemon (sshd, the background program that answers SSH connections) and a Python interpreter (the program that runs Python code). Individual modules can want a little more on top, apt wants python3-apt present, but that pair is the base.

Every run leaves a specific footprint on the managed machine, and you can go and look at it. On web-01, the authentication log (the file where Linux records logins and privilege changes) tells the whole story.

# run this ON the managed host, right after a play finishes
$ sudo grep -E 'Accepted publickey|BECOME-SUCCESS|Disconnected' /var/log/auth.log | tail -5
 
# no auth.log? Debian 12 ships without rsyslog. Ask the journal instead:
# sudo journalctl -t sshd -t sudo --since "10 min ago"

Jul 21 09:14:02 web-01 sshd[24417]: Accepted publickey for deploy from 10.0.4.11 port 51422 ssh2: ED25519 SHA256:2Xq9pLd7mQK0fVn8sJ1cRb4tYh6WzA3eG5uI7oP9kXs
Jul 21 09:14:03 web-01 sudo:   deploy : TTY=unknown ; PWD=/home/deploy ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-mkdwbrqvhtdfjxznlgqcpysewabmtruo ; /usr/bin/python3 /home/deploy/.ansible/tmp/ansible-tmp-1784625242.3160996-24431-172358911403772/AnsiballZ_setup.py
Jul 21 09:14:06 web-01 sudo:   deploy : TTY=unknown ; PWD=/home/deploy ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-qplzntgvxymrjbhcweudfkasoivxtprn ; /usr/bin/python3 /home/deploy/.ansible/tmp/ansible-tmp-1784625246.0287514-24438-92316657021094/AnsiballZ_apt.py
Jul 21 09:14:19 web-01 sudo:   deploy : TTY=unknown ; PWD=/home/deploy ; USER=root ; COMMAND=/bin/sh -c echo BECOME-SUCCESS-fdkwjeohtnzsrmcvbaqlypxugidnwrhs ; /usr/bin/python3 /home/deploy/.ansible/tmp/ansible-tmp-1784625259.8842123-24451-238067541199836/AnsiballZ_copy.py
Jul 21 09:14:24 web-01 sshd[24417]: Disconnected from user deploy 10.0.4.11 port 51422

Three things to notice. One login covers the whole play, not one per task, because Ansible opens the SSH connection once and reuses it for everything that follows (the ControlPersist option, set in the config file further down), so do not go hunting for a login line per task. BECOME-SUCCESS is the privilege-escalation handshake: sudo starts a shell and echoes a random 32-character token, and Ansible watches for exactly that token to know the escalation worked rather than sitting at a password prompt it cannot answer. TTY=unknown is there because no terminal is attached to the session. And the long path ending in AnsiballZ_setup.py is the module itself. The word before .py names it: setup gathered the facts, then apt, then copy.

Two notes for anyone writing detections. Turn on pipelining (a speed setting that feeds the module to the remote Python over standard input instead of writing it to disk) and the AnsiballZ file never touches the filesystem at all. The sudo line then stops at a bare /usr/bin/python3 with no script path after it. Hunt for both shapes. And because the pattern is so recognizable, it is easy to imitate: anyone holding the deploy key and those sudo rights produces identical-looking lines. Correlate against the control node's own run log. An AnsiballZ execution with no matching playbook run, or one sourced from an address that is not your control node, earns a phone call.

The Inventory Decides Your Blast Radius

An inventory is a mailing list. It records who receives the message and lets you address a whole department without typing out every name. Ansible's departments are groups: webservers, dbservers. Groups can hold other groups, so one name covers an entire environment, and a variable attached to a group is inherited by every member. Grouping is what turns a text file into fleet management. It is also what turns one typo into an outage, because the group name at the top of your playbook is the list of machines you are about to change.

# Static inventory in INI format: sections in [brackets], one host per line.
# A YAML inventory does the same job with different punctuation.
 
[webservers]
web-01.acme.internal
web-02.acme.internal
 
[dbservers]
db-01.acme.internal
 
# a group made of other groups
[prod:children]
webservers
dbservers
 
[webservers:vars]
# every member of this group inherits this
nginx_worker_processes=auto
 
[all:vars]
# the SSH login used on every host
ansible_user=deploy
ansible_ssh_private_key_file=~/.ssh/ansible_ed25519
# name the interpreter outright and skip discovery
ansible_python_interpreter=/usr/bin/python3

Notice that every comment sits on its own line. That is not style. Inside a :vars section Ansible keeps everything to the right of the = as the value, comment and all, so writing ansible_python_interpreter=/usr/bin/python3 # skip discovery hands the module a path with a sentence glued to the end of it. Host lines tolerate a trailing comment; variable lines do not. And never work out an inventory's contents by reading it, especially a dynamic one (a plugin that asks AWS or Azure which instances exist right now, so the list shifts under you between runs). Ask the tool. These two read-only commands answer the only question that matters before you press go.

$ ansible-inventory -i inventory.ini --graph

@all:
  |--@prod:
  |  |--@dbservers:
  |  |  |--db-01.acme.internal
  |  |--@webservers:
  |  |  |--web-01.acme.internal
  |  |  |--web-02.acme.internal
  |--@ungrouped:

# host patterns: ':' unions groups, '!' subtracts. Check before you act.
$ ansible 'webservers:!web-02*' -i inventory.ini --list-hosts

  hosts (1):
    web-01.acme.internal

A Playbook Is A Recipe Card

A recipe card does not say stir for ninety seconds. It says what the dish should look like when it is done. Playbooks work like that. A playbook holds one or more plays, and a play is a pairing: this group of hosts, this list of tasks. Each task names a module (a small purpose-built program: apt for packages, copy for files, service for units run by systemd, the program that starts and supervises services on modern Linux) and describes the end state you want. The module works out whether reality already matches. If it does, nothing happens and the task reports ok. If it does not, the module fixes it and reports changed. That property has a name: idempotence. Run the playbook ten times and the host ends up exactly where one run left it.

- name: Baseline the web servers
  hosts: webservers              # a group from the inventory, never "all"
  become: true                   # escalate to root through sudo
  gather_facts: true             # runs the setup module: OS, IPs, memory, mounts
 
  tasks:
    - name: Install nginx
      ansible.builtin.apt:       # FQCN = namespace.collection.module
        name: nginx
        state: present
        update_cache: true
        cache_valid_time: 3600   # skip apt-get update if it ran in the last hour
 
    - name: Ship the site config
      ansible.builtin.copy:
        src: files/site.conf
        dest: /etc/nginx/conf.d/site.conf
        owner: root
        group: root
        mode: "0644"             # quote it, or YAML hands Ansible a decimal
      notify: Reload nginx       # queue the handler ONLY if this task changed
 
    - name: Harden sshd with a drop-in
      ansible.builtin.template:
        src: templates/10-hardening.conf.j2
        dest: /etc/ssh/sshd_config.d/10-hardening.conf
        owner: root
        group: root
        mode: "0600"
        validate: /usr/sbin/sshd -t -f %s   # refuse to install a broken config
      notify: Restart sshd
 
    - name: Make sure nginx is running and survives a reboot
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true
 
  handlers:
    - name: Reload nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded
 
    - name: Restart sshd
      ansible.builtin.service:
        name: ssh                # unit is 'ssh' on Debian/Ubuntu, 'sshd' on RHEL
        state: restarted

# Managed by Ansible. Local edits are reverted on the next run.
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
PubkeyAuthentication yes
AllowGroups sshusers
ClientAliveInterval 300
ClientAliveCountMax 2
ListenAddress {{ ansible_facts['default_ipv4']['address'] }}   # filled per host

Several details across those two files earn their keep. Module names are written out in full as ansible.builtin.apt, the fully qualified collection name (namespace, then collection, then module), so there is no argument about which apt you meant. become: true escalates every task in the play to root, which is why you set it on the plays that need it and nowhere else. The validate line runs sshd's own syntax checker against the new file while it is still a temporary file, and if that check fails Ansible refuses to install it, so you keep a working sshd instead of a host nobody can reach. The quoted modes are deliberate. Write mode: 644 without quotes and YAML hands Ansible the decimal number 644, which is 1204 in octal, and the file lands as --w----r-T instead of rw-r--r--.

Two more, both worth remembering. notify does not fire the handler on the spot. Handlers are queued and run once at the end of the play, however many tasks notified them, and they run in the order they are written in the handlers section rather than the order they were notified. And on Debian 12 and Ubuntu 22.04 the main /etc/ssh/sshd_config opens with Include /etc/ssh/sshd_config.d/*.conf, while sshd keeps the first value it reads for most keywords. Because the include sits at the top, your drop-in (a small extra config file the main file pulls in) beats anything further down that main file. It also means a drop-in that sorts earlier, say 05-vendor.conf, silently beats yours. Read the whole directory, not only your own file.

$ ansible-playbook -i inventory.ini site.yml

PLAY [Baseline the web servers] ***********************************************
 
TASK [Gathering Facts] ********************************************************
ok: [web-02.acme.internal]
ok: [web-01.acme.internal]
 
TASK [Install nginx] **********************************************************
ok: [web-02.acme.internal]
changed: [web-01.acme.internal]
 
TASK [Ship the site config] ***************************************************
ok: [web-02.acme.internal]
changed: [web-01.acme.internal]
 
TASK [Harden sshd with a drop-in] *********************************************
changed: [web-01.acme.internal]
changed: [web-02.acme.internal]
 
TASK [Make sure nginx is running and survives a reboot] ***********************
ok: [web-01.acme.internal]
ok: [web-02.acme.internal]
 
RUNNING HANDLER [Reload nginx] ************************************************
changed: [web-01.acme.internal]
 
RUNNING HANDLER [Restart sshd] ************************************************
changed: [web-01.acme.internal]
changed: [web-02.acme.internal]
 
PLAY RECAP ********************************************************************
web-01.acme.internal       : ok=7    changed=5    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web-02.acme.internal       : ok=6    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Read that recap like a change log, because that is what it is. web-01 was a fresh machine, so nginx was installed, both files were written, and both handlers fired. web-02 already had nginx and the right site config, so only the sshd drop-in changed there and only the sshd handler ran. Handlers fire per host, for the hosts whose task actually reported changed. One trap sits in that arrangement: if a later task in the play fails on a host, the handlers queued for that host never run, which leaves a new config file on disk and the service still running the old one. Pass --force-handlers when that gap would hurt.

# run the exact same command again, nothing else touched
$ ansible-playbook -i inventory.ini site.yml

PLAY RECAP ********************************************************************
web-01.acme.internal       : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web-02.acme.internal       : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The second run is the interesting one. Nothing to do, so nothing is done, and the two handlers drop out of the count entirely. That all-zeros line is your steady state, and it makes a fine alarm. Once a play runs clean every night, an unexpected changed=1 means something on that host stopped matching the file you review in Git. Someone edited a config by hand, or a package upgrade replaced it, or an intruder did. A task that reports changed on every single run is a broken task, and the noise it makes is what hides the real signal.

Check Mode Is The Plan Step

A surveyor walks the site and writes down what would move before anyone picks up a hammer. That is --check: it runs the playbook without touching a thing, and --diff prints the exact lines that would change in every file it manages. Together they are Ansible's answer to terraform plan. Say you have added a security header to files/site.conf since last night's clean run. Aim the pair at one host with --limit and read the diff before you let the play near the group.

$ ansible-playbook -i inventory.ini site.yml --check --diff --limit web-02.acme.internal

PLAY [Baseline the web servers] ***********************************************
 
TASK [Gathering Facts] ********************************************************
ok: [web-02.acme.internal]
 
TASK [Install nginx] **********************************************************
ok: [web-02.acme.internal]
 
TASK [Ship the site config] ***************************************************
--- before: /etc/nginx/conf.d/site.conf
+++ after: /home/deploy/ops/files/site.conf
@@ -1,5 +1,6 @@
 server {
     listen 80;
     server_name acme.example.com;
     root /var/www/acme;
+    add_header X-Content-Type-Options "nosniff" always;
 }
 
changed: [web-02.acme.internal]
 
TASK [Harden sshd with a drop-in] *********************************************
ok: [web-02.acme.internal]
 
TASK [Make sure nginx is running and survives a reboot] ***********************
ok: [web-02.acme.internal]
 
RUNNING HANDLER [Reload nginx] ************************************************
changed: [web-02.acme.internal]
 
PLAY RECAP ********************************************************************
web-02.acme.internal       : ok=6    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check mode has honest limits. Tasks that depend on an earlier change often report ok or fail outright, because the earlier change did not really happen. command and shell tasks are skipped unless you mark them check_mode: false. A template's validate step is skipped as well, since there is no temporary file to validate. Check mode is an excellent way to catch a bad diff and a poor way to prove a complicated play works end to end.

Where Idempotence Dies

A module is a thermostat. You set the temperature you want and it decides whether the boiler needs to fire. The command module (which runs a program directly) and the shell module (which runs it through a shell, so pipes and redirects work) are a hand on the boiler switch. Ansible will run anything you like through either one, and both break the property everything else depends on. A raw command runs every time. Ansible cannot see inside it, so it has no way to know whether anything changed, and it reports CHANGED regardless. Here is a completely read-only command claiming it changed two hosts.

# no -m, so ad-hoc mode defaults to ansible.builtin.command
$ ansible webservers -i inventory.ini --become -a 'systemctl is-active nginx ssh'

web-01.acme.internal | CHANGED | rc=0 >>
active
active
web-02.acme.internal | CHANGED | rc=0 >>
active
active

That output is accurate and useless at the same time. systemctl is-active only reads state, yet the run is now indistinguishable in your logs from one that restarted the whole web tier. Guards fix it. creates skips the task entirely when a file already exists, changed_when tells Ansible how to judge the result for itself, and check_mode: false marks a task as safe to run during a dry run.

# Guards for the rare job no module covers.
- name: Install the vendor repo signing key
  ansible.builtin.shell: |
    set -o pipefail
    curl -fsSL https://repo.acme.com/key.asc | gpg --dearmor -o /usr/share/keyrings/acme.gpg
  args:
    executable: /bin/bash                    # /bin/sh is dash here: no pipefail
    creates: /usr/share/keyrings/acme.gpg    # file exists? skip the task entirely
 
- name: Read the effective sshd config
  ansible.builtin.command: /usr/sbin/sshd -T
  register: sshd_effective
  changed_when: false        # read-only, so it must never report "changed"
  check_mode: false          # safe to run during a --check dry run
 
- name: Fail the run if password logins are still allowed
  ansible.builtin.assert:
    that: "'passwordauthentication no' in sshd_effective.stdout_lines"
    fail_msg: "{{ inventory_hostname }} still accepts SSH passwords"

The Control Node Holds Root For The Whole Fleet

Follow the trust backwards. A playbook is a list of commands that will run as root on every host in a group, so merging a change to it changes what root does on hundreds of machines at once. That makes the playbook repository a production access-control system, and pull request review the gate on it. Give it the discipline you would give a firewall change: protect the branch, require a reviewer, sign the commits, and keep the control node's SSH key on a hardware token or in an agent that needs a human touch per use.

# Keep comments on their own line here too.
 
[defaults]
inventory          = ./inventory.ini
remote_user        = deploy
# the default. Never turn it off to silence a warning
host_key_checking  = True
# hosts configured in parallel
forks              = 20
# your side of the audit trail
log_path           = /var/log/ansible/run.log
interpreter_python = auto_silent
callbacks_enabled  = ansible.posix.profile_tasks
 
[privilege_escalation]
# opt in per play, not fleet-wide by default
become             = False
become_method      = sudo
 
[ssh_connection]
# module over stdin: faster, and no temp file on disk
pipelining         = True
ssh_args           = -C -o ControlMaster=auto -o ControlPersist=60s

The config file deserves as much care as the playbooks. Whoever can edit ansible.cfg controls how Ansible connects, whether it verifies host keys, and where it writes its logs. Ansible looks for it in the ANSIBLE_CONFIG environment variable first, then ./ansible.cfg in the current directory, then ~/.ansible.cfg, then /etc/ansible/ansible.cfg, and stops at the first one it finds. That directory-local lookup is why Ansible ignores an ansible.cfg sitting in a world-writable directory: otherwise dropping a file into /tmp and getting someone to run a play from there would be enough to hijack the run.

Three settings there have consequences worth spelling out. log_path records task output, so keep it at mode 0600 and owned by the account that actually runs Ansible, which is the CI user rather than root in most shops. Every password and token belongs in ansible-vault (Ansible's built-in encryption for variable files), with no_log: true on the tasks that handle them. And pipelining is the setting that rewrites the evidence you read earlier: switch it on and those sudo lines lose their AnsiballZ path, so update your detections in the same change. It needs sudo without requiretty, which is already how Debian and Ubuntu ship.

One last habit, the one that saves you. Verify from the host's point of view, not the playbook's. A task reporting ok means Ansible believes the state is right. Asking sshd what it actually loaded tells you whether the hardening took. Note the CHANGED label on a read-only command, exactly as promised.

$ ansible webservers -i inventory.ini --become -m ansible.builtin.shell \
    -a "/usr/sbin/sshd -T | grep -E '^(permitrootlogin|passwordauthentication|allowgroups)'"

web-01.acme.internal | CHANGED | rc=0 >>
permitrootlogin no
passwordauthentication no
allowgroups sshusers
web-02.acme.internal | CHANGED | rc=0 >>
permitrootlogin no
passwordauthentication no
allowgroups sshusers

Run that from a session you already had open before the sshd change, and keep it open. AllowGroups sshusers in that template is the line to watch: it locks out every account that is not in the sshusers group, including the deploy account Ansible connects with, unless you put deploy in that group first. If the only way back in was the connection Ansible used, you have locked yourself out of the fleet you were busy securing.

Try this

Run sudo grep -E 'Accepted publickey|BECOME-SUCCESS|Disconnected' /var/log/auth.log | tail -5 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: hosts: all is a loaded gun. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related