What Ansible is & the push model

Agentless automation over SSH.

Forty servers need the same fix by Friday. You can log into each one and type the same six commands, and that works right up until you fat-finger host twenty-three and nobody notices for a month. Ansible does the typing for you: the same change, on every host, with a report at the end telling you exactly which machines actually changed. It installs packages, writes config files, starts and stops services, and sequences work across many machines in an order you decide.

What makes it unusual is what it refuses to install. A travelling electrician turns up with a van of tools, does the job, and drives off. A live-in caretaker stays in the building forever. Ansible is the electrician. It is agentless, which means no permanent background program (no daemon, no service quietly running and listening on a port) is installed on the machines it manages. Between runs there is nothing of Ansible on those hosts at all. Nothing waking on a timer, nothing holding a connection open, nothing extra for you to patch.

The machine you run Ansible from is called the control node: your laptop, a bastion host (a hardened server that acts as the single front door into a network), or a CI runner (CI is continuous integration, the pipeline that builds and tests your code automatically). It needs Python and either the ansible or the ansible-core package. Everything else is a managed host, and a managed host needs exactly two things: an SSH server (SSH is Secure Shell, the encrypted remote-login protocol every Linux box already runs) that you can authenticate to, and a Python interpreter. For ansible-core 2.16, the version used throughout this lesson, that means Python 3.10 to 3.12 on the control node, and Python 2.7 or Python 3.6 and newer on the targets. Later releases tightened both ends, and 2.17 dropped Python 2 on targets completely. Windows machines can be managed over WinRM (Windows Remote Management, Microsoft's remote-command service) or SSH, but Windows cannot be a control node at all; people who need one run it inside WSL (Windows Subsystem for Linux).

Push, pull, and who starts the conversation

The older way to run a fleet is a noticeboard in the staff room. Every worker checks it on their own schedule and does whatever it currently says. That is the pull model, and it is how Puppet and Chef work by default: a small agent process lives on every node, wakes up roughly every thirty minutes, calls home to a central server, asks what it is supposed to look like, and repairs itself on the spot. Salt often gets lumped in with those two, and it should not be. Salt does install an agent, but that agent holds an outbound connection open and waits for the master to push work down it, so Salt is agent-based and master-driven at the same time.

Ansible throws the noticeboard away. You start the conversation. The control node opens the connections, sends the work, watches it happen, and hangs up. Nothing occurs on a managed host unless you, or a pipeline acting for you, kicked it off, so change lands at a moment you chose and can point to afterwards. Because one brain is driving every host, ordering across machines comes free. "Take these five out of the load balancer one at a time, patch, check it came back, put it back in" is an ordinary Ansible play. An agent waking on its own timer has no idea what the other nine nodes are doing.

The costs are real and you should know them before picking a side. Nothing enforces state between runs. If somebody hand-edits /etc/nginx/nginx.conf at 2am, that edit stays live until a human runs the playbook again, where a Puppet agent would have stamped it out within its check-in interval. Push also needs a network path inward, so firewalls have to allow SSH from the control node to every host, while pull agents only ever connect outward and are much happier behind NAT (Network Address Translation, where many machines share one outward address and cannot be dialled directly). And a push run costs wall-clock time. Ansible works on five hosts at a time by default, per task, so a thousand hosts means two hundred rounds of every single task until you raise forks. For the cases where that genuinely hurts, Ansible ships a pull mode of its own: ansible-pull -U https://git.acme.com/ops.git local.yml clones the repo onto the host and applies the playbook locally, usually fired by a systemd timer.

Prove the connection before you trust it

Two things describe the work. The inventory is the address book: which hosts exist and which groups they belong to (web, db, prod). Playbooks are the job sheet: what should be true on those hosts. Start with a short inventory and one throwaway command, because if authentication and Python are not working, nothing else you write matters.

[web]
web1.acme.internal
web2.acme.internal
web3.acme.internal
 
[db]
db1.acme.internal
 
[prod:children]
web
db
 
[web:vars]
ansible_user=deploy

$ ansible --version

ansible [core 2.16.3]
  config file = /home/deploy/ops/ansible.cfg
  configured module search path = ['/home/deploy/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = /usr/lib/python3/dist-packages/ansible
  ansible collection location = /home/deploy/.ansible/collections:/usr/share/ansible/collections
  executable location = /usr/bin/ansible
  python version = 3.12.3 (main, Feb  4 2025, 14:48:35) [GCC 13.3.0] (/usr/bin/python3)
  jinja version = 3.1.2
  libyaml = True

$ ansible -i inventory.ini web -m ansible.builtin.ping

web1.acme.internal | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}
web3.acme.internal | UNREACHABLE! => {
    "changed": false,
    "msg": "Failed to connect to the host via ssh: [email protected]: Permission denied (publickey).",
    "unreachable": true
}
web2.acme.internal | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

ansible.builtin.ping has nothing to do with the ping command you already know. It sends no ICMP packet anywhere (ICMP is Internet Control Message Protocol, the thing ordinary ping uses). It logs in over SSH, copies a tiny Python module across, runs it, and gets the word pong back. That proves the whole chain you actually depend on: the name resolved, the network path is open, your key was accepted, and the target has a working Python interpreter. The long module name is an FQCN, a fully qualified collection name, written namespace.collection.module. Short names like ping still resolve, but they go through a search path and can quietly pick up a different module from a different collection that happens to sit earlier in it. Write the full name.

Stare at the web3 line for a moment, because UNREACHABLE and failed mean different things. Failed means Ansible got in and the task did not work. Unreachable means it never got in at all, so that host is dropped for the rest of the play and every later task shows nothing for it. Reading a recap, seeing failed=0, and not noticing unreachable=1 sitting next to it is a classic way to believe a change went out fleet-wide when it did not. "Permission denied (publickey)" says the SSH server rejected your key, which in practice is almost always the wrong ansible_user, a key missing from that host's authorized_keys file, or a home directory with permissions loose enough that sshd refuses to trust it.

What actually crosses the wire

Ansible does not open a shell and type into it like a very fast human. For each task it slides a note under the door. It builds a small, self-contained Python program out of the module's code plus the arguments you gave it, zipped and encoded into a single file (the machinery is called AnsiballZ), copies that file into a temporary directory on the target, runs it with the target's own Python, reads back exactly one JSON document printed on standard output (JSON is JavaScript Object Notation, a text format machines can parse reliably), deletes the temporary directory, and moves on. Add -vvv to any command and you can watch it happen. The repeated pile of ssh options is trimmed to three dots below so the shape stays readable.

$ ansible -i inventory.ini web1.acme.internal -m ansible.builtin.ping -vvv

ansible [core 2.16.3]
  config file = /home/deploy/ops/ansible.cfg
<web1.acme.internal> ESTABLISH SSH CONNECTION FOR USER: deploy
<web1.acme.internal> SSH: EXEC ssh -C -o ControlMaster=auto -o ControlPersist=60s -o KbdInteractiveAuthentication=no -o PreferredAuthentications=gssapi-with-mic,gssapi-keyex,hostbased,publickey -o PasswordAuthentication=no -o 'User="deploy"' -o ConnectTimeout=10 -o ControlPath=/home/deploy/.ansible/cp/8f2c1a4e9b web1.acme.internal '/bin/sh -c '"'"'echo ~deploy && sleep 0'"'"''
<web1.acme.internal> (0, b'/home/deploy\n', b'')
<web1.acme.internal> EXEC ... '( umask 77 && mkdir -p "` echo /home/deploy/.ansible/tmp `"&& mkdir "` echo /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845 `" && echo ansible-tmp-1784625120.86-9241-271828182845="` echo /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845 `" ) && sleep 0'
<web1.acme.internal> (0, b'ansible-tmp-1784625120.86-9241-271828182845=/home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845\n', b'')
Using module file /usr/lib/python3/dist-packages/ansible/modules/ping.py
<web1.acme.internal> PUT /home/deploy/.ansible/tmp/ansible-local-9235hq1ke/tmp8k2jvr TO /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845/AnsiballZ_ping.py
<web1.acme.internal> SSH: EXEC sftp -b - -C -o ControlMaster=auto ... '[web1.acme.internal]'
<web1.acme.internal> EXEC ... 'chmod u+x /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845/ /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845/AnsiballZ_ping.py && sleep 0'
<web1.acme.internal> EXEC ... '/usr/bin/python3 /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845/AnsiballZ_ping.py && sleep 0'
<web1.acme.internal> EXEC ... 'rm -f -r /home/deploy/.ansible/tmp/ansible-tmp-1784625120.86-9241-271828182845/ > /dev/null 2>&1 && sleep 0'
web1.acme.internal | SUCCESS => {
    "ansible_facts": {
        "discovered_interpreter_python": "/usr/bin/python3"
    },
    "changed": false,
    "ping": "pong"
}

Read that trace top to bottom and the whole push model is sitting there: connect, find the home directory, make a temp directory, upload AnsiballZ_ping.py, make it executable, run it with /usr/bin/python3, delete the temp directory, print the result. That is the default behaviour, with pipelining off. Two settings turn it from slow into fine. ControlPersist, visible in the ssh line, keeps the authenticated session open for sixty seconds so the next task skips the handshake entirely. Pipelining feeds the module straight into the remote Python's standard input instead of writing a file, which collapses that whole string of round trips into a single SSH invocation. It ships off by default for a historical reason: requiretty in /etc/sudoers breaks it, and old RHEL set that. Current Debian, Ubuntu and RHEL do not, so switch it on.

This one design decision explains the rest of Ansible's behaviour. The target needs Python because the module is a Python program that executes there, not on your laptop. There is no daemon because the "agent" lives for about a second per task and is then deleted. And because the module hands back structured data rather than screen scrapings, Ansible can tell you ok or changed per host instead of dumping raw text at you and leaving you to squint at it. For a machine too bare to have Python at all, ansible.builtin.raw pushes a command straight down the SSH pipe with no Python involved, which is exactly how you bootstrap Python onto a fresh box before anything else will work.

Desired state, and why the second run is boring

A shopping list and driving directions are different kinds of instruction. "Drive to the shop and buy milk", run twice, gets you two bottles. "There should be milk in the fridge", run twice, gets you one, because the second time you look it is already there. Playbooks are shopping lists. You declare the state you want (nginx installed, this file present with these permissions, this service running and set to start at boot) in YAML, the indented text format Ansible reads, and each module works out for itself whether reality already matches. That property is idempotence: running the same play again changes nothing, because there is nothing left to change.

- name: Baseline web servers
  hosts: web
  become: true
  tasks:
    - name: Install nginx
      ansible.builtin.apt:
        name: nginx
        state: present
        update_cache: true
 
    - name: Ship the login banner
      ansible.builtin.copy:
        src: files/issue.net
        dest: /etc/issue.net
        owner: root
        group: root
        mode: '0644'
 
    - name: Ensure nginx is running and enabled at boot
      ansible.builtin.service:
        name: nginx
        state: started
        enabled: true

$ ansible-playbook -i inventory.ini site.yml --limit web1.acme.internal,web2.acme.internal

PLAY [Baseline web servers] ****************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
TASK [Install nginx] ***********************************************************
changed: [web1.acme.internal]
changed: [web2.acme.internal]
 
TASK [Ship the login banner] ***************************************************
changed: [web1.acme.internal]
changed: [web2.acme.internal]
 
TASK [Ensure nginx is running and enabled at boot] *****************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2.acme.internal         : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

$ ansible-playbook -i inventory.ini site.yml --limit web1.acme.internal,web2.acme.internal

PLAY [Baseline web servers] ****************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
TASK [Install nginx] ***********************************************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
TASK [Ship the login banner] ***************************************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
TASK [Ensure nginx is running and enabled at boot] *****************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2.acme.internal         : ok=4    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

First run: changed=2 per host, because nginx got installed and the banner got written. The service task said ok rather than changed, because installing nginx on Debian and Ubuntu already starts it and enables it at boot, so there was nothing left for that task to do. Second run: changed=0 everywhere, and that zero is the whole point. It is also how you check your own work. After editing a playbook, run it, then run it again. If the second run still reports changes, some task cannot describe the state it wants, and what you have is a bug rather than a converged fleet. One aside on the playbook itself: the mode is quoted as '0644' deliberately. Unquoted, YAML reads 0644 as a number and you end up with permissions nobody asked for. And "Gathering Facts" is the free task at the top of every play, where Ansible runs the setup module to collect a few hundred details about each host (distribution and version, IP addresses, memory, mounted filesystems) that your later tasks can branch on.

There is an escape hatch, and it is where most Ansible codebases go wrong. ansible.builtin.command and ansible.builtin.shell run whatever string you hand them. They have no idea what state you were aiming for, so they report changed on every run, forever, and a playbook full of them is a bash script wearing YAML with none of the safety. When you genuinely need one, fence it in. creates: /opt/app/installed tells Ansible to skip the task when that path already exists. changed_when: false tells it that a read-only command never counts as a change. And when you want a service restarted after a config edit, use a handler: a task parked at the bottom of the play that only runs if another task notifies it by actually changing something.

Look before you touch

You would read a builder's quote before agreeing to the work. The --check flag is that quote. Ansible asks each module "would you change anything here?" and reports the answer without doing it. The --diff flag prints the exact before and after of file contents and permissions. Say you have added a second line to files/issue.net and want to know what that does to a host before it does it anywhere.

$ ansible-playbook -i inventory.ini site.yml --limit web1.acme.internal --check --diff

PLAY [Baseline web servers] ****************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web1.acme.internal]
 
TASK [Install nginx] ***********************************************************
ok: [web1.acme.internal]
 
TASK [Ship the login banner] ***************************************************
--- before: /etc/issue.net
+++ after: /home/deploy/ops/files/issue.net
@@ -1 +1,2 @@
 Authorized use only. All activity on this system is recorded.
+Disconnect now if you are not an authorized user.
 
changed: [web1.acme.internal]
 
TASK [Ensure nginx is running and enabled at boot] *****************************
ok: [web1.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=4    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Check mode is honest but not omniscient. Its accuracy depends on each module implementing it, and command and shell do not execute at all in check mode rather than guessing at an answer. So a play that installs a package and then configures it can report confusing nonsense on a host where the first step has not happened yet. Pair it with --limit, the cheapest blast-radius control Ansible gives you. Prove the change on one host, read the diff, then widen. A forgotten --limit is how a routine banner update becomes a fleet-wide outage.

The control node holds every key you own

Somewhere in most houses there is a drawer with spare keys to everything. The control node is that drawer, for the entire fleet. It holds the SSH private key every managed host trusts, the sudo rights that key implies, the Ansible Vault password that decrypts your secrets, and playbooks capable of running an arbitrary command on several hundred machines at once. An attacker who lands a shell there needs no exploit for anything else. They use the tool exactly as designed: one ad-hoc command with ansible.builtin.shell against the all group, and every host in your inventory runs their payload within minutes, from a source address that looks completely normal in your logs.

The quiet version is worse. Add three lines to a playbook in the ops repository, let the pipeline deploy it on the next merge, and the backdoor arrives through your own reviewed, audited, entirely legitimate change process. So treat the control node like a jump box. Named human accounts with multi-factor authentication, not a shared "ansible" login that four people know the password to. Set log_path in ansible.cfg so every run leaves a record, and ship that log off the box immediately, because whoever owns the control node owns any log stored on it. Keep playbooks in version control with required review, and run them from a pipeline rather than from someone's home directory. Apply become only to the tasks that need root instead of blanketing every play with it. And be clear-eyed about Vault. It is real encryption, AES-256 with a tamper check, so an encrypted vars file is genuinely unreadable rather than merely scrambled. The soft spot is the password, which is stretched into a key with 10,000 rounds of PBKDF2 (a deliberately slow way of turning a password into a key). That count was sensible in 2014 and is thin now, so a short vault password on a file that reaches a public repository is a cracking exercise, not a secret.

Where the settings live

Nearly everything above is tunable in ansible.cfg. Picture four notes pinned in four different places around the house: Ansible reads the first one it finds and throws the rest away. The order is the ANSIBLE_CONFIG environment variable, then ansible.cfg in the directory you are standing in, then ~/.ansible.cfg, then /etc/ansible/ansible.cfg. First match wins, and settings do not merge across files, which surprises people who expect layering.

[defaults]
inventory = ./inventory.ini
# leave this on: it is the default, and the only proof of host identity
host_key_checking = True
# an audit trail of every task, on every host, for every run
log_path = /var/log/ansible/ansible.log
# 5 by default: how many hosts Ansible works on at once, per task
forks = 20
 
[privilege_escalation]
become = False
become_method = sudo
 
[ssh_connection]
# module goes down stdin instead of a temp file: one SSH round trip per task
pipelining = True
# this REPLACES the built-in default rather than adding to it, so keep -C
ssh_args = -C -o ControlMaster=auto -o ControlPersist=60s

One more thing about that file, and it is a security control people trip over rather than thank. Ansible refuses to read an ansible.cfg from the current directory when that directory is world-writable, and it says so loudly. The reasoning is direct: ansible.cfg can point at arbitrary plugin and module paths, so a config file that any local user could edit is a way to make your next ansible-playbook run their code with your privileges. If your settings look like they are being ignored, run ansible --version, see which config file it actually loaded, and check the permissions on the directory holding it.

$ ls -ld /srv/shared
$ cd /srv/shared && ansible --version | head -2

drwxrwxrwx 6 root root 4096 Jul 21 09:12 /srv/shared
[WARNING]: Ansible is being run in a world writable directory (/srv/shared),
ignoring it as an ansible.cfg source. For more information see
https://docs.ansible.com/ansible/devel/reference_appendices/config.html#cfg-in-
world-writable-dir
ansible [core 2.16.3]
  config file = /etc/ansible/ansible.cfg

Try this

Run ansible --version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: never set host_key_checking = False. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related