Modules & idempotency
The unit of work that runs safely twice.
A light switch obeys. A thermostat decides. Flip the switch and it flips, whatever the room was doing a second earlier. Set a thermostat to 20 degrees and it reads the room first, then fires the furnace only if there is a gap to close. An Ansible module is the thermostat, wired to one slice of a machine: a package, a file, a service, a user account. You describe the finished state. The module works out whether anything needs doing. That one design choice is why you can run the same play at 3pm, run it again at 3:05pm, and not break production.
One Module, One Job
Modules are specialists you call out to a job. The locksmith touches locks. The plumber touches pipes. Neither has an opinion about the other's work. In Ansible a module is a small program that gets copied to the managed host, runs there under the host's Python interpreter, and is cleaned up afterwards, and every task in a playbook calls exactly one of them. You name a module by its FQCN (fully qualified collection name, the full three-part address, like ansible.builtin.copy). The bare short name copy still works, and it is legacy: install collections other people wrote and two of them can end up owning the same short name, and you will not enjoy finding out which one ran at 2am. Everything under ansible.builtin ships inside ansible-core itself. Everything else arrives through ansible-galaxy collection install.
The contract stays the same whether you fire one module at a group of hosts right now (an ad-hoc command: -m names the module, -a carries its arguments) or write the same call as a task in a playbook. You hand the module parameters that describe a result, never a list of steps. name=nginx state=present means nginx should be installed on this host. It does not mean run apt-get now. ansible.builtin.package reads the pkg_mgr fact (a fact is a detail Ansible measured about the host before your tasks ran) and calls apt on Debian or dnf on Rocky without you writing a branch. Watch what the same command does twice in a row. --become is the flag that escalates to root, through sudo by default.
$ ansible web -i inventory.ini -m ansible.builtin.file \-a "path=/etc/app state=directory mode=0755" --become# identical command, run again straight away$ ansible web -i inventory.ini -m ansible.builtin.file \-a "path=/etc/app state=directory mode=0755" --become
web1.acme.internal | CHANGED => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"},"changed": true,"gid": 0,"group": "root","mode": "0755","owner": "root","path": "/etc/app","size": 4096,"state": "directory","uid": 0}web1.acme.internal | SUCCESS => {"ansible_facts": {"discovered_interpreter_python": "/usr/bin/python3"},"changed": false,"gid": 0,"group": "root","mode": "0755","owner": "root","path": "/etc/app","size": 4096,"state": "directory","uid": 0}
Read the word before the arrow. CHANGED says the module altered the host to match what you asked for. SUCCESS says the host already matched and the module kept its hands in its pockets. Same command, same parameters, different verdict, because the verdict describes the host and not the effort. Both runs still report the mode, the owner and the size, so even the quiet run hands you a small audit of that path. The third verdict you will meet is FAILED!, which means the module could not get the host to the state you asked for, and it will tell you why.
Idempotent Means the Second Run Is Boring
Jab the call button by the lift ten times and one lift still comes. Every press after the first one changes nothing. That property has a name, idempotent, and it is what makes running the same play over and over a safe habit rather than a gamble. A module earns the property by measuring before it acts: it asks the host what is true right now, compares that against your parameters field by field, and touches only the fields that disagree. ansible.builtin.copy takes a fingerprint (a checksum) of the file it is about to write and holds it against the fingerprint of the one already on disk. ansible.builtin.service asks the init system (systemd on any modern Linux, the program that starts services and keeps them running) whether the unit is up and whether it is set to come back after a reboot. No gap, no work.
- name: Web tier baselinehosts: webbecome: truetasks:- name: Ensure nginx is installedansible.builtin.package:name: nginxstate: present- name: Ensure nginx runs now and after a rebootansible.builtin.service:name: nginxstate: startedenabled: true- name: Deploy the site configansible.builtin.copy:src: files/site.confdest: /etc/nginx/conf.d/site.confowner: rootgroup: rootmode: "0644"
$ ansible-playbook -i inventory.ini site.yml# nothing edited, nothing deployed, run it straight back$ ansible-playbook -i inventory.ini site.yml
PLAY [Web tier baseline] ******************************************************TASK [Gathering Facts] ********************************************************ok: [web1.acme.internal]TASK [Ensure nginx is installed] **********************************************changed: [web1.acme.internal]TASK [Ensure nginx runs now and after a reboot] *******************************changed: [web1.acme.internal]TASK [Deploy the site config] *************************************************changed: [web1.acme.internal]PLAY RECAP ********************************************************************web1.acme.internal : ok=4 changed=3 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0PLAY [Web tier baseline] ******************************************************TASK [Gathering Facts] ********************************************************ok: [web1.acme.internal]TASK [Ensure nginx is installed] **********************************************ok: [web1.acme.internal]TASK [Ensure nginx runs now and after a reboot] *******************************ok: [web1.acme.internal]TASK [Deploy the site config] *************************************************ok: [web1.acme.internal]PLAY RECAP ********************************************************************web1.acme.internal : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
ok=4 counts four tasks that finished cleanly, and the first of the four is Gathering Facts, a step Ansible slots in at the top of every play to collect details about the host (package manager, init system, addresses, disks) before your own tasks run. The first run reports changed=3 because the host was bare. The second reports changed=0 with every task sitting on ok, and that zero is the proof you came for. It says the play converged: the host now matches the file, and running it again costs three cheap questions and no writes. A task that reports changed on every single run is a defect, not a personality. Something inside it is acting before it measures.
Changed Is Your Drift Alarm
A smoke alarm that chirps every night gets its battery pulled by March. A quiet second run buys you the opposite: changed stops being background noise and becomes a signal worth waking someone for. Someone opens an SSH (secure shell, the encrypted remote login you use to reach a server) session to web1 at 2am and widens a listen directive to chase a customer bug. A stolen deploy token rewrites a config. A colleague hand-patches a file and never opens the pull request. The next scheduled run notices the difference, puts the file back the way the repository describes it, and prints changed on exactly that task, on exactly that host. That is drift detection, drift being the slow wander of a live host away from what your files say it should be, and you get it for every file, package and service your play already describes without buying anything. Bury that signal under fifteen tasks that shout changed every night and nobody will ever spot the one that matters.
Re-asserting state is blunt, and you should know what you are agreeing to. Ansible will overwrite the emergency fix a human made at 3am with whatever git says, without asking, because the repository is the source of truth by design. The fix belongs in the repository or it does not survive the week. When you want to look without touching, --check is Ansible's dry run: modules work out what they would do and report it, writing nothing. Add --diff and you get the actual bytes.
# somebody edited the config on the box by hand last night$ ansible-playbook -i inventory.ini site.yml --check --diff
PLAY [Web tier baseline] ******************************************************TASK [Gathering Facts] ********************************************************ok: [web1.acme.internal]TASK [Ensure nginx is installed] **********************************************ok: [web1.acme.internal]TASK [Ensure nginx runs now and after a reboot] *******************************ok: [web1.acme.internal]TASK [Deploy the site config] *************************************************--- before: /etc/nginx/conf.d/site.conf+++ after: /home/deploy/infra/files/site.conf@@ -1,5 +1,5 @@server {- listen 8080;+ listen 80;server_name app.acme.internal;root /srv/app/public;}changed: [web1.acme.internal]PLAY RECAP ********************************************************************web1.acme.internal : ok=4 changed=1 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
The two headers tell you which side is which. before: is the path on the managed host, after: is the source file on your control node. So the box is serving on 8080, the repository says 80, and you know it within one line of output. Nothing was written, because --check writes nothing. A --check --diff pass across the fleet on a timer is a drift report you already own, it needs no extra agent installed on the hosts, and the diff itself is the evidence you paste into the incident ticket. There is one class of task it cannot tell you the truth about, and it is the one you are most tempted to write.
The Escape Hatch: Command and Shell
Every toolbox has the hammer you reach for when nothing else fits, and two modules are that hammer. ansible.builtin.command runs a program directly with the arguments you give it, no shell involved. ansible.builtin.shell hands your string to /bin/sh on the target, so pipes, redirects, wildcards and $VARIABLES all work. That convenience is also the security difference: a shell string that pastes in a variable from inventory, from a fact, or from anything a user can influence is a command injection waiting for a semicolon. Prefer a real module. Failing that, prefer command. Neither of the two has any idea what finished looks like, so both do the work and report changed every time, forever.
- name: Web tier chores, unmanagedhosts: webbecome: truetasks:- name: Rotate the deploy keyansible.builtin.shell: /usr/local/sbin/rotate-keys.sh- name: Check the running kernelansible.builtin.command: uname -rregister: kernel- name: Report the kernelansible.builtin.debug:msg: "kernel is {{ kernel.stdout }}"
# fourth run in a row, nothing on the host has moved$ ansible-playbook -i inventory.ini bad.yml# and now the dry run you were going to trust$ ansible-playbook -i inventory.ini bad.yml --check
PLAY [Web tier chores, unmanaged] *********************************************TASK [Gathering Facts] ********************************************************ok: [web1.acme.internal]TASK [Rotate the deploy key] **************************************************changed: [web1.acme.internal]TASK [Check the running kernel] ***********************************************changed: [web1.acme.internal]TASK [Report the kernel] ******************************************************ok: [web1.acme.internal] => {"msg": "kernel is 6.8.0-79-generic"}PLAY RECAP ********************************************************************web1.acme.internal : ok=4 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0PLAY [Web tier chores, unmanaged] *********************************************TASK [Gathering Facts] ********************************************************ok: [web1.acme.internal]TASK [Rotate the deploy key] **************************************************skipping: [web1.acme.internal]TASK [Check the running kernel] ***********************************************skipping: [web1.acme.internal]TASK [Report the kernel] ******************************************************ok: [web1.acme.internal] => {"msg": "kernel is "}PLAY RECAP ********************************************************************web1.acme.internal : ok=2 changed=0 unreachable=0 failed=0 skipped=2 rescued=0 ignored=0
Two problems on one screen. Reading the kernel version changes nothing on the host, yet it books a changed every run, so the recap lies and any handler watching that task fires for nothing. A handler, in case you have not met one yet, is a task that only runs when something it watches reports a change: the nginx reload, the auditd restart. Then look at the dry run, where both tasks vanish. command and shell do support check mode, but only partly: with no creates or removes to look at, they skip themselves and return msg: Command would have run if not in check mode. Now watch what that does to the task underneath. register had saved the command's result into a variable called kernel, and kernel.stdout came back as an empty string rather than an error, so the debug printed kernel is and nothing after it. Every conditional reading that variable quietly takes the other branch.
--check, a command or shell task with no creates or removes returns skipped: true, rc: 0 and stdout: "". Tasks downstream do not fail loudly; they read an empty string and a zero exit code and carry on, so a when: flips, a set_fact stores nothing, and the plan you approved has little to do with the run that follows. Index into the empty result (kernel.stdout_lines[0]) and you finally do get a hard failure: list object has no element 0. Give every command a creates or removes so it can predict itself, or check_mode: false when it only reads and is safe to run for real during a dry run.Three controls put the honesty back, and you pick by what the command actually does. creates, with its mirror removes, names a path that proves the work is already done, so the module looks for that path first and skips the command when it is there. changed_when: false labels a read-only command as never changing anything, which is right for uname -r, for kubectl get nodes, for any check you registered so you could make a decision later. changed_when with an expression reads the command's own output and books a change only when that output says work happened, and failed_when does the same job for the failure verdict when a non-zero exit code is not really an error. While we are in here, the rotate task never needed a shell, so it moves to command.
- name: Web tier chores, made honesthosts: webbecome: truetasks:- name: Rotate the deploy key, changed only if it rotatedansible.builtin.command: /usr/local/sbin/rotate-keys.shregister: rotatechanged_when: "'rotated' in rotate.stdout"- name: Check the running kernelansible.builtin.command: uname -rregister: kernelchanged_when: false # read-only, never a changecheck_mode: false # and safe to run during a dry run- name: Run the vendor installer, but only onceansible.builtin.command:cmd: /opt/vendor/install.sh --prefix /opt/vendorcreates: /opt/vendor/bin/vendord
$ ansible-playbook -i inventory.ini good.yml# why did the installer task stay quiet? -v prints what the module returned$ ansible-playbook -i inventory.ini good.yml -v | grep -A1 "vendor installer"
PLAY [Web tier chores, made honest] *******************************************TASK [Gathering Facts] ********************************************************ok: [web1.acme.internal]TASK [Rotate the deploy key, changed only if it rotated] **********************ok: [web1.acme.internal]TASK [Check the running kernel] ***********************************************ok: [web1.acme.internal]TASK [Run the vendor installer, but only once] ********************************ok: [web1.acme.internal]PLAY RECAP ********************************************************************web1.acme.internal : ok=4 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0TASK [Run the vendor installer, but only once] ********************************ok: [web1.acme.internal] => {"changed": false, "cmd": ["/opt/vendor/install.sh", "--prefix", "/opt/vendor"], "delta": null, "end": null, "msg": "Did not run command since '/opt/vendor/bin/vendord' exists", "rc": 0, "start": null, "stderr": "", "stderr_lines": [], "stdout": "skipped, since /opt/vendor/bin/vendord exists", "stdout_lines": ["skipped, since /opt/vendor/bin/vendord exists"]}
Three raw commands in one play, and the recap still reads changed=0. The -v line shows the mechanism. creates sent the module looking for /opt/vendor/bin/vendord, it found it, and the module returned Did not run command since '/opt/vendor/bin/vendord' exists with rc: 0, empty timings, and no installer process at all. Notice the display says ok, not skipping, because the module ran and made a decision. The same check works under --check, which is the part people miss: a command with creates predicts itself honestly in a dry run, reporting changed when the marker file is missing and ok when it is there. The key rotation still skips during a dry run, and that is the honest answer, because the only way to know what that script would do is to let it do it.
changed_when: false does not stop the command running. It only tells Ansible not to count it. Put it on a command that genuinely modifies the host and you have muted your own alarm: the recap stays a clean changed=0 while the box drifts underneath it, and every handler waiting on that task (the nginx reload, the auditd restart, the config validation) never fires. Use it for reads. For anything that writes, drive changed_when off the command's own output, or go and find the module that measures state properly.Prove It in the Pipeline
Two cheap gates catch nearly all of this before a play touches production. ansible-lint reads your YAML (the indented text format playbooks are written in) and flags no-changed-when on every command or shell task with no honesty control attached, plus command-instead-of-shell wherever you reached for a shell you did not need. It runs in seconds, talks to no hosts and needs no credentials, so it belongs on every pull request. Point it at the broken play from earlier.
$ ansible-lint bad.yml
WARNING Listing 3 violation(s) that are fatalcommand-instead-of-shell: Use shell only when shell functionality is required.bad.yml:5 Task/Handler: Rotate the deploy keyno-changed-when: Commands should not change things if nothing needs doing.bad.yml:5 Task/Handler: Rotate the deploy keyno-changed-when: Commands should not change things if nothing needs doing.bad.yml:8 Task/Handler: Check the running kernelRead documentation for instructions on how to ignore specific rule violations.Rule Violation Summarycount tag profile rule associated tags1 command-instead-of-shell basic command-shell, idiom2 no-changed-when shared command-shell, idempotencyFailed: 3 failure(s), 0 warning(s) on 1 files. Last profile that met the validation criteria was 'min'.
The linter reads text, so it can only catch the tasks that look wrong. The second gate proves the real thing on a real host: run the play, run it again, and fail the build if the second run reports any change at all. Molecule, the role testing tool, ships this as molecule idempotence and runs it against a throwaway container. In a plain CI pipeline (continuous integration, the checks that fire automatically on every change you push) it is a dozen lines of shell against a staging host.
#!/usr/bin/env bashset -euo pipefail# First run: bring the staging host to the state the repo describes.ansible-playbook -i inventory.ini site.yml# Second run: nothing should move. Anything that moves is a defect.ansible-playbook -i inventory.ini site.yml | tee /tmp/second.logif grep -qE 'changed=[1-9]' /tmp/second.log; thenecho "play is not idempotent: something changed on the second run"exit 1fi
When that gate goes red, do not start reading the whole log. Run the play a third time with --diff and look only at the task names that flip to changed. A file task that flips every run is usually an ansible.builtin.template rendering a timestamp or a freshly generated value into the output, or an ansible.builtin.lineinfile whose regexp can never match the line it writes, so it appends another copy on every pass. A command task that flips is missing its creates. The task name on the yellow line is the file to open.
ok=4 changed=0. What has that proved?skipping: and lands in the skipped= column.ok means the module measured, compared and found no gap to close, so the play has converged on the state the file describes.failed= and printed in red, so ok=4 with nothing failed is the healthy outcome.changed on every run, which is the opposite of this output.ansible.builtin.command: /usr/local/bin/audit.sh with register: audit and nothing else set. You run the play with --check. What does the next task see in audit.stdout?creates or removes, command skips under check mode and returns rc: 0 with stdout left as an empty string, so conditionals downstream silently take the wrong branch.check_mode: false to the task.stdout_lines[0]) raises an error.ansible.builtin.shell: /usr/sbin/logrotate -f /etc/logrotate.d/audit. The host is healthy. Which fix keeps the alarm useful?-f forces a rotation on every run, so the task really does change the host, and labelling it as never-changing mutes a genuine change plus any handler waiting on it.changed meaning something happened.Try this
Run ansible-playbook -i inventory.ini site.yml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: a dry run over raw commands is a comfortable lie. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.