Idempotency and check mode
Dry runs and why shell module is a smell.
A thermostat and a plug-in heater on a wall timer both warm a room. Only one of them is safe to leave running while you sleep. The thermostat reads the temperature first and does nothing at all if the room is already at 20C. The timer pumps out heat on a schedule, whether the room needs it or not. Ansible is built on the thermostat model. An operation is idempotent (a word borrowed from maths, meaning that doing it again changes nothing) when running it once or fifty times leaves the machine in exactly the same state, because every run after the first one looks around and finds nothing left to do.
Ansible does not hand you that property for free. You write it, or you fail to write it, one task at a time.
The failure idempotency prevents is drift through fear. Say a playbook appends a firewall rule every time it runs. Rerun it and you get duplicate rules and a bounced service, so running it during an incident causes a second incident. The team learns the playbook is unsafe to repeat. They stop running it. They start fixing hosts by hand instead. Within a few months the code in git no longer describes production, and nobody can say by how much.
For a security team that is a serious problem. You cannot audit a configuration you cannot reproduce, and you cannot tell a colleague's undocumented hotfix from an attacker's persistence mechanism, because both look identical from the outside: the host differs from the code. Flip the property around and the economics change completely. When reruns are free you run them constantly, and a task that reports changed on a host that should already be converged becomes a signal worth chasing.
Two ways to disable root login
# The wrong way. It cannot see what is already in the file, so it appends# the line again on every run, and the handler restarts sshd every run.- name: Disable root login (not idempotent)ansible.builtin.shell: echo 'PermitRootLogin no' >> /etc/ssh/sshd_configbecome: truenotify: restart sshd
- name: Harden SSHhosts: webbecome: truetasks:# Reads the file, rewrites the line matching the regexp only if it does# not already read "PermitRootLogin no", and writes nothing otherwise.- name: Disable root loginansible.builtin.lineinfile:path: /etc/ssh/sshd_configregexp: '^#?\s*PermitRootLogin'line: 'PermitRootLogin no'validate: /usr/sbin/sshd -t -f %snotify: restart sshdhandlers:- name: restart sshdansible.builtin.systemd_service:name: sshdstate: restarted
The shell version has no idea what is already in the file. It appends and reports changed every single time, so the handler restarts sshd (the Secure Shell server daemon, the program that accepts your remote logins) on every run, on every host, forever. The lineinfile version describes a destination instead of an action: there should be a line matching that pattern, and it should read PermitRootLogin no. Run it against a host that already says so and the module reads the file, finds nothing to do, and returns changed: false. No restart, no downtime, no noise.
One caution, because it is easy to over-read what lineinfile promises. It does not enforce one line and one only. When your regular expression matches several lines, it rewrites the last one and leaves the earlier ones exactly where they are. That detail bites hardest right here, because sshd's own manual says that for each keyword, the first obtained value will be used. So a file already carrying PermitRootLogin yes near the top and a commented #PermitRootLogin further down ends up with your no written over the comment, the untouched yes still winning, and the task reporting changed once and ok forever after. Ansible did precisely what you asked. What you asked was not what you meant. For a setting that decides who may log in as root, own the whole file with ansible.builtin.template so its content is exactly what you wrote, or drop a small file into /etc/ssh/sshd_config.d/ on distributions that ship an Include line at the top of sshd_config (Ubuntu 22.04 and later, RHEL 9 and later), where being parsed first is what makes the drop-in win.
The validate argument is the quiet hero in that task. lineinfile never edits the live file in place. It builds the new content in a temporary file, runs your validate command with %s replaced by that temporary path, and moves the file into position only if the command exits zero. /usr/sbin/sshd -t -f %s asks sshd itself to parse the candidate config. One fat-fingered directive then fails the task on host one instead of locking you out of four hundred machines at once.
There is a second reason shell earns a raised eyebrow in review. Anything you template into a shell string is a command injection waiting for the right input. Writing ansible.builtin.shell: "rm -rf /srv/{{ app_name }}" is fine until app_name arrives from a dynamic inventory or a fact gathered off the managed host, and the value contains a semicolon. ansible.builtin.command is safer because it never goes through a shell at all, and since ansible-core 2.16 it accepts expand_argument_vars: false, which stops Ansible expanding $HOME and friends inside your arguments before running them. A purpose-built module is safer still, because your arguments never become a string that a shell has to parse.
Prove it, twice
$ ansible-playbook -i inventory/prod.ini harden-ssh.yml --limit web01
PLAY [Harden SSH] **************************************************************TASK [Gathering Facts] *********************************************************ok: [web01]TASK [Disable root login] ******************************************************changed: [web01]RUNNING HANDLER [restart sshd] *************************************************changed: [web01]PLAY RECAP *********************************************************************web01 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
# exactly the same command, run straight afterwards$ ansible-playbook -i inventory/prod.ini harden-ssh.yml --limit web01
PLAY [Harden SSH] **************************************************************TASK [Gathering Facts] *********************************************************ok: [web01]TASK [Disable root login] ******************************************************ok: [web01]PLAY RECAP *********************************************************************web01 : ok=2 changed=0 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
changed=0 on the second run is the whole proof. Notice what did not happen: the handler stayed quiet. Handlers fire only when a notifying task reports changed, so honest state checks are what keep service restarts, and the seconds of downtime they cost, tied to real configuration changes. A playbook stuffed with shell tasks restarts everything on every run, which is how you end up scheduling it for 3am, which is how a tool stops being usable during an incident.
Underneath, nearly every well-written module follows the same short loop. It probes the current state (reads the file, queries the package database, stats the permissions), compares what it found against the arguments you gave it, applies a change only when that comparison comes back non-empty, then returns a result in JSON (JavaScript Object Notation, a plain-text data format) whose changed field records whether anything was genuinely touched. One boolean, and everything downstream hangs off it: the handlers, the recap counters, and check mode itself.
Check mode is a rehearsal
--check is a dress rehearsal on the real stage with the real cast, and nothing actually moved. Every module runs its probe and its comparison against the live host, then stops short of the write and reports what it would have done. It is not the same as --syntax-check, which only parses your YAML (the indented plain-text format playbooks are written in) and never contacts a host at all. Add --diff and the file-editing modules print a unified diff, the same before-and-after format git diff uses, where a line starting with a minus is being removed and a line starting with a plus is being added.
$ ansible-playbook -i inventory/prod.ini harden-ssh.yml --limit web01 --check --diff
PLAY [Harden SSH] **************************************************************TASK [Gathering Facts] *********************************************************ok: [web01]TASK [Disable root login] ******************************************************--- before: /etc/ssh/sshd_config (content)+++ after: /etc/ssh/sshd_config (content)@@ -32,7 +32,7 @@# Authentication:#LoginGraceTime 2m-PermitRootLogin yes+PermitRootLogin no#StrictModes yes#MaxAuthTries 6changed: [web01]RUNNING HANDLER [restart sshd] *************************************************changed: [web01]PLAY RECAP *********************************************************************web01 : ok=3 changed=2 unreachable=0 failed=0 skipped=0 rescued=0 ignored=0
Nothing on web01 was modified. Both the diff and the notified restart are predictions. That costs almost nothing: no writes, no restarts, no interruption. It also gives you a change-review artifact for free. Attach that output to the merge request and a reviewer can see the exact bytes your playbook wants to alter on production before anyone approves it.
Treat the output as sensitive, though. --diff prints file contents. A template carrying a database password or an API token (application programming interface token, the string one service uses to prove its identity to another) lands in your terminal scrollback and, far worse, in a CI (continuous integration) job log that outlives the run and is readable by everyone with pipeline access. Put no_log: true on any task handling secret material and Ansible throws the result away, diff included, keeping only changed, attempts and retries alongside a one-line notice that the output was hidden. Be clear about what that buys you. no_log is a curtain, not a safe. The secret still travels to the managed host and still gets written wherever your task writes it, and a value pulled out of Ansible Vault is decrypted in memory before any of this happens, so vault protects the file sitting in git and nothing else.
Where check mode lies to you
Check mode predicts one task at a time against the host as it is right now, not as it will be once the tasks above it have run. On a machine already close to converged that is fine. On a fresh one it falls apart, and not gently. Take a play where one task installs nginx and the next makes sure the service is running, then aim it at a brand-new host.
# web09 was built ten minutes ago and has nothing on it yet$ ansible-playbook -i inventory/prod.ini site.yml --limit web09 --check
PLAY [Web servers] *************************************************************TASK [Gathering Facts] *********************************************************ok: [web09]TASK [Install nginx] ***********************************************************changed: [web09]TASK [Ensure nginx is running] *************************************************fatal: [web09]: FAILED! => {"changed": false, "msg": "Could not find the requested service nginx: host"}PLAY RECAP *********************************************************************web09 : ok=2 changed=1 unreachable=0 failed=1 skipped=0 rescued=0 ignored=0
The install never happened, so there is no nginx unit for the service module to inspect, and it fails outright. A real run would have worked perfectly. Rehearse with --check against converged infrastructure, and prove first-run behaviour against throwaway hosts you build from scratch in CI. The same shape bites whenever a template writes into a directory that an earlier task creates.
Commands, and the escape hatch that makes them honest
Ansible cannot predict what an arbitrary program will do, so under --check it declines to guess. ansible.builtin.command and ansible.builtin.shell return skipped with the message "Command would have run if not in check mode", and the display shows skipping rather than a prediction. Any third-party module whose author never declared check-mode support gets the same treatment with a blunter message: remote module (foo) does not support check mode.
One exception is worth carrying around, because most people have never met it. Give the task a creates or removes path and command stops skipping and starts predicting. With creates, the module globs that path before doing anything. File already there? The task reports ok and never runs the command, in check mode and in real runs alike, which is exactly how you make a one-shot command idempotent. The wording even tells you which kind of run you are in: "Did not run command since '/etc/ssh/ssh_host_ed25519_key' exists" for real, "Would not run command since ..." under --check. File missing? Under --check the task reports changed, the prediction, and still executes nothing.
# Best: a read-only module. slurp declares check-mode support, so it runs# under --check exactly as it does in a real run.- name: Read current ip_forward settingansible.builtin.slurp:src: /proc/sys/net/ipv4/ip_forwardregister: ipfwd- name: Fail if this web host is routing trafficansible.builtin.assert:that: (ipfwd.content | b64decode | trim) == '0'fail_msg: 'net.ipv4.ip_forward is on; this host is forwarding packets'# Acceptable: a command you own, told to be honest about itself.- name: Read the running audit rule countansible.builtin.command: auditctl -lregister: auditruleschanged_when: false # reading state never mutates itcheck_mode: false # run this one even under --checkbecome: true# A one-shot command, made idempotent and check-mode aware by `creates`.- name: Generate the host key exactly onceansible.builtin.command:cmd: ssh-keygen -t ed25519 -f /etc/ssh/ssh_host_ed25519_key -N ''creates: /etc/ssh/ssh_host_ed25519_keybecome: true
The slurp version is the one to reach for. It is a real module (it reads a remote file, encodes the bytes as base64 so any binary content survives the trip, and hands them back), it declares check-mode support so it runs everywhere without special-casing, it needs no changed_when, and it gives you a value you can assert on. That assertion is a security control in its own right: it fails the play on any web server that has quietly turned into a router.
The check_mode keyword works in both directions, which is the half most people never use. check_mode: false forces a task to execute for real during a --check run. check_mode: true does the reverse, pinning a task to rehearsal mode permanently so it never changes anything, even on a normal run. That is how you ship an audit-only task inside an otherwise enforcing playbook: a lineinfile that reports whether /etc/login.defs matches your password-ageing baseline, running nightly with --diff, changing nothing, while you are still negotiating the rollout with the team that owns those hosts.
Turning changed=0 into a control
Make idempotence something your pipeline enforces rather than something you hope for. Molecule, the standard test harness for Ansible roles, has an idempotence step baked into its default test sequence: it converges a throwaway container, converges it a second time, and fails the build if that second pass reports a single change. Without Molecule the gate is four lines of shell. Run the playbook twice against an ephemeral host and refuse the merge unless the second recap says changed=0.
At fleet scale, check mode becomes a detection control. A nightly ansible-playbook site.yml --check --diff across every host costs a fraction of a real run and produces a precise list of machines whose state has wandered away from the code: a hand-edit, a half-finished deploy, or an intruder flipping PermitRootLogin back to yes to keep a way in. You get file-integrity monitoring for every setting Ansible manages, with the remediation already written, reviewed and tested.
$ ansible-playbook -i inventory/prod.ini site.yml --check --diff > drift-$(date +%F).log$ echo "exit status: $?"$ grep -E 'changed=[1-9]' drift-$(date +%F).log
exit status: 0web07 : ok=42 changed=1 unreachable=0 failed=0 skipped=3 rescued=0 ignored=0
Look closely at that exit status. --check returns 0 whether it predicts zero changes or four hundred, because the exit code only reflects failed and unreachable hosts. A CI job written as ansible-playbook --check && echo "no drift" will cheerfully report no drift forever. Parse the recap, or run the play through a machine-readable callback, and alert on changed above zero for any host you believe is converged.
Two honest limits before you build a dashboard on this. It only sees what Ansible manages, so a rogue systemd unit sitting in a directory no task touches stays invisible. And every non-idempotent task in your repo is a permanent false positive: a role with five shell tasks reports changed on every host every night, and a report that is always red is a report nobody reads. That is the practical reason shell is a smell. Switch on the ansible-lint rules that catch it, command-instead-of-module, command-instead-of-shell, no-changed-when and risky-shell-pipe, and every one you fix makes the nightly drift report quieter, until the only thing left in it is something that actually happened.