Courses Ansible for secure automationRunning it safely

Guarding destructive tasks

assert, when, and validate before you break it.

Advanced12 min · lesson 8 of 12

A deploy playbook ran against forty web servers on a Tuesday afternoon. Thirty-nine got a clean release. The fortieth lost every application on the box. One line in one host_vars file (the per-host variables file Ansible loads for that machine) said app_name: with nothing after it, and the task that clears the release directory was written as rm -rf {{ app_dir }}/*.

Nothing about that run was a bug in Ansible. The variable was defined. It happened to be defined as nothing. Ansible built the string exactly as instructed, handed rm -rf /srv/* to a shell running as root, and reported changed. Green recap, empty disk.

A pharmacist does not fill a prescription because the handwriting looks confident. They check the name, the dose and the allergy list, and they refuse the whole thing if a single field is blank. Destructive automation needs the same reflex. A dangerous task should not ask what you told it to do. It should ask whether the world looks the way you claimed it would, and refuse when it does not.

The variable that came out empty

Undefined and empty look like the same problem to a human skimming a variables file. To Ansible they are nothing alike. Undefined is a letter that was never posted: no inventory file, no vars file, no role default and no command-line flag ever set that name. When Ansible tries to fill in the {{ }} placeholder (that is Jinja, the templating language it uses to build strings out of variables), it raises an error and the task fails before a single byte moves. That is the safe outcome. Empty is a letter that turns up with no address on the envelope. A zero-length string is a real value, and it renders perfectly happily into the middle of a command.

group_vars/web.yml

app_name: shop
app_dir: "/srv/{{ app_name }}"

host_vars/web3.acme.internal.yml

# somebody blanked this during a rename and never put it back
app_name: ""

deploy.yml

- name: Redeploy the shop app
  hosts: web
  become: true
 
  tasks:
    - name: Clear the release directory
      ansible.builtin.shell: rm -rf {{ app_dir }}/*

terminal

$ ansible-playbook -i inventory.ini deploy.yml --limit web3.acme.internal

output

PLAY [Redeploy the shop app] ***************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web3.acme.internal]
 
TASK [Clear the release directory] *********************************************
changed: [web3.acme.internal]
 
PLAY RECAP *********************************************************************
web3.acme.internal         : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

app_dir came out as /srv/ and the asterisk took care of the rest. It is worth knowing which safety nets here are real and which are folklore. GNU coreutils (the standard command-line tools on any Linux box) refuses rm -rf / outright, and you have to pass --no-preserve-root to make it obey, so the literal root directory is protected. Nothing protects /srv/* or /*, because the shell expands the asterisk into a list of ordinary paths before rm ever sees it. That expansion, called globbing, only happens with ansible.builtin.shell, which pipes your string through a shell on the managed host. ansible.builtin.command starts no shell at all, so an asterisk stays a literal asterisk. Reach for command first, and for shell only when you genuinely need a pipe, a redirect or a wildcard.

terminal

$ ansible web3.acme.internal -i inventory.ini -b \
    -m ansible.builtin.shell -a "ls -A /srv | wc -l"

output

web3.acme.internal | CHANGED | rc=0 >>
0

The -A flag counts hidden entries too, so that zero is the whole directory rather than a polite subset of it. Thirty-nine hosts fine, one host bare.

A gate at the top of the play

A pilot's pre-flight checklist is not a list of things to do. It is a list of reasons not to take off. ansible.builtin.assert is that checklist. You hand it conditions under the that key, and if any one of them comes back false the task fails and drops that host out of the play. Put it in pre_tasks (the tasks a play runs before any of its roles) and it fires before your roles, before your handlers, before anything has a chance to touch the disk.

deploy.yml

- name: Redeploy the shop app
  hosts: web
  become: true
 
  pre_tasks:
    - name: Refuse to run without a real target directory
      ansible.builtin.assert:
        that:
          - app_dir is defined
          - app_dir | length > 0
          - "app_dir is match('^/srv/[a-z0-9][a-z0-9-]{1,30}$')"
        fail_msg: "app_dir resolved to '{{ app_dir | default('UNDEFINED') }}'. Refusing to run destructive tasks; it must look like /srv/<appname>."
        success_msg: "Target directory is {{ app_dir }}."
        quiet: true

Three things in that block are worth slowing down for. The conditions are Jinja expressions written without the curly braces, the same way you write when. The order carries weight, because assert returns on the first condition that comes back false, so app_dir is defined has to sit above anything that reads app_dir. And match is a genuine Ansible test that wraps Python's re.match, which anchors the pattern at the start of the string; the dollar sign at the end of the pattern pins the other end. That is what turns "not empty" into "actually shaped like a path I own". Write fail_msg for the colleague reading it at 3am: what the value was, and what it should have been.

terminal

$ ansible-playbook -i inventory.ini deploy.yml

output

TASK [Gathering Facts] *********************************************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
ok: [web3.acme.internal]
 
TASK [Refuse to run without a real target directory] ***************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
fatal: [web3.acme.internal]: FAILED! => {"assertion": "app_dir is match('^/srv/[a-z0-9][a-z0-9-]{1,30}$')", "evaluated_to": false, "msg": "app_dir resolved to '/srv/'. Refusing to run destructive tasks; it must look like /srv/<appname>."}

Ansible names the assertion that tripped and tells you what it evaluated to. web3 is out of the play now and will not see another task. The other two carry on, which is usually right for a rolling deploy and badly wrong for a fleet-wide destructive change. Set any_errors_fatal: true at play level and one failing host stops the run for everybody, before any of the forty get touched. Reach for ansible.builtin.fail when the check is not a tidy list of conditions, for example inside a rescue block or after a registered command whose output you need to pick apart first. It accepts a msg and nothing else, and it always fails.

Mandatory catches typos, not blanks

The mandatory filter is the bouncer who checks that you are holding a ticket at all. It does not read what is printed on it. Pass a variable through it and the value comes back untouched if it is defined, and the run stops if it is not. It reads well inline, and it fails at templating time, before the module gets anywhere near the filesystem.

deploy.yml

    - name: Remove the release we are replacing
      ansible.builtin.file:
        path: "{{ superseded_release_dir | mandatory }}"
        state: absent

Misspell that variable name and the task dies with "Mandatory variable 'superseded_release_dir' not defined." before the file module is ever shipped to the host. Read that sentence again, because it is also the limit of the tool. mandatory catches undefined. It does not catch empty. A zero-length string sails straight through, which is exactly the case that emptied /srv on web3. Treat mandatory as a cheap typo-catcher for variable names and keep the length check in your assert. You can supply your own wording with mandatory('set superseded_release_dir before running this play') when the stock message would mean nothing to whoever hits it.

Do not let a config flag be the thing keeping undefined variables loud

You will find error_on_undefined_vars in old ansible.cfg files, in the [defaults] section (the setting Ansible knows internally as DEFAULT_UNDEFINED_VAR_BEHAVIOR). It defaults to true. On the versions that still act on it, that default is doing real work: an undefined variable stops the task. Turn it off and the expression is left in place exactly as written instead, so rm -rf {{ app_dir }}/* becomes a command containing literal curly braces, a config file gets a template tag baked into it, and nothing fails loudly. People flip it to quieten noisy playbooks, and every future typo turns from a stopped run into a silent wrong answer. Current ansible-core no longer reads the setting at all; it is deprecated, with removal scheduled for 2.23. So if you inherit that line, do not assume it is either protecting you or endangering you until you check which version you actually run. The guard that holds on every version is the assert you wrote yourself.

When as a guard, not as a shortcut

assert refuses to let the van leave the depot. when is a lock on one specific door. Both are useful, and they behave very differently when they trip: assert fails the host, when quietly skips the task. On a destructive task, quiet skipping is exactly what you want. Conditions are evaluated per host, so on a forty-host run you can genuinely end up with thirty-nine skips and one deletion.

deploy.yml

    - name: Remove the superseded release
      ansible.builtin.file:
        path: "{{ app_dir }}/releases/{{ superseded_release }}"
        state: absent
      when:
        - superseded_release is defined
        - superseded_release | length > 0
        - superseded_release != current_release

output

TASK [Remove the superseded release] *******************************************
skipping: [web1.acme.internal]
skipping: [web2.acme.internal]
changed: [web3.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
web2.acme.internal         : ok=4    changed=0    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
web3.acme.internal         : ok=5    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The third condition is doing something different from the other two. It checks that you are not about to delete the release currently serving traffic. The guards worth writing are usually about the state of the world, not the state of a variables file. The trade-off is that a skip looks like success. Misspell the variable name inside a when clause and Ansible errors out, which is good. "Fix" that by writing superseded_relase | default('') and the error becomes a permanent silent skip: your cleanup task stops running, forever, and the recap still looks healthy. Use default() on values you genuinely want a fallback for, never on the names inside a guard.

Let the module check the ground first

A removals firm sends someone to look at the flat before backing the lorry up to it. ansible.builtin.command and ansible.builtin.shell can take that same look for you, on the managed host, before the command runs at all. creates takes a path: if something already exists there, the command does not run. removes takes a path: the command runs only when that path exists. Both accept wildcard patterns as well as plain paths.

decommission.yml

    - name: Initialise the database schema exactly once
      ansible.builtin.command:
        cmd: /usr/local/bin/shop-admin db init
        creates: /var/lib/shop/.schema-initialised
 
    - name: Run the vendor uninstaller once
      ansible.builtin.command:
        cmd: /opt/acme/bin/uninstall.sh --purge
        removes: /opt/acme/bin/uninstall.sh

terminal

$ ansible-playbook -i inventory.ini decommission.yml --limit web1.acme.internal
$ ansible-playbook -i inventory.ini decommission.yml --limit web1.acme.internal

output

# first run
TASK [Initialise the database schema exactly once] *****************************
changed: [web1.acme.internal]
 
TASK [Run the vendor uninstaller once] *****************************************
changed: [web1.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 
# second run
TASK [Initialise the database schema exactly once] *****************************
ok: [web1.acme.internal]
 
TASK [Run the vendor uninstaller once] *****************************************
ok: [web1.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Look closely at what the second run does not say. It does not say skipping. The command module returns changed: false with a message explaining itself, and it never sets the skipped flag on this path, so the recap files both tasks under ok and skipped stays at zero. Run with -v and the reason comes back word for word: "Did not run command since '/opt/acme/bin/uninstall.sh' does not exist". Plenty of people expect a skip count here and write dashboards or CI checks against a number that never moves.

The uninstaller deletes itself, so removes makes that task fire at most once no matter how often the play runs. Be honest about what you are buying, though. This is a presence check, not a correctness check. If the sentinel file exists but the schema migration behind it half-failed, creates will happily do nothing forever and Ansible will never tell you. For anything a real module can express, use the real module: ansible.builtin.file with state: absent already knows how to do nothing when the path is gone.

Check the file before it replaces the live one

A locksmith cuts your new key and tries it in a spare cylinder on the bench. They do not test it by locking your front door at midnight and hoping. template and copy do the same for you through the validate option. The rendered content lands in a temporary file on the managed host, Ansible runs your validation command with %s replaced by that temporary path, and only if the command exits zero does the file get moved into place. A file that fails validation is never the live file, not for an instant.

templates/sudoers-deploy.j2

# Managed by Ansible. Local edits will be overwritten.
Cmnd_Alias DEPLOY_SVC = /usr/bin/systemctl restart {{ app_name }}
%deploy ALL=(root) NOPASSWORD: DEPLOY_SVC

deploy.yml

    - name: Install the deploy sudo rule
      ansible.builtin.template:
        src: sudoers-deploy.j2
        dest: /etc/sudoers.d/deploy
        owner: root
        group: root
        mode: "0440"
        backup: true
        validate: /usr/sbin/visudo -cf %s

terminal

$ ansible-playbook -i inventory.ini deploy.yml --limit web1.acme.internal

output

TASK [Install the deploy sudo rule] ********************************************
fatal: [web1.acme.internal]: FAILED! => {"exit_status": 1, "msg": "failed to validate", "stderr": ">>> /root/.ansible/tmp/ansible-tmp-1753106512.84-4471-1766234298/source: syntax error near line 3 <<<\nvisudo: parse error in /root/.ansible/tmp/ansible-tmp-1753106512.84-4471-1766234298/source near line 3\n", "stdout": ""}

NOPASSWORD is not a sudoers keyword. The correct spelling is NOPASSWD. Without validate, that file lands in /etc/sudoers.d/ and sudo refuses to parse the whole directory, which means nobody on that host can use sudo again, including you, including the account Ansible was going to reconnect with. Notice the path in the error: it points at the temporary copy, not at /etc/sudoers.d/deploy. Check the live configuration and you can see for yourself.

terminal

$ ansible web1.acme.internal -i inventory.ini -b \
    -m ansible.builtin.command -a "/usr/sbin/visudo -c"

output

web1.acme.internal | CHANGED | rc=0 >>
/etc/sudoers: parsed OK
/etc/sudoers.d/README: parsed OK

The deploy file is not in that listing at all, because it was never created. One wrinkle if you are replacing an existing file rather than installing a new one: the backup is taken before the validation command runs, so a failed validate can leave a timestamped copy of the old file sitting next to an original that was never touched. Untidy, not dangerous, and the naming rules in the next section are the reason it stays harmless.

The same pattern covers the other file that locks you out of your own fleet. validate: /usr/sbin/sshd -t -f %s runs the SSH daemon (SSH is Secure Shell, the protocol Ansible uses to reach every managed host) in test mode against the candidate config. That check reads the host keys as part of its work, so the task needs become: true or it fails for reasons that have nothing to do with your config. On distributions that ship an Include line pulling in /etc/ssh/sshd_config.d/*.conf, validating one drop-in fragment on its own proves the fragment parses and nothing more. It says nothing about what the merged result ends up meaning.

validate checks grammar, not meaning

sshd -t will cheerfully approve a syntactically perfect config containing AllowUsers nobody, and visudo -cf will approve a sudoers file that grants nothing to anyone. Both are valid. Both lock you out. validate protects you from the typo class of outage, which is most of them, and gives you nothing at all against a change that is spelled correctly and wrong. Keep a second SSH session open while you push SSH changes, keep an out-of-band console for the hosts that matter, and pair the change with a task that reconnects and proves login still works before you move on to the next batch.

One more limit is worth knowing before you lean on a dry run. In check mode the file is never written, so there is nothing to validate, and the module returns before your validation command would have run. ansible-playbook --check --diff will show you the rendered sudoers file and tell you it would change. It will not tell you it is broken. Only a real run does that.

Keep the file you are about to overwrite

Before an electrician rips out old wiring, they photograph it. backup: true on template, copy, lineinfile or blockinfile takes a dated copy of the existing file before the new one replaces it, and hands the path back in the task result as backup_file. The name is the original, plus the process ID of the module run (the PID, the number Linux gives every running program), plus the date and the time.

terminal

$ ansible web1.acme.internal -i inventory.ini -b \
    -m ansible.builtin.command -a "ls -l /etc/ssh"

output

web1.acme.internal | CHANGED | rc=0 >>
total 608
-rw-r--r-- 1 root root 577388 Mar 12 09:14 moduli
-rw-r--r-- 1 root root   1650 Mar 12 09:14 ssh_config
drwxr-xr-x 2 root root   4096 Mar 12 09:14 ssh_config.d
-rw------- 1 root root    505 Mar 12 09:16 ssh_host_ecdsa_key
-rw-r--r-- 1 root root    179 Mar 12 09:16 ssh_host_ecdsa_key.pub
-rw------- 1 root root    411 Mar 12 09:16 ssh_host_ed25519_key
-rw-r--r-- 1 root root     99 Mar 12 09:16 ssh_host_ed25519_key.pub
-rw------- 1 root root   2602 Mar 12 09:16 ssh_host_rsa_key
-rw-r--r-- 1 root root    571 Mar 12 09:16 ssh_host_rsa_key.pub
-rw------- 1 root root   3268 Jul 21 14:32 sshd_config
-rw------- 1 root root   3211 Jul 14 09:05 sshd_config.4471.2026-07-21@14:32:07~
drwxr-xr-x 2 root root   4096 Jul 21 14:32 sshd_config.d

The backup keeps the original's modification time, and its owner and permission bits come across too, so a 0600 root-owned file stays 0600 and root-owned. That is why you can tell at a glance that the previous config had been in place since 14 July. Two practical notes go with it. Ansible never cleans these up, so a file rewritten weekly grows a pile of copies forever. And a backup of a config holding a database password is a plaintext copy of that password sitting somewhere nobody thinks to look.

Be careful what you expect no_log to do about that second one. no_log: true keeps a task's arguments and return values out of Ansible's own console output and its log file on the control node. It changes nothing on the managed host and it encrypts nothing. The password in that backup is exactly as readable as it was a second ago. What actually helps is pulling old copies off the box and deleting them on a schedule. The good news is that the leftovers will not be loaded by accident in the usual places: sudo ignores any file in /etc/sudoers.d whose name contains a dot or ends in a tilde, cron applies much the same naming rules to /etc/cron.d, and a backup of site.conf no longer matches an include pattern of *.conf.

Make command and shell tell the truth

A smoke alarm that goes off every time you make toast is an alarm nobody hears. Ansible can tell whether ansible.builtin.package installed something. It has no idea what your shell one-liner did, so command and shell report changed every single time they run successfully, and fail whenever the exit code is anything other than zero. Both defaults are wrong for a task that only reads. grep returns 1 when it finds nothing, and that is an answer, not a failure.

output

# without the guards
TASK [Check whether root SSH login is already disabled] ************************
fatal: [web2.acme.internal]: FAILED! => {"changed": true, "cmd": ["grep", "-qx", "PermitRootLogin no", "/etc/ssh/sshd_config"], "msg": "The command exited with a non-zero return code.", "rc": 1, "stderr": "", "stdout": ""}

deploy.yml

    - name: Check whether root SSH login is already disabled
      ansible.builtin.command:
        cmd: grep -qx 'PermitRootLogin no' /etc/ssh/sshd_config
      register: rootlogin
      changed_when: false
      failed_when: rootlogin.rc not in [0, 1]

output

# with the guards
TASK [Check whether root SSH login is already disabled] ************************
ok: [web1.acme.internal]
ok: [web2.acme.internal]
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2.acme.internal         : ok=5    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Return code 0 means the setting is there, 1 means it is not, and anything else is a real problem worth failing on (grep uses 2 for a file it cannot read or find). This is a security control rather than tidiness. A playbook where six read-only tasks always claim changed trains everyone to skim past the recap, and then the one run where something genuinely changed on a host you did not expect looks exactly like every other run. changed=0 across the fleet has to mean something before anybody can use it as evidence.

failed_when replaces the exit-code check, it does not add to it

Write failed_when: "'BAD SIGNATURE' in verify.stdout" on a task calling /usr/local/bin/verify-release and you have thrown the return code away entirely. If that binary is missing from the host, the module fails with rc 2 (the error number Linux uses for "no such file") and an empty stdout, your condition evaluates false, Ansible overwrites the failure with your answer, and the task reports ok. A signature check that reports success because the checker is not installed is worse than no check at all. Keep the return code in the expression (failed_when: verify.rc != 0 or 'BAD SIGNATURE' in verify.stdout), and treat failed_when: false and ignore_errors: true on a destructive task as what they are: a switch that hides a half-finished deletion rather than preventing it.

Impossible, not unlikely

Every control above stops a different thing. Stack the ones that fit and a destructive play stops being something you run carefully and becomes something that cannot run against the wrong target at all. The gate goes in pre_tasks so it fires before any role gets a turn, any_errors_fatal turns one bad host into a stopped fleet run, and the conditions are ordered so that a missing variable fails cleanly instead of blowing up halfway through an expression. Note that there is no vars: block in this play: play vars outrank inventory host_vars, so pinning app_name there would quietly paper over the very mistake you are trying to catch.

deploy.yml

- name: Redeploy the shop app
  hosts: web
  become: true
  any_errors_fatal: true
 
  pre_tasks:
    - name: Refuse to run unless the target is a real app directory on a web host
      ansible.builtin.assert:
        that:
          - app_name | default('') | length > 0
          - "app_dir is match('^/srv/[a-z0-9][a-z0-9-]{1,30}$')"
          - "'web' in group_names"
        fail_msg: "app_dir resolved to '{{ app_dir | default('UNDEFINED') }}' on {{ inventory_hostname }}. Refusing to run destructive tasks."
        success_msg: "Target directory is {{ app_dir }}."
        quiet: true

output

TASK [Refuse to run unless the target is a real app directory on a web host] ***
ok: [web1.acme.internal]
ok: [web2.acme.internal]
fatal: [web3.acme.internal]: FAILED! => {"assertion": "app_name | default('') | length > 0", "evaluated_to": false, "msg": "app_dir resolved to '/srv/' on web3.acme.internal. Refusing to run destructive tasks."}
 
NO MORE HOSTS LEFT *************************************************************
 
PLAY RECAP *********************************************************************
web1.acme.internal         : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web2.acme.internal         : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web3.acme.internal         : ok=1    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0

Forty hosts, one blank variable, zero bytes deleted, and a message naming both the host and the value that caused it. The way to find out whether any of this is actually wired up is to break it on purpose in a lab: blank the variable, push the sudoers typo, point the play at a host outside the group, and confirm each one stops exactly where you expect. A guard nobody has ever watched trip is a guard nobody knows is there.

Gates a destructive change has to pass

1assert in pre_tasks

Defined, non-empty, matches the pattern. Fails the host out of the play before any role runs.

2when on the task

Right host, right state, not the live release. Skips the task instead of running it.

3creates / removes

The module tests the path on the host itself. Work already done means the command never fires, and the task reports ok with changed false.

4validate on the temp copy

visudo -cf %s or sshd -t -f %s. A file that fails is never moved into place.

5backup: true

The old file is copied aside under a dated name, keeping its owner, mode and timestamp.

6changed_when / failed_when

The recap reports what actually happened, so changed=0 becomes evidence.

Bounding the blast radiusThe control node & the SSH path

What Ansible is & the push modelAnsible What Chef is & the pull modelChef What Puppet is & the modelPuppet What Salt is & the master/minion modelSalt Ansible cheat sheetCheat sheet Ansible vs Chef vs PuppetComparison

Guarding destructive tasks

The variable that came out empty

A gate at the top of the play

Mandatory catches typos, not blanks

When as a guard, not as a shortcut

Let the module check the ground first

Check the file before it replaces the live one

Keep the file you are about to overwrite

Make command and shell tell the truth

Impossible, not unlikely

Related