Handlers, loops & conditionals

React to change; repeat; branch.

A note taped to the fridge says: if you touched the thermostat today, restart the furnace before bed. Two rules hide inside that note. The follow-up happens at the end of the day, not the second you touch the dial. And it happens only if you actually touched something. Fiddle with the thermostat five times and you still restart the furnace once. Fiddle with nothing and you skip it.

Ansible calls that note a handler. It exists because of the single most common rule in configuration management (the practice of writing down a machine's desired state and letting a tool enforce it): reload the service if, and only if, its config changed. Loops and conditionals round out the set. A loop repeats one task over a list. A conditional decides, per host, whether a task runs at all. Together the three turn a flat list of commands into a play that reacts to what it finds. Everything below targets ansible-core 2.16 and later and uses fully qualified collection names (FQCN, the full namespace.collection.module spelling like ansible.builtin.copy), because the bare short names are legacy. For security work that reaction is the whole point. A hardening playbook that bounces sshd (the SSH server daemon, the process that accepts your remote logins) on every single run is a playbook nobody will let near production.

Handlers React to Change, Not to Running

A handler is an ordinary task that lives under a handlers: key instead of tasks:. It sits there doing nothing until some task notifies it by name. The trigger is one word: changed. That is Ansible's way of reporting that a task genuinely altered the machine. The template rewrote the file. The package really was installed. The account did not exist and now does. If a task finds the machine already in the state you asked for, it reports ok, no notification is queued, and the handler stays quiet. That quietness is what makes a run idempotent (running it twice changes nothing the second time) all the way down to service restarts.

Notifications get de-duplicated too. Notify Restart sshd from twelve different tasks and it still runs exactly once. Queued handlers then wait until the end of the play, after the last task has finished on the last host, and run together. Strictly speaking Ansible flushes the queue at each section boundary, so after pre_tasks, after roles and tasks, and after post_tasks. The batching is deliberate. Rewrite six config fragments and you pay for one restart instead of six.

---
- name: Harden SSH on the web tier
  hosts: web
  become: true
 
  tasks:
    - name: Install the SSH hardening drop-in
      ansible.builtin.template:
        src: 10-hardening.conf.j2
        dest: /etc/ssh/sshd_config.d/10-hardening.conf
        owner: root
        group: root
        mode: "0600"
        validate: /usr/sbin/sshd -t -f %s
      notify: Restart sshd
 
  handlers:
    - name: Restart sshd
      ansible.builtin.service:
        name: sshd
        state: restarted

# {{ ansible_managed }}
# Managed by the platform team. Local edits are overwritten.
 
PermitRootLogin no
PasswordAuthentication no
KbdInteractiveAuthentication no
X11Forwarding no
MaxAuthTries {{ ssh_max_auth_tries | default(3) }}
AllowGroups {{ ssh_allow_groups | join(' ') }}

One line in that playbook is load-bearing far beyond the file it writes. Think of validate as the taste test before you serve the dish. It runs a command against the temporary copy Ansible stages on the host, before that copy is moved into place, with %s swapped for the temp path. If /usr/sbin/sshd -t rejects the content, the task fails, the real file is never touched, the handler is never notified, and the daemon is never restarted with a config that would kill it. Skip validate, add one typo in a Jinja2 template (Jinja2 is the templating language Ansible uses to fill values into files), keep an eager restart handler, and that is exactly how you lock yourself out of a whole fleet at once. Existing SSH (Secure Shell) sessions survive a restart, so nothing looks broken until the next person tries to log in.

Two details about the destination matter as much as the content. OpenSSH reads /etc/ssh/sshd_config.d/*.conf in filename order and keeps the first value it sees for most keywords, so the 10- prefix deliberately puts your hardening ahead of vendor files like 50-cloud-init.conf that happily re-enable password logins. Second, state: restarted is heavier than sshd needs. state: reloaded sends the daemon a SIGHUP (a hangup signal, the traditional nudge that means re-read your config), which picks up the new settings without dropping the listening socket. ansible.builtin.service is the generic wrapper that works across init systems; reach for ansible.builtin.systemd_service (renamed from ansible.builtin.systemd in ansible-core 2.15) when you need systemd-specific options like daemon_reload.

$ ansible-playbook -i inventory.ini harden-ssh.yml

PLAY [Harden SSH on the web tier] **********************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
ok: [web02]
 
TASK [Install the SSH hardening drop-in] ***************************************
changed: [web02]
changed: [web01]
 
RUNNING HANDLER [Restart sshd] *************************************************
changed: [web01]
changed: [web02]
 
PLAY RECAP *********************************************************************
web01                      : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web02                      : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The template task reported changed, so a RUNNING HANDLER section appeared after every normal task had finished. Now run the identical playbook a second time. What matters is what goes missing.

$ ansible-playbook -i inventory.ini harden-ssh.yml

PLAY [Harden SSH on the web tier] **********************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
ok: [web02]
 
TASK [Install the SSH hardening drop-in] ***************************************
ok: [web01]
ok: [web02]
 
PLAY RECAP *********************************************************************
web01                      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web02                      : ok=2    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

No handler section at all. The file already matched, the task reported ok, nothing was queued, and nobody's session got interrupted. That is the difference between a playbook a pipeline can run every hour and one you run by hand while holding your breath. It also hands you a free drift detector. On a playbook that has already converged, changed=0 across the fleet means nobody has hand-edited those files since the last run. Any host that suddenly reports changed=1 is telling you somebody did.

One notification can fan out to several handlers through listen, which subscribes a handler to a topic name rather than matching its own name. It behaves like a mailing list: you write to the list address, and everyone subscribed gets a copy. Every handler listening on that topic fires. This is how a single 'security config changed' notify reaches each daemon that reads the file you just rewrote.

- name: Restart sshd
  ansible.builtin.service:
    name: sshd
    state: restarted
  listen: security config changed
 
- name: Restart rsyslog
  ansible.builtin.service:
    name: rsyslog
    state: restarted
  listen: security config changed
 
# A task now reaches both of them with one line:
#   notify: security config changed

Handler names are free text, matched exactly, whitespace and capitalisation included, so typos are easy to make. The failure is at least loud. Notify a name that nothing answers to and the run stops with ERROR! The requested handler 'Restart sshd' was not found in either the main handlers list nor in the listening handlers list. You can downgrade that to a warning by setting error_on_missing_handler = False under [defaults] in ansible.cfg. Occasionally handy for optional roles, and otherwise a fine way to never notice a broken notify again.

Loops Repeat One Task Over a List

A loop is a shopping list handed to one shopper. The loop: keyword takes a list and runs its task once per element, exposing the current element as the variable item. Three service accounts, twenty firewall rules, a dozen sudoers fragments (the files under /etc/sudoers.d that grant specific commands to specific users): one task definition instead of copy-paste, and one place to edit when the rule changes. Elements can be plain strings or dictionaries.

---
- name: Baseline accounts
  hosts: web
  become: true
 
  tasks:
    - name: Create service accounts
      ansible.builtin.user:
        name: "{{ item.name }}"
        groups: "{{ item.groups }}"
        shell: "{{ item.shell | default('/usr/sbin/nologin') }}"
        append: true
        state: present
      loop:
        - { name: deploy,  groups: sudo, shell: /bin/bash }
        - { name: metrics, groups: adm }
        - { name: backup,  groups: backup }
      loop_control:
        label: "{{ item.name }}"

$ ansible-playbook -i inventory.ini accounts.yml --limit web01

PLAY [Baseline accounts] *******************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
 
TASK [Create service accounts] *************************************************
changed: [web01] => (item=deploy)
changed: [web01] => (item=metrics)
changed: [web01] => (item=backup)
 
PLAY RECAP *********************************************************************
web01                      : ok=2    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The loop_control: label line is doing real work in that output. Without it Ansible prints the whole dictionary on every iteration, so the first line reads changed: [web01] => (item={'name': 'deploy', 'groups': 'sudo', 'shell': '/bin/bash'}). Multiply that by fifty accounts and the CI (continuous integration) log turns to soup. The sharper problem is disclosure. If any element carries an API token or a password, the default label writes it straight into a build log that everyone with pipeline access can read. Set label to something harmless, and add no_log: true to the task itself when the list holds secrets, which suppresses that task's arguments and results across every callback and log line. The price is that a failure there tells you almost nothing, so flip it off deliberately while debugging and put it back.

Registering a looped task changes the shape of what comes back, and this catches everyone exactly once. A normal task registers a dictionary with keys like rc (return code, the exit status of a command, where 0 means success) and stdout. A looped task registers a dictionary whose useful content sits under results, a list holding one entry per iteration. Each entry carries the item that produced it alongside that iteration's own rc, stdout or stat.

---
- name: Audit leftover SSH keys
  hosts: web
  become: true
 
  tasks:
    - name: Check for a leftover authorized_keys file
      ansible.builtin.stat:
        path: "/home/{{ item }}/.ssh/authorized_keys"
      register: key_files
      loop: [deploy, metrics, backup]
 
    - name: List accounts that still hold a key
      ansible.builtin.debug:
        msg: "{{ key_files.results | selectattr('stat.exists') | map(attribute='item') | list }}"

$ ansible-playbook -i inventory.ini audit-keys.yml --limit web01

PLAY [Audit leftover SSH keys] *************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
 
TASK [Check for a leftover authorized_keys file] *******************************
ok: [web01] => (item=deploy)
ok: [web01] => (item=metrics)
ok: [web01] => (item=backup)
 
TASK [List accounts that still hold a key] *************************************
ok: [web01] => {
    "msg": [
        "deploy"
    ]
}
 
PLAY RECAP *********************************************************************
web01                      : ok=3    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Filtering the results list with selectattr beats looping over it with a conditional, because you get one readable line instead of one console line per account. A different flavour of repetition is waiting for something to become true, the way you keep wandering back to check whether the kettle has boiled. until retries a task until its condition holds, with retries attempts and delay seconds between them (3 and 5 respectively if you leave them out). That is how you confirm the daemon your handler restarted actually came back.

---
- name: Verify sshd came back
  hosts: web
  become: true
 
  tasks:
    - name: Confirm sshd is back up
      ansible.builtin.command: systemctl is-active sshd
      register: sshd_state
      until: sshd_state.stdout == "active"
      retries: 5
      delay: 3
      changed_when: false

$ ansible-playbook -i inventory.ini verify.yml --limit web01

TASK [Confirm sshd is back up] *************************************************
FAILED - RETRYING: [web01]: Confirm sshd is back up (5 retries left).
FAILED - RETRYING: [web01]: Confirm sshd is back up (4 retries left).
ok: [web01]

Unit names are not universal, and a health check built on the wrong one lies to you. Red Hat family hosts call the unit sshd. Debian and Ubuntu call it ssh and only alias sshd on recent releases. Recent Ubuntu also puts SSH behind socket activation (ssh.socket), where the service unit honestly reports inactive until a connection arrives, which would send the loop above through all five retries and then fail on a perfectly healthy box. Check the unit name on the distribution you actually run before you trust a check like this.

changed_when: false on that task is not tidiness. The ansible.builtin.command and ansible.builtin.shell modules have no idea whether what they ran altered anything, so they report changed every single time. Leave that alone in a playbook that owns handlers and every run notifies every handler, restarting daemons on hosts where nothing was touched, which trains the whole team to ignore the changed column. Pin your read-only checks to changed_when: false and the number means something again. Its sibling failed_when does the same job for exit codes that are not really failures, like a grep that found nothing.

One loop habit worth unlearning: do not loop a package module. ansible.builtin.apt, ansible.builtin.dnf and ansible.builtin.package all accept a list in name: and resolve it as a single transaction, which is faster and lets the package manager settle dependencies in one pass. Looping apt over twenty packages runs twenty separate apt transactions and can add minutes per host. Reach for loop when the module genuinely handles one thing at a time, like user or lineinfile. You will still meet with_items in older code; loop is the current form and the one to write.

Conditionals Decide Per Host

when: is the bouncer at the door. It decides whether a task runs at all for a given host, and it is evaluated separately for every host in the play, which is why one playbook can do the right thing across a mixed fleet. The expression is raw Jinja2, so it takes no curly braces. Write when: {{ x }} and Ansible tells you off: [WARNING]: conditional statements should not include jinja2 templating delimiters such as {{ }} or {% %}. Conditions usually read a fact (a piece of information Ansible gathers about the host before the first task runs, like its operating system family or how much memory it has) or a value you captured earlier with register.

---
- name: Baseline audit tooling
  hosts: all
  become: true
 
  # group_vars/all.yml holds:
  #   stale_accounts:
  #     - { name: jdoe,    days_idle: 141 }
  #     - { name: metrics, days_idle: 2 }
 
  tasks:
    - name: Install the audit daemon (Debian family)
      ansible.builtin.apt:
        name:
          - auditd
          - audispd-plugins
        state: present
        update_cache: true
      when: ansible_facts['os_family'] == "Debian"
 
    - name: Look for a pending reboot
      ansible.builtin.stat:
        path: /var/run/reboot-required
      register: reboot_flag
 
    - name: Reboot only where the kernel asked for it
      ansible.builtin.reboot:
        reboot_timeout: 600
      when:
        - reboot_flag.stat.exists
        - allow_reboots | default(false) | bool
 
    - name: Lock accounts idle for more than 90 days
      ansible.builtin.user:
        name: "{{ item.name }}"
        password_lock: true
      loop: "{{ stale_accounts }}"
      when: item.days_idle > 90
      loop_control:
        label: "{{ item.name }}"

$ ansible-playbook -i inventory.ini baseline.yml

PLAY [Baseline audit tooling] **************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
ok: [rhel01]
 
TASK [Install the audit daemon (Debian family)] ********************************
skipping: [rhel01]
changed: [web01]
 
TASK [Look for a pending reboot] ***********************************************
ok: [web01]
ok: [rhel01]
 
TASK [Reboot only where the kernel asked for it] *******************************
skipping: [rhel01]
skipping: [web01]
 
TASK [Lock accounts idle for more than 90 days] ********************************
changed: [web01] => (item=jdoe)
skipping: [web01] => (item=metrics)
changed: [rhel01] => (item=jdoe)
skipping: [rhel01] => (item=metrics)
 
PLAY RECAP *********************************************************************
web01                      : ok=4    changed=2    unreachable=0    failed=0    skipped=1    rescued=0    ignored=0
rhel01                     : ok=3    changed=1    unreachable=0    failed=0    skipped=2    rescued=0    ignored=0

Three things to read out of that run. Handing when a YAML list ANDs the conditions together, which beats a chain of and keywords and lets you comment each line on its own; the reboot task needed both a kernel flag on disk and an explicit opt-in variable, and neither host had both. Skipped is a distinct outcome from ok and failed and gets its own column in the recap, so you can tell 'this host did not need it' apart from 'this host refused'. And because when is re-evaluated for every element, pairing it with loop filters the list as you iterate instead of forcing you to pre-build a trimmed one.

There is a trap hiding in that same play. /var/run/reboot-required is a Debian and Ubuntu convention, written by update-notifier. Red Hat family hosts have no such file, so rhel01 will report exists: false forever and never reboot, no matter how far behind its kernel falls. On that side of the fence you want dnf needs-restarting -r, which exits 1 when a reboot is due and 0 when it is not. A conditional is only ever as honest as the fact it reads.

Two more tests earn their keep in day-to-day security plays. is defined and is not defined guard against a missing variable blowing up the whole expression, as in when: vault_token is defined. And in checks membership, so when: inventory_hostname in groups['prod'] lets one task behave differently on production without a second playbook.

Prove the Change Actually Landed

Two flags turn a play into something you can inspect before it touches anything. --check runs it without making changes and reports what would happen. --diff prints the line-by-line difference for file, template, copy and lineinfile tasks. Together they are the closest thing Ansible has to a plan step, and pointing them at a single host with --limit before releasing the play on four hundred machines costs about ten seconds. Check mode does have a hole worth knowing about: command and shell tasks are skipped rather than simulated, so anything downstream that reads their registered output will misbehave in a check run. Mark the genuinely read-only ones with check_mode: false to make them run anyway.

$ ansible-playbook -i inventory.ini harden-ssh.yml --limit web01 --check --diff

PLAY [Harden SSH on the web tier] **********************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
 
TASK [Install the SSH hardening drop-in] ***************************************
--- before: /etc/ssh/sshd_config.d/10-hardening.conf
+++ after: /home/ops/.ansible/tmp/ansible-local-4812jx0k1w_a/tmpv3n8q1r7/10-hardening.conf.j2
@@ -2,7 +2,7 @@
 # Managed by the platform team. Local edits are overwritten.
 
 PermitRootLogin no
-PasswordAuthentication yes
+PasswordAuthentication no
 KbdInteractiveAuthentication no
 X11Forwarding no
 MaxAuthTries 3
 
changed: [web01]
 
RUNNING HANDLER [Restart sshd] *************************************************
changed: [web01]
 
PLAY RECAP *********************************************************************
web01                      : ok=3    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The handler line still shows up because the simulated change still fired the notification, and the handler is running in check mode too, so nothing was actually restarted. What you are reading is a plan: this file would gain PasswordAuthentication no, and sshd would be restarted once. An empty diff with changed=0 in the recap means the host already matches and there is nothing to apply. A diff that surprises you means somebody edited that file by hand, and you have found the drift on your own terms rather than during an incident review.

Try this

Run ansible-playbook -i inventory.ini harden-ssh.yml on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: handlers run in defined order, and a failed play skips them. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related