Bounding the blast radius

limit, serial, forks, and fail-fast.

A Friday afternoon, a one-line edit to an nginx template, and a command typed from muscle memory: ansible-playbook site.yml. No --limit. The inventory holds four hundred hosts. Ansible does exactly what it was asked, and ninety seconds later every web server in production is serving a config with a broken upstream block. Nothing errored, so nothing stopped. The run finished green.

That is the bargain Ansible offers. One command, a thousand machines, root on all of them. The speed is the product and the speed is the hazard. Blast radius is the plain name for what sits between them: if this run goes wrong, how many hosts go with it, and how far does it get before something stops it.

There is a security version of the same question. Someone who lands on your control node, or on the CI runner (continuous integration: the build server that runs your pipelines) that holds your vault password, needs no exploit at all. They already have your playbooks and your credentials, and the fastest route to four hundred machines is your own automation, pointed wide. The keywords in this lesson will not stop a determined attacker, and the last section is honest about why. What they do is make the wide, fast run the unusual shape, so the everyday shape stays small enough for a person or an alert to catch one in progress.

Aim before you fire

Your inventory is an address book. A playbook is a letter you are about to mail-merge. Say nothing about who gets it and every address on the list receives one. The --limit flag (short form -l) is the note taped to the mailroom door: today we deliver to these streets only. It can only narrow. Ansible intersects your limit with the hosts each play already asks for, so a limit can never reach a machine the play did not already target.

The pattern language is the same one your play's hosts: line uses. Learn it properly, because most limit accidents are pattern accidents.

# every pattern here is also legal in a play's hosts: line
ansible-playbook site.yml -l 'webservers'          # one group
ansible-playbook site.yml -l 'web01,web02'         # two named hosts
ansible-playbook site.yml -l 'webservers,&edge'    # in webservers AND in edge
ansible-playbook site.yml -l 'webservers,!web07'   # webservers except web07
ansible-playbook site.yml -l 'web0*'               # shell-style wildcard
ansible-playbook site.yml -l '~web0[1-3]'          # regex, note the leading tilde
ansible-playbook site.yml -l 'webservers[0:2]'     # first three hosts of the group, by position
ansible-playbook site.yml -l @canary.txt           # host list read from a file

Two things trip newcomers here. The first is the separator. Older docs and older playbooks join patterns with a colon, as in webservers:&edge, and Ansible still accepts it. Use the comma anyway. A colon is ambiguous against IPv6 addresses (Internet Protocol version 6, the newer address format, which is stuffed with colons) and against host ranges like web[01:09], so Ansible's own pattern splitter treats the colon as a backwards-compatibility case. Quote the pattern either way, because ! and & mean something to your shell.

The second is the bracket. webservers[0:2] is not a hostname range. It is a positional slice: the first three hosts of the group, counting from zero, both ends included. A range like web[01:09] belongs in the inventory file, where it expands into nine real host names. As a pattern it will not do what you want.

Before a run that matters, rehearse the aim. The --list-hosts flag resolves the pattern and prints what would be touched, without touching anything.

ansible-playbook -i inventory/production edge-config.yml \
  --limit 'webservers,&edge' --list-hosts

playbook: edge-config.yml
 
  play #1 (webservers): Roll the edge config	TAGS: []
    pattern: ['webservers']
    hosts (3):
      web01
      web02
      web03

Now the failure that costs an afternoon. Sometimes a bad limit shouts.

# the group is "webservers"; this says "webserver"
ansible-playbook -i inventory/production edge-config.yml --limit webserver
echo "exit=$?"

[WARNING]: Could not match supplied host pattern, ignoring: webserver
ERROR! Specified inventory, host pattern and/or --limit leaves us with no hosts to target.
exit=1

That one is fine. Ansible could not match the pattern against anything in the inventory, so it refused to run, and the exit status says so. The dangerous case is a limit that matches real machines which this play does not target.

# 'db*' matches real machines, but this play targets webservers
ansible-playbook -i inventory/production edge-config.yml --limit 'db*'
echo "exit=$?"

PLAY [Roll the edge config] ****************************************************
skipping: no hosts matched
 
PLAY RECAP *********************************************************************
 
exit=0

Zero hosts. Zero tasks. Exit status 0. Your pipeline goes green, the change ticket gets closed, nothing was deployed. The same thing happens when the typo sits in the play's own hosts: line, and it is worse there, because the mistake is committed and repeats on every run.

Re-running only what broke

When a run stops partway you rarely want to start again from the top. Ansible will hand back a list of the machines it did not finish with, the way a delivery driver hands back the parcels still in the van. You have to switch the feature on first, because retry files are off by default in ansible-core.

[defaults]
inventory             = inventory/production
forks                 = 20
retry_files_enabled   = True
retry_files_save_path = ~/.ansible/retry

After a failed run that file holds one host per line, and you feed it back with the @ prefix.

cat ~/.ansible/retry/edge-config.retry

web04
web05
web06

Look at who is in that file. The host that actually failed is there, and so is every other host in the batch that got cut off, because those are the machines Ansible knows it left unfinished. Hosts it could not reach at all go in too. One sharp edge comes with the @ form: it opens the path literally, with no tilde expansion, so --limit @~/.ansible/retry/site.retry dies with Unable to find limit file. The retry_files_save_path setting in ansible.cfg does expand the tilde, because Ansible parses that setting as a path. On the command line, use "@$HOME/..." or an absolute path.

The next run overwrites the retry file, so treat it as a convenience and never as a record. The same @ syntax works on any file of host patterns, which is how you keep a canary list in version control: --limit @canary.txt, three host names, changed by pull request like anything else.

Turning a fleet-wide change into a rolling one

A kitchen putting a brand new dish on the menu does not fire two hundred plates at once. One table gets it, someone watches the room, then a section, then everybody. The serial keyword is that idea at play level. Instead of running every task against every host, Ansible takes a batch of hosts, runs the whole play against that batch, then moves to the next one.

It comes in three shapes. A plain integer is a fixed batch size. A quoted percentage is a share of the play's total hosts. A list is a ramp: one host, then five, then a quarter of the fleet, which is how you canary and widen. If the list runs out before the hosts do, the last value repeats for every remaining batch.

---
- name: Ramp probe
  hosts: fleet                # 40 hosts
  gather_facts: false
  serial: [1, 5, "25%"]
 
  tasks:
    - name: Show this batch
      ansible.builtin.debug:
        msg: "batch of {{ ansible_play_batch | length }}: {{ ansible_play_batch | join(',') }}"
      run_once: true

ansible-playbook -i inventory/fleet batch-probe.yml | grep msg

    "msg": "batch of 1: node01"
    "msg": "batch of 5: node02,node03,node04,node05,node06"
    "msg": "batch of 10: node07,node08,node09,node10,node11,node12,node13,node14,node15,node16"
    "msg": "batch of 10: node17,node18,node19,node20,node21,node22,node23,node24,node25,node26"
    "msg": "batch of 10: node27,node28,node29,node30,node31,node32,node33,node34,node35,node36"
    "msg": "batch of 4: node37,node38,node39,node40"

Forty hosts, batches of 1, 5, 10, 10, 10 and 4. The percentage is measured against the play's forty hosts, not against whatever is left over, which is why the third batch is ten and not nine. If a dashboard is going to notice a bad config, it now notices while one machine is broken instead of forty.

Serial rewrites the play, and handlers notice

Here is the part that catches experienced operators. serial does not loop over batches inside one play. It rewrites the play into one play per batch, and you can watch it happen, because the PLAY banner prints again for every batch.

TASK [Return to the load balancer pool] ****************************************
changed: [web01 -> localhost]
changed: [web02 -> localhost]
changed: [web03 -> localhost]
 
RUNNING HANDLER [reload nginx] *************************************************
changed: [web01]
changed: [web02]
changed: [web03]
 
PLAY [Roll the edge config] ****************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web04]
ok: [web05]
ok: [web06]

Follow that through and the consequence lands. A handler is a task that only runs when something else reported a change, like a reminder you act on only if the thing you were watching actually moved. Handlers flush at the end of each batch, not at the end of the play. A play that writes a config and notifies reload nginx reloads once per batch. Usually that is the point, since a rolling restart is exactly what you asked for. It stops being the point when the handler is expensive or global, like rebuilding a search index or reloading a shared load balancer (the traffic cop in front of your web servers, handing each request to one of them), because now it fires once per batch instead of once. Move that work into its own play at the end of the file.

Facts follow the same shape: each batch gathers its own. Variables set during an earlier batch stay readable through hostvars, but anything marked run_once runs again, once per batch.

Making the run stop

Batching slows the damage down. It does not stop it. By default Ansible keeps going as long as one host is still standing: a failed host drops out, everyone else carries on, the next batch starts as normal. Left that way, serial delivers your bad config to the whole fleet politely, in groups of three.

Two play keywords make it stop. max_fail_percentage is a failure budget, the way a print run tolerates a few spoiled sheets before someone stops the press: it is the share of a batch allowed to fail before Ansible abandons the play. any_errors_fatal is the blunt instrument. One host fails and the play ends for everybody, after the current task finishes across the batch.

The arithmetic on max_fail_percentage has two edges. Ansible aborts when the failure share is strictly greater than your number, never when it merely equals it, so serial: 5 with max_fail_percentage: 20 and one dead host out of five carries straight on to the next batch. And the share is measured against the current batch rather than the fleet, so with serial: 3 a single failure is already 33 percent. If you want "stop on any failure", the value is 0. One more limit: the budget only works on the linear strategy family, so a play running strategy: free gets no failure budget at all, and Ansible warns you about it rather than failing.

---
- name: Roll the edge config
  hosts: webservers            # 9 hosts: web01 .. web09
  become: true
  serial: 3
  max_fail_percentage: 0       # any failure in a batch ends the run
 
  tasks:
    - name: Drain from the load balancer
      ansible.builtin.uri:
        url: "https://lb.example.com/api/v1/pool/web/{{ inventory_hostname }}/drain"
        method: POST
        status_code: 200
      delegate_to: localhost
      become: false
      changed_when: true
 
    - name: Deploy nginx site config
      ansible.builtin.template:
        src: app.conf.j2
        dest: /etc/nginx/conf.d/app.conf
        owner: root
        group: root
        mode: "0644"
      notify: reload nginx
 
    - name: Validate the full nginx configuration
      ansible.builtin.command:
        cmd: nginx -t
      changed_when: false
 
    - name: Return to the load balancer pool
      ansible.builtin.uri:
        url: "https://lb.example.com/api/v1/pool/web/{{ inventory_hostname }}/enable"
        method: POST
        status_code: 200
      delegate_to: localhost
      become: false
      changed_when: true
 
  handlers:
    - name: reload nginx
      ansible.builtin.service:
        name: nginx
        state: reloaded

ansible-playbook -i inventory/production edge-config.yml
echo "exit=$?"

TASK [Validate the full nginx configuration] ***********************************
ok: [web04]
ok: [web06]
fatal: [web05]: FAILED! => {"changed": false, "cmd": ["nginx", "-t"], "msg": "non-zero return code", "rc": 1, "stderr": "nginx: [emerg] duplicate server name \"app.example.com\" in /etc/nginx/conf.d/legacy.conf:3\nnginx: configuration file /etc/nginx/nginx.conf test failed", "stdout": ""}
 
NO MORE HOSTS LEFT *************************************************************
 
NO MORE HOSTS LEFT *************************************************************
	to retry, use: --limit @/home/ops/.ansible/retry/edge-config.retry
 
PLAY RECAP *********************************************************************
web01                      : ok=6    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web02                      : ok=6    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web03                      : ok=6    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web04                      : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web05                      : ok=3    changed=2    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
web06                      : ok=4    changed=2    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
 
exit=2

Read that recap as a map of what you now own. web01 to web03 finished the play: config deployed, nginx reloaded, back in the pool. web05 failed on its own leftover file. web04 and web06 report failed=0, which is true and misleading, because they did not fail, they were cut off mid-play. Their lower ok= count is the giveaway. They have the new config on disk, they are still drained out of the load balancer, and their handler never fired, so nginx is still serving the old config. web07 to web09 are absent from the recap entirely, and that absence is the tell that Ansible never touched them.

The benefit and the cost sit on one screen. Six hosts affected instead of nine, three of them needing cleanup, and exit status 2 telling the pipeline it failed. Without max_fail_percentage the third batch would have gone ahead and you would be reading the same nginx error nine times.

One difference between the two keywords matters at three in the morning. A host Ansible could not connect to lands in the unreachable= column, not failed=. any_errors_fatal treats an unreachable host as a reason to stop the play. max_fail_percentage does its arithmetic only over hosts that failed a task, so a batch of machines that are all simply offline will not spend a penny of your budget. Decide what a dead host means to your rollout before an incident decides for you.

A dry run cannot prove any of this, because nothing fails in a rehearsal and the budget is never exercised. Break one staging host deliberately and run for real. If the recap stops where you predicted, and the untouched hosts are missing from it, the control is live. Check that every play in a multi-play file carries its own keywords, because serial and max_fail_percentage are per-play and do not inherit.

Forks, and one task at a time

serial decides how many hosts are in the room. forks decides how many pairs of hands are working at once. It is a control-node setting: the number of parallel worker processes Ansible spawns to talk to hosts. The default is 5, low enough that plenty of teams raise it and then forget they did.

ansible-config dump --only-changed

CONFIG_FILE() = /home/ops/infra/ansible.cfg
DEFAULT_FORKS(/home/ops/infra/ansible.cfg) = 20
DEFAULT_HOST_LIST(/home/ops/infra/ansible.cfg) = ['/home/ops/infra/inventory/production']
RETRY_FILES_ENABLED(/home/ops/infra/ansible.cfg) = True
RETRY_FILES_SAVE_PATH(/home/ops/infra/ansible.cfg) = /home/ops/.ansible/retry

The parentheses are the useful part, naming where each value came from. Ansible picks exactly one ansible.cfg, the first it finds while checking ANSIBLE_CONFIG, then the current directory, then ~/.ansible.cfg, then /etc/ansible/ansible.cfg, and it does not merge them. Someone running your playbook from their home directory can be on a completely different forks value from your pipeline and never know. There is one quiet safety net in that list: if the current directory is world-writable, Ansible refuses to load an ansible.cfg from it and warns instead, so a shared scratch directory cannot feed your run a config a stranger wrote.

The two settings stack. Inside a batch, forks caps how many hosts move at once, so serial: 3 with forks: 20 still only ever has three hosts in flight. Raising forks does not widen the blast radius by itself, but it compresses it in time, and time is what lets a person notice and press Ctrl-C. High fork counts also load the control node, roughly one process per host, and they hammer whatever every host reaches for at the same instant: your package mirror, your internal certificate authority, or an API (application programming interface, the machine-to-machine endpoint another service exposes) that meters how fast you may call it.

For that last case there is throttle, which caps workers for one task and leaves the rest of the play at full speed. If ninety hosts hitting a licensing API at once gets you rate-limited, or gets your automation's address blocked outright, throttle the single task that talks to it.

- name: Register the node with the licence service
  ansible.builtin.uri:
    url: https://licence.example.com/api/v1/register
    method: POST
    body_format: json
    body:
      fqdn: "{{ ansible_facts['fqdn'] }}"
    status_code: [200, 201]
  throttle: 2          # at most two hosts in this task at a time

The task that must happen exactly once

Some work is like unlocking the shop in the morning. It happens once, however many staff turn up afterwards. Posting to a change channel, running a database migration, flipping a feature flag: doing any of those per host is somewhere between wasteful and destructive. run_once: true runs the task on the first host of the batch and shares the result with the rest, and delegate_to says which machine actually executes it, usually localhost for anything talking to a remote service.

- name: Announce the rollout once
  ansible.builtin.uri:
    url: https://chatops.example.com/hooks/deploys
    method: POST
    body_format: json
    body:
      text: "edge config rollout: {{ ansible_play_hosts_all | length }} hosts"
  delegate_to: localhost
  become: false
  run_once: true

The trap is in the wording. run_once means once per batch, not once per play. Because serial splits the play into one play per batch, a run_once task with serial: 3 across nine hosts runs three times, and the batch probe earlier printed six times for exactly this reason. If you need it once per rollout, give it its own play at the top of the file, targeting a single host or localhost, with no serial on that play.

The rehearsal

The --check flag runs the playbook without changing anything: each module reports what it would have done. The --diff flag prints the text that would change inside files. Together they are the cheapest blast-radius control you have, because they cost one run and nothing else.

ansible-playbook -i inventory/production edge-config.yml \
  --limit web01 --check --diff

PLAY [Roll the edge config] ****************************************************
 
TASK [Gathering Facts] *********************************************************
ok: [web01]
 
TASK [Drain from the load balancer] ********************************************
skipping: [web01]
 
TASK [Deploy nginx site config] ************************************************
--- before: /etc/nginx/conf.d/app.conf
+++ after: /home/ops/.ansible/tmp/ansible-local-12175dgktu59/tmpb35025f_/app.conf.j2
@@ -1,4 +1,4 @@
 upstream app_backend {
-    server 10.4.11.20:8080;
+    server 10.4.11.20:8080 max_fails=3 fail_timeout=30s;
     keepalive 32;
 }
 
changed: [web01]
 
TASK [Validate the full nginx configuration] ***********************************
skipping: [web01]
 
TASK [Return to the load balancer pool] ****************************************
skipping: [web01]
 
RUNNING HANDLER [reload nginx] *************************************************
changed: [web01]
 
PLAY RECAP *********************************************************************
web01                      : ok=3    changed=2    unreachable=0    failed=0    skipped=3    rescued=0    ignored=0

Look hard at that recap. Three of the five tasks were skipped. Check mode is a rehearsal with half the cast missing, because any module that cannot pretend gets skipped: ansible.builtin.uri declares no check-mode support at all, and ansible.builtin.command skips itself unless you gave it creates or removes to reason about. Both load-balancer calls and the nginx -t validation never happened. The template diff is real and worth reading. The clean recap is not a promise about the real run.

Put the numbers in the file

Everything here falls into two piles. One lives in the file: serial, max_fail_percentage, any_errors_fatal, throttle, run_once. Those get argued over in a pull request, they get versioned, and they apply whether or not the operator was paying attention. The other lives on the command line: --limit, --check, --diff, --forks. Those depend on what somebody remembered to type at four on a Friday afternoon.

So push what you can into the first pile. A production play should state its own batching and failure tolerance at the top, where a reviewer meets them in the diff.

---
- name: Roll the edge config
  hosts: webservers
  become: true
 
  # Blast radius, stated here so a reviewer can argue with it.
  serial: [1, 5, "25%"]        # canary, small wave, then quarters of the play
  max_fail_percentage: 0       # any failure in a batch ends the run
  order: shuffle               # do not always canary the same machine
 
  # tasks: ...

Resist templating those numbers unless you really mean it. serial: "{{ batch_size | default(5) }}" reads like flexibility and behaves like a bypass, because anyone can pass -e batch_size=500 on the command line and your rolling deploy becomes the ninety-second incident from the top of this lesson. If environments genuinely need different batching, put the number in group_vars/production/ where it gets reviewed.

Which leaves the honest limit of all of it: keywords in a play bound accidents, not people. Anyone who can edit the play can raise serial, and anyone with a shell can drop --limit. If your fleet is big enough for that to matter, move production runs behind a job template in AWX (the upstream open-source project) or Ansible Automation Platform (Red Hat's supported build of the same thing), or behind a CI job that hard-codes its arguments, so the inventory, the limit and the forks belong to the template and the operator gets a button instead of a prompt. That is where --limit stops being something you remember and becomes something the system will not let you skip.

Related