Dynamic inventory & delegation

Inventory from the cloud; run elsewhere.

A printed guest list works right up until the guests start arriving and leaving every few minutes. That is a cloud fleet. An autoscaler (the service that adds and removes servers on its own as traffic rises and falls) starts three web servers at 09:14 and kills two of them at 09:31, and the inventory file you hand-edited on Tuesday now describes a party that has already moved on. Dynamic inventory swaps the printed list for a question asked at the door. The moment you run a playbook, Ansible calls the cloud provider's API (application programming interface, the machine-to-machine front door of a service) and builds the host list out of whatever exists right then.

Delegation is the other half of the day job. Some steps for a host cannot run on that host. Pulling a node out of a load balancer (the traffic cop that spreads incoming requests across your servers) has to happen at the load balancer. A database migration has to happen once, on the database. Delegation points one task's connection at a different machine while the play keeps walking through its original targets.

Ask The Cloud Who Is Actually There

Ansible reads inventory through plugins. The old way was an executable script that printed JSON (JavaScript Object Notation, a plain-text way of writing structured data). Those scripts still run, and you should not build anything new on one. A plugin is configured with a short YAML file instead. YAML is the indented plain-text format Ansible uses for nearly everything, and the name is an in-joke that expands to "YAML Ain't Markup Language". Treat that file as a standing order at a shop rather than a shopping list: you write the rule down once, and the answer gets fetched fresh every time.

For Amazon EC2 (Elastic Compute Cloud, Amazon's rentable virtual machines) the plugin is amazon.aws.aws_ec2. That long name is a fully qualified collection name, written namespace.collection.plugin, and it exists so two collections can both ship something called aws_ec2 without fighting over the short name. Two things have to be present on the control node, meaning the machine you run Ansible from: the collection itself, and boto3, the Python library that actually speaks to the Amazon Web Services API.

# The plugin ships inside a collection, and the collection needs boto3.
ansible-galaxy collection install amazon.aws
python3 -m pip install --user --quiet boto3 botocore
 
# On recent Debian and Ubuntu the system Python refuses --user installs
# outright. There, build a virtualenv (a private Python folder) and put
# ansible-core and boto3 inside it together.
 
# Which inventory plugins came with the collection?
ansible-doc -t inventory -l amazon.aws

Starting galaxy collection install process
Process install dependency map
Starting collection install process
Downloading https://galaxy.ansible.com/api/v3/plugin/ansible/content/published/collections/artifacts/amazon-aws-8.1.0.tar.gz to /home/deploy/.ansible/tmp/ansible-local-4471rbf1u3ug/tmp8k2p_ycx/amazon-aws-8.1.0-yq3tzs1s
Installing 'amazon.aws:8.1.0' to '/home/deploy/.ansible/collections/ansible_collections/amazon/aws'
amazon.aws:8.1.0 was installed successfully
amazon.aws.aws_ec2 EC2 inventory source
amazon.aws.aws_rds RDS instance inventory source

One rule catches everyone. The amazon.aws.aws_ec2 plugin only claims files whose names end in aws_ec2.yml or aws_ec2.yaml. So prod.aws_ec2.yml is fine and aws-prod.yml is not. That suffix list is hard-coded in the plugin, so it is not a style preference you can argue with.

There is a catch that hides the mistake from you. The built-in auto plugin is enabled by default, and it will happily load any YAML file that declares a plugin: key, whatever the file is called. So a badly named source often works anyway and you never learn. Then somebody tightens enable_plugins and drops auto, and the file stops resolving: you get [WARNING]: Unable to parse ... as an inventory source and a run with no hosts in it. The other cloud plugins guard their names the same way (azure_rm.yml for azure.azcollection.azure_rm, gcp_compute.yml for google.cloud.gcp_compute), so treat the suffix as part of the plugin's name.

plugin: amazon.aws.aws_ec2
 
regions:
  - eu-west-1
  - us-east-1
 
# Narrowing happens server-side. The EC2 API returns only what matches,
# so hosts outside production never enter the run at all.
filters:
  tag:Environment: production
  instance-state-name: running
 
# Tag values become group names. Role=web builds the group role_web.
# No replace() needed: an inventory plugin rewrites characters that are
# illegal in a group name (the hyphens in eu-west-1a) all by itself.
keyed_groups:
  - key: tags.Role | default('untagged') | lower
    prefix: role
  - key: placement.availability_zone
    prefix: az
 
# A group built from a condition, for what a tag cannot say.
groups:
  public_facing: public_ip_address is defined
 
# What each host is called. First option that produces a value wins.
hostnames:
  - tag:Name
  - private-ip-address
 
# Extra host variables, built from the API response.
compose:
  ansible_host: private_ip_address
  patch_group: tags.PatchGroup | default('unassigned')
 
# Break loudly if an expression above fails, instead of skipping it.
strict: true

Read that file as four separate decisions. filters are sent to the EC2 API itself, so a staging instance sitting in the same region is never downloaded, never grouped, and never reachable by a typo in --limit. One source file per environment beats one file plus a careful operator. keyed_groups turns tag values into group names, and that is what lets a playbook say hosts: role_web and actually mean it.

hostnames decides what each machine is called in output and in --limit, and the order matters more than it looks. If your autoscaling group stamps the same Name tag on every instance, all three machines collapse into one inventory entry, the last one's variables win, and the other two are never configured by anything. Put instance-id or private-ip-address first when names are not unique. compose builds extra host variables out of the API response using Jinja2 (Ansible's templating language) written without the usual curly braces, and strict: true turns a broken expression into a hard error rather than a missing variable you find out about at 3am.

Prove The Inventory Before A Playbook Touches It

# Nothing here connects to a managed host. It only asks AWS.
ansible-inventory -i inventory/prod.aws_ec2.yml --graph

@all:
  |--@ungrouped:
  |--@aws_ec2:
  |  |--web-prod-01
  |  |--web-prod-02
  |  |--web-prod-03
  |  |--db-prod-01
  |  |--10.0.2.99
  |--@public_facing:
  |  |--web-prod-01
  |  |--web-prod-02
  |  |--web-prod-03
  |--@role_web:
  |  |--web-prod-01
  |  |--web-prod-02
  |  |--web-prod-03
  |--@az_eu_west_1a:
  |  |--web-prod-01
  |  |--db-prod-01
  |--@az_eu_west_1b:
  |  |--web-prod-02
  |  |--web-prod-03
  |  |--10.0.2.99
  |--@role_db:
  |  |--db-prod-01
  |--@role_untagged:
  |  |--10.0.2.99

That was one call to the EC2 API and zero SSH (Secure Shell, the encrypted remote login protocol) sessions, which is why it is safe to run against production while you are still working out what your filters do. Everything the plugin found also lands in a group named after the plugin, aws_ec2, on top of whatever your own rules built. Add --vars to print each host's variables underneath it, or --yaml if JSON makes your eyes slide off the page.

Two lines there are findings, not decoration. role_untagged exists because one production instance carries no Role tag, so default('untagged') caught it. That box is running in production and no playbook is configuring it. It shows up as 10.0.2.99 rather than a name because it has no Name tag either, so hostnames fell through to the private IP address. Go and find out what it is.

Two things about that tree surprise people. The groups are not alphabetical: ansible-inventory prints them in the order they were created, which is why ungrouped sits at the top (it is made first, and it is empty here because every host landed in aws_ec2). And the availability zone eu-west-1a, one data centre inside the eu-west-1 region, became the group az_eu_west_1a without you asking. Group names built by an inventory plugin are sanitized automatically and silently: illegal characters become underscores every single time, whatever force_valid_group_names is set to. That setting governs group names read from a static file instead. Write [web-servers] in an INI inventory and you get [WARNING]: Invalid characters were found in group names but not replaced, use -vvvv to see details, the group keeps its hyphen, and since a hyphen is not legal in a Jinja2 name, groups.web-servers blows up while groups['web-servers'] still resolves. Set force_valid_group_names = always and it is renamed to web_servers, with a warning saying so.

# What variables did the plugin actually hand one host?
# jq is a small command-line tool for pulling fields out of JSON.
ansible-inventory -i inventory/prod.aws_ec2.yml --host web-prod-01 \
  | jq '{ansible_host, patch_group, instance_type, state: .state.name, role: .tags.Role}'

{
  "ansible_host": "10.0.2.31",
  "patch_group": "weekly",
  "instance_type": "t3.medium",
  "state": "running",
  "role": "web"
}

ansible_host is the one to check every single time. It is the address Ansible dials, and compose is what set it to the private IP. Drop that line, or let the expression fail quietly under strict: false, and Ansible falls back to resolving the inventory hostname through DNS (Domain Name System, the internet's name-to-address phone book). Now you are trusting whatever your resolver thinks web-prod-01 means, which could be a decommissioned box, a machine in a different account, or nothing at all. Connecting over the private address keeps the traffic inside the network you control and keeps the public interface out of the path.

An Empty Inventory Is A Green Build

Here is the failure that gets people fired rather than paged. Somebody renames the Environment tag to env in Terraform, the tool that creates the instances in the first place. Nothing errors. The EC2 API cheerfully reports that zero instances match tag:Environment: production, and Ansible treats zero hosts as a perfectly good run. Every night the hardening playbook exits 0, the pipeline goes green, and not one machine is being hardened. The fix is to make emptiness fail on purpose. Put a bouncer on the door: a first play against localhost that counts heads and refuses to let the rest of the run start when the number looks wrong.

# The tag was renamed in Terraform and nobody told the inventory.
ansible-playbook -i inventory/prod.aws_ec2.yml harden.yml
echo "exit code: $?"

[WARNING]: provided hosts list is empty, only localhost is available. Note that
the implicit localhost does not match 'all'
 
PLAY [Check the inventory before we touch anything] ****************************
 
TASK [The cloud must return the fleet we expect] *******************************
fatal: [localhost]: FAILED! => {
    "assertion": "groups['role_web'] | default([]) | length >= 2",
    "changed": false,
    "evaluated_to": false,
    "msg": "Inventory returned 0 hosts in role_web, expected at least 2. Refusing to continue."
}
 
PLAY RECAP *********************************************************************
localhost                  : ok=0    changed=0    unreachable=0    failed=1    skipped=0    rescued=0    ignored=0
 
exit code: 2

Credentials, Cache And Who Can Add A Host

[defaults]
inventory = inventory/prod.aws_ec2.yml
force_valid_group_names = always   # only affects names read from static files
host_key_checking = True
 
[inventory]
# This list REPLACES the default (host_list, script, auto, yaml, ini, toml),
# so whatever you leave out stops working. 'auto' reads a source file's
# plugin: key; naming aws_ec2 as well documents the dependency.
enable_plugins = amazon.aws.aws_ec2, auto, yaml, ini, host_list
 
# A source that fails to parse is a failure, not a warning you scroll past.
any_unparsed_is_failed = True
 
# One API call per run adds up and eventually gets rate-limited. Cache the
# answer, and keep it somewhere only this account can read.
cache = True
cache_plugin = jsonfile
cache_connection = /var/lib/ansible/inventory-cache
cache_timeout = 900

Caching is the difference between one API call per run and a pipeline that trips Amazon's rate limits at the worst possible moment. It is a photograph of the shelf rather than a look at the shelf. The trade-off is honest, and it only runs one way: for up to fifteen minutes, a machine created two minutes ago does not exist as far as Ansible is concerned. Put ANSIBLE_INVENTORY_CACHE=False in front of the command when you need the live answer, or delete the cache file. And notice where the cache is not: /tmp. That file is a plaintext map of your estate, every private IP, every tag, every AMI id (Amazon Machine Image, the disk template an instance booted from). Sitting in a world-readable directory on a shared host, it is a free reconnaissance report for any local account.

The credentials that build inventory need exactly one IAM permission (Identity and Access Management is the AWS system that decides who is allowed to call what): ec2:DescribeInstances. Add ec2:DescribeRegions if you leave regions out and let the plugin discover them. Nothing else. Read-only keys cannot stop, start, or retag anything, so a control node that gets compromised hands the attacker a map instead of a lever.

Now the sharper edge, which runs in the other direction. Your groups are built from tags, and tags are name badges that anyone holding ec2:CreateTags can print. Someone who can stamp Environment=production and Role=web onto an instance they control has added their machine to your role_web group. On the next run your playbook connects to it and installs whatever the web role installs, which usually means a TLS (Transport Layer Security, the encryption behind HTTPS) private key, a database password, and an API token. Tag-based grouping quietly turns ec2:CreateTags into a read of your application secrets. Scope that permission tightly.

Host key checking is the last speed bump standing, because a host nobody has seen before fails its very first connection. So host_key_checking = False does not buy you convenience here. It removes your final control. It does cost you something real, mind: legitimate new instances fail that first connection too, which is why people reach for StrictHostKeyChecking=accept-new or bake host keys in at image build time instead of switching the check off wholesale.

Aim One Task At A Different Machine

When a supermarket takes one till out of service, the sign goes on the till but the queue gets redirected by the person at the door. The work is about the till. The action happens at the door. delegate_to is that split: it runs one task on a different host while the play carries on iterating over its real targets. local_action is older shorthand for delegate_to: localhost. run_once: true fires a task once for the whole batch instead of once per host, which is what a schema migration wants. Add serial: 1 to the play header and you have a rolling change: drain one node at the balancer, change it, reload, put it back, move to the next.

---
- name: Check the inventory before we touch anything
  hosts: localhost
  gather_facts: false
  tasks:
    - name: The cloud must return the fleet we expect
      ansible.builtin.assert:
        that:
          - groups['role_web'] | default([]) | length >= 2
        fail_msg: >-
          Inventory returned {{ groups['role_web'] | default([]) | length }}
          hosts in role_web, expected at least 2. Refusing to continue.
 
- name: Roll the SSH hardening out one node at a time
  hosts: role_web
  serial: 1                 # one host per batch
  max_fail_percentage: 0    # one failure stops the whole rollout
  become: true
 
  tasks:
    - name: Take this node out of the load balancer
      ansible.builtin.uri:
        url: "https://lb.internal/v1/pools/web/members/{{ inventory_hostname }}/drain"
        method: POST
        headers:
          Authorization: "Bearer {{ lb_api_token }}"
        status_code: [200, 204]
      delegate_to: localhost
      become: false         # sudo on the control node is not wanted here
      changed_when: false   # draining is not a configuration change
      no_log: true          # the token would otherwise land in the log
 
    - name: Apply the hardened sshd config
      ansible.builtin.template:
        src: sshd_config.j2
        dest: /etc/ssh/sshd_config
        owner: root
        group: root
        mode: "0600"
        validate: /usr/sbin/sshd -t -f %s   # never install a config that locks you out
      notify: Reload sshd
 
    - name: Run the schema change exactly once for the whole fleet
      ansible.builtin.command:
        cmd: /opt/app/migrate.sh --step 42
        creates: /var/lib/app/.migrated-42
      run_once: true
      delegate_to: "{{ groups['role_db'] | first }}"
 
    - name: Record the deploy on the control node
      ansible.builtin.lineinfile:
        path: "{{ playbook_dir }}/deploys.log"
        line: "{{ inventory_hostname }} hardened at {{ ansible_date_time.iso8601 }}"
        create: true
      delegate_to: localhost
      become: false
 
  handlers:
    - name: Reload sshd
      ansible.builtin.service:
        name: sshd
        state: reloaded
 
  post_tasks:
    - name: Put the node back in the pool
      ansible.builtin.uri:
        url: "https://lb.internal/v1/pools/web/members/{{ inventory_hostname }}/enable"
        method: POST
        headers:
          Authorization: "Bearer {{ lb_api_token }}"
        status_code: [200, 204]
      delegate_to: localhost
      become: false
      changed_when: false
      no_log: true

Four details in that file are doing more work than they look. become: false on the delegated tasks matters because become: true sits at the play level, and without the override Ansible tries to sudo (switch to the root account) on your control node in order to make an outbound web call. no_log: true keeps the bearer token out of the log, and the cost of that is real: when the task fails, all you get is "censored": "the output has been hidden due to the fact that 'no_log: true' was specified for this result". Turn it off deliberately while you debug, turn it back on before you merge.

validate: runs sshd -t against the rendered file before it is moved into place, which is the whole difference between a config typo and losing SSH to an entire tier. And handlers flush at the end of the task section, before post_tasks run, so sshd reloads while the node is still drained and only then goes back into the pool. If the reload fails, the node stays out of the pool and max_fail_percentage: 0 stops the rollout at one broken machine instead of three.

# Roll one node by hand first. The limit applies to every play in the
# file, so localhost has to be named or the guard play is skipped.
ansible-playbook -i inventory/prod.aws_ec2.yml harden.yml \
  --limit 'localhost,web-prod-01'

PLAY [Check the inventory before we touch anything] ****************************
 
TASK [The cloud must return the fleet we expect] *******************************
ok: [localhost] => {
    "changed": false,
    "msg": "All assertions passed"
}
 
PLAY [Roll the SSH hardening out one node at a time] ***************************
 
TASK [Gathering Facts] *********************************************************
ok: [web-prod-01]
 
TASK [Take this node out of the load balancer] *********************************
ok: [web-prod-01 -> localhost]
 
TASK [Apply the hardened sshd config] ******************************************
changed: [web-prod-01]
 
TASK [Run the schema change exactly once for the whole fleet] ******************
changed: [web-prod-01 -> db-prod-01]
 
TASK [Record the deploy on the control node] ***********************************
changed: [web-prod-01 -> localhost]
 
RUNNING HANDLER [Reload sshd] **************************************************
changed: [web-prod-01]
 
TASK [Put the node back in the pool] *******************************************
ok: [web-prod-01 -> localhost]
 
PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web-prod-01                : ok=7    changed=4    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

The arrows are your proof. ok: [web-prod-01 -> localhost] says the play was working on web-prod-01 while the task itself ran on your control node. changed: [web-prod-01 -> db-prod-01] says the migration ran on the database while the play was still holding a web server. If you expect an arrow and there is none, the task ran on the target, and you should stop and reread the file. Short of switching on -vvv and reading raw connection lines, this is where you confirm that delegation went where you meant it to go.

# Now the rest of the tier, one node at a time
ansible-playbook -i inventory/prod.aws_ec2.yml harden.yml | tail -n 8

TASK [Put the node back in the pool] *******************************************
ok: [web-prod-03 -> localhost]
 
PLAY RECAP *********************************************************************
localhost                  : ok=1    changed=0    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web-prod-01                : ok=6    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web-prod-02                : ok=7    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0
web-prod-03                : ok=7    changed=3    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0

Three things in that recap deserve a slow read. web-prod-01 shows changed=1 where the others show changed=3, because the canary run already hardened it. The single change left is the deploy log line, which carries a timestamp and therefore appends on every run. Everything that describes the machine's state is idempotent (running it twice changes nothing the second time), and the audit line deliberately is not.

Next, go looking for the migration and you will not be able to pick it out. run_once means once per batch, and serial: 1 makes every host its own batch, so the task was attempted three separate times. The creates: argument is the only reason the schema change did not actually run three times. Here is the part worth memorising: a creates that is already satisfied reports a plain ok, not a skipped. Nothing in PLAY RECAP tells you the command was a no-op, which is exactly why these counts read as though every task ran normally.

Last, a word about --check, the dry run. Modules that cannot predict their own effect get skipped in check mode, and uri and command are both in that group: uri does not support check mode at all, and command declines to run. So a dry run of this playbook rehearses the sshd template and the log line while skipping the drain and the migration entirely. That is a rehearsal of part of the change. Know it before you tell anyone the dry run came back clean.

Give yourself a way to notice the fleet changing shape when nobody is deploying. --graph output is stable text, so a nightly snapshot and a diff turns "a new host appeared in role_web" into a line somebody has to explain.

# Nightly: snapshot the fleet and compare it with yesterday
ansible-inventory -i inventory/prod.aws_ec2.yml --graph > inventory.today
diff inventory.yesterday inventory.today

8a9
>   |  |--web-prod-04
12a14
>   |  |--web-prod-04
16a19
>   |  |--web-prod-04
23a27
>   |  |--web-prod-04

Four added lines, one host. web-prod-04 joined aws_ec2, public_facing, role_web and an availability zone group overnight. If no change request explains it, do not run the playbook, because the next run hands that machine your TLS key. Go and read who called ec2:CreateTags.

Try this

Run ansible-galaxy collection install amazon.aws on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: zero hosts is a success, not an error. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related