Roles and collections supply chain
Galaxy content is third-party code.
A contractor turns up at your building with a clipboard and a van that has the right logo on the door. You hand over the master key, because the name looked familiar. Nobody phoned the company. Nobody looked inside the van. That is the deal your pipeline strikes every time an unpinned ansible-galaxy install runs: code written by people you have never met, fetched from the internet by name, then executed on every server you manage, usually with the root privileges you spent a whole lesson scoping down.
Two words get mixed up constantly, so here is the difference, with a kitchen for scaffolding. A **role** is a recipe card, a reusable folder of Ansible content holding tasks, handlers, templates and default variables in a fixed layout. A **collection** is the boxed meal kit built around that idea, a namespaced bundle such as community.general that can carry roles, modules, plugins and playbooks together. **Ansible Galaxy** (galaxy.ansible.com) is the public shop both are sold from, and ansible-galaxy is the client on your machine that does the shopping. Write community.general.ufw in a task and you have declared a dependency on a stranger's Python.
What the Attacker Gets for Free
The attack arrives in three familiar shapes. Someone steals a maintainer's account and publishes a poisoned release of a collection you already trust. A maintainer sells out and does the same thing legitimately. Or somebody registers a namespace one character away from the one you meant, then waits for a typo. npm (the package registry for JavaScript) and PyPI (the Python Package Index) have been hit by all three, more than once, at scale.
Ansible makes the consequences worse than usual. A module is Python that the controller copies onto the target machine and runs there, and under become (Ansible's privilege escalation, normally sudo) it runs as root. So a release published at 09:00 has root on every host in your inventory by 09:15, the moment a scheduled job installs whatever is newest. Nothing along that path pauses for a human being. Every control below exists to force that pause.
What the Client Checks, and What It Refuses to Promise
Open a collection archive and you find two index files. FILES.json lists every file in the bundle next to a SHA-256 checksum (Secure Hash Algorithm 256, a short fingerprint of a file's contents, where changing one byte changes the fingerprint past recognition). MANIFEST.json holds the collection metadata plus the checksum of FILES.json. One number at the top therefore covers the whole tree. During install the client compares the downloaded archive against the SHA-256 that the registry's API (application programming interface, the machine-readable side of the site) advertises. Then it extracts MANIFEST.json before anything else, checks it, and only then walks FILES.json, hashing each file as it unpacks.
That chain answers one question well. Did these bytes arrive the way the registry meant to send them? It says nothing at all about whether the person who uploaded them should have been allowed to. If the publishing account was stolen, every checksum still matches perfectly, and the install is silent and green.
Standalone roles have it worse. A role pulled from Galaxy or from GitHub is a plain source tarball: no MANIFEST.json, no FILES.json, no checksum recorded anywhere upstream. There is nothing for the client to compare against, so nothing gets compared. Read the role installer inside ansible-core and you will not find a hashing step in it at all. For roles, the version you pin is the entire control, which means the pin has to be something nobody can move underneath you.
# Committed next to the playbooks. Exact pins only, changed through a reviewed diff.collections:- name: community.generalversion: "11.4.9" # exact, never ">=11.0.0"source: https://hub.internal.example.com/api/galaxy/- name: ansible.posixversion: "2.2.2"source: https://hub.internal.example.com/api/galaxy/roles:# Roles ship no checksum manifest at all, so pin something that cannot be moved:# a full commit hash, not the 7.4.1 tag that happens to point at it today.- name: geerlingguy.dockersrc: https://github.com/geerlingguy/ansible-role-dockerversion: 3e2ec1f9b5f7a5f0d2c7e4a1b8d6c3f0a9e2b7d4
Treat that file the way you treat package-lock.json or poetry.lock. A range like >=11.0.0 hands tonight's decision to whoever published last. And notice that the role pin is a full commit hash rather than a version tag. A git tag is a sticky note on a jar. Anyone with write access can peel it off and stick it on a different jar in two commands, and the label never changes. A commit hash is computed from what is inside the jar, so moving it produces a different hash and your install breaks loudly instead of quietly. Content-addressed pins are the only kind that survive a hostile upstream.
# Install exactly what the lockfile pins, into a directory this repo ownsansible-galaxy collection install -r requirements.yml -p ./collections
Starting galaxy collection install process[WARNING]: The specified collections path '/home/dev/repo/collections' is notpart of the configured Ansible collections paths'/home/dev/.ansible/collections:/usr/share/ansible/collections'. The installedcollection will not be picked up in an Ansible run, unless within a playbook-adjacent collections directory.Process install dependency mapStarting collection install processDownloading https://hub.internal.example.com/api/galaxy/v3/plugin/ansible/content/published/collections/artifacts/community-general-11.4.9.tar.gz to /home/dev/.ansible/tmp/ansible-local-9182p3k1x8/tmpv7c0qzhaInstalling 'community.general:11.4.9' to '/home/dev/repo/collections/ansible_collections/community/general'community.general:11.4.9 was installed successfullyDownloading https://hub.internal.example.com/api/galaxy/v3/plugin/ansible/content/published/collections/artifacts/ansible-posix-2.2.2.tar.gz to /home/dev/.ansible/tmp/ansible-local-9182p3k1x8/tmpv7c0qzhaInstalling 'ansible.posix:2.2.2' to '/home/dev/repo/collections/ansible_collections/ansible/posix'ansible.posix:2.2.2 was installed successfully
Two things happened there. -p ./collections put the tree inside the repository instead of the shared ~/.ansible/collections, and ansible-galaxy warned you about it, because that directory is not yet in the configured search path. Read that warning rather than scrolling past it: a collection installed somewhere Ansible never looks is a collection that never runs. The ansible.cfg further down settles it by pointing collections_path at the same directory, and everything after this point assumes that file exists. The payoff is one repository, one dependency tree. Upgrading a collection for one project then cannot quietly change what a different project executes tonight, a failure mode that is miserable to diagnose precisely because nothing in either repository changed.
ansible-galaxy collection install -r requirements.yml reads the whole file and installs only what sits under collections:. Anything under roles: is dropped, and the message explaining that is printed at -vvv verbosity, which nobody enables in a pipeline. The job exits 0, the role never lands, and your play either fails later or picks up a stale copy already sitting on the runner. Use ansible-galaxy install -r requirements.yml with no subcommand to get both in one pass. Adding -p to that combined form flips it back to roles only, this time with a real warning about the collections it skipped.Signatures Answer the Other Half
A checksum is the tamper-evident seal on a parcel. It tells you the box was not opened in transit. It cannot tell you who packed it. For that you need a signature, and since ansible-core 2.13 the client can demand one. Servers built on Galaxy NG (the newer Galaxy server software behind Red Hat Automation Hub and self-hosted private hubs) attach a detached GPG signature over MANIFEST.json. GPG is GNU Privacy Guard, the standard open-source signing tool, and detached means the signature travels as its own small file rather than being wrapped around the thing it signs. You import the publisher's public key into a keyring you keep yourself, then tell ansible-galaxy to reject anything that key will not vouch for. Because MANIFEST.json covers FILES.json, and FILES.json covers every file, one verified signature covers the whole artifact.
# One time: put the publisher's public key in a keyring used for nothing elsegpg --no-default-keyring --keyring ~/.ansible/galaxy-keyring.kbx \--import automation-hub-signing.asc# Now refuse anything that keyring cannot vouch for.# Aimed at public Galaxy, where community content is unsigned, this should fail.ansible-galaxy collection install community.general:11.4.9 -p ./collections \-s https://galaxy.ansible.com/api/ \--keyring ~/.ansible/galaxy-keyring.kbx \--required-valid-signature-count +all
gpg: keybox '/home/dev/.ansible/galaxy-keyring.kbx' createdgpg: key 3A9C41F2D8E07B55: public key "Example Corp Automation Signing <[email protected]>" importedgpg: Total number processed: 1gpg: imported: 1Starting galaxy collection install processProcess install dependency mapStarting collection install processSignature verification failed for 'community.general': no successful signaturesERROR! Not installing community.general because GnuPG signature verification failed.
That refusal is the control working. Most content on public Galaxy carries no signature at all, so a strict install against it stops dead. Point the same command at a hub that signs on import and it proceeds normally. The whole difference between a real gate and a decorative one is the plus sign.
--required-valid-signature-count all reads as absolute and is not. The check passes when every signature that was supplied verifies, and a collection with zero signatures supplies zero failures, so it sails straight through. The same trap applies to numbers: --required-valid-signature-count 2 on an unsigned collection also passes. Prefix the value with a plus (+all, +1) to make "no valid signatures at all" a failure. Worse, if no keyring is configured, signature checking is skipped entirely no matter what count you set. When the server does hand over signatures and you have no keyring, ansible-galaxy prints a warning ending in "Skipping signature verification" and installs anyway, and that line scrolls past in CI (continuous integration) logs unread.Verify What Is on Disk, Not What You Meant to Install
Installing correctly and staying correct are two different claims. ansible-galaxy collection verify re-reads every installed file, re-hashes it, and compares the result against the manifest. By default it also pulls the registry's own copy of the collection to compare against, so a file edited on the controller after install shows up. Run it in the pipeline before the play. The only honest way to know a control works is to break something on purpose, so change one installed file and watch.
# Tamper with a single installed module, then ask whether disk still matchesprintf '\n# harmless\n' >> collections/ansible_collections/community/general/plugins/modules/ufw.pyansible-galaxy collection verify -p ./collections community.generalecho "exit: $?"
Verifying 'community.general:11.4.9'.Installed collection found at '/home/dev/repo/collections/ansible_collections/community/general'MANIFEST.json hash: 6d3a1f0c9b2e47a5c8d1e0f7b4a29c63d5e8f01a2b3c4d5e6f7089abcdef0123Collection community.general contains modified content in the following files:plugins/modules/ufw.pyexit: 1
Two limits deserve saying bluntly. verify compares your disk against the registry, so a registry that has been fully compromised can serve a lie and then confirm the same lie, handing you a clean result. Only a signature checked against a key you hold yourself breaks that circle. And --offline, which skips the network and reports whether the tree is internally consistent with its own manifest, is weaker still: an attacker who replaced the whole directory replaced the manifests along with it, and offline verify will cheerfully agree that everything matches.
Resolution Order Is the Policy
Servers are consulted in the order you list them, like ringing round suppliers and buying from the first one who says yes. The first server that answers for a collection wins. That single sentence turns a config file into a security policy. If public Galaxy is listed ahead of your private hub, anyone in the world can register yourcorp.tools on the public registry and have their code installed in place of yours, without ever touching your network. This is dependency confusion, the same trick that walked into dozens of large companies through npm and PyPI. The fix is short: list the internal hub, and list nothing behind it. Keep the API token out of ansible.cfg, because that file reaches git eventually, and pass it through the per-server environment variable instead.
[defaults]# One repository, one dependency tree. Nothing shared, nothing surprising.collections_path = ./collections[galaxy]# Resolution order is the policy. One internal entry, nothing behind it.server_list = internal_hub# Signature checking is opt-in twice over: no keyring means no check at all,# and "all" without the plus is satisfied by zero signatures.gpg_keyring = ~/.ansible/galaxy-keyring.kbxrequired_valid_signature_count = +all[galaxy_server.internal_hub]url = https://hub.internal.example.com/api/galaxy/# No token in this file. In CI, export it instead:# ANSIBLE_GALAXY_SERVER_INTERNAL_HUB_TOKEN=<token>
There is a second ordering problem that has nothing to do with registries. When you run a playbook, Ansible searches the collections/ directory sitting next to that playbook before it consults collections_path, and before it scans installed Python packages. Whatever lives beside the playbook wins, the way a note left on your desk beats the filing cabinet nobody opens. A stale directory left behind by a previous branch, a cached CI workspace nobody cleans, or a path an attacker can write to on the controller quietly outranks the pinned, verified copy you installed. Package scanning is on by default too (collections_scan_sys_path), so any pip package shipping an ansible_collections tree joins the search path uninvited. List what is actually there.
ansible-galaxy collection list
# /home/dev/repo/collections/ansible_collectionsCollection Version----------------- -------ansible.posix 2.2.2community.general 11.4.9# /usr/lib/python3.12/site-packages/ansible_collectionsCollection Version----------------- -------ansible.posix 1.6.2community.general 10.7.9
Two copies of community.general on one controller, four major versions apart, and which one a play executes depends entirely on which path it reaches first. Print this list in CI and fail the job when a path or a version you did not expect shows up. It costs one command and catches both the stale-workspace accident and the deliberate plant.
Own the Distribution Point
At any real size the answer is architectural: stop letting build machines talk to public registries. A private hub becomes the one door content comes through, where a new version is imported only after somebody has read the changelog, re-signed it with your key, and staged it before production sees it. For air-gapped estates, meaning machines with no route to the internet at all, ansible-galaxy collection download pulls the pinned tarballs and writes a matching requirements file beside them, and --offline installs from that bundle without contacting a server.
# Connected side: pull the exact pinned tarballs into a directory you can carryansible-galaxy collection download -r requirements.yml -p ./vendor# Air-gapped side: install from the generated file, no registry contact at allansible-galaxy collection install -r vendor/requirements.yml -p ./collections --offline
Process download dependency mapStarting collection download process to '/home/dev/repo/vendor'Downloading collection 'community.general:11.4.9' to '/home/dev/repo/vendor'Collection 'community.general:11.4.9' was downloaded successfullyDownloading collection 'ansible.posix:2.2.2' to '/home/dev/repo/vendor'Collection 'ansible.posix:2.2.2' was downloaded successfullyWriting requirements.yml file of downloaded collections to '/home/dev/repo/vendor/requirements.yml'
Be honest about the costs. Pinning moves work from hope to process: somebody now owns reading changelogs and raising bump pull requests, and a team that quietly stops doing that will sit on known-vulnerable content for years. The generated vendor/requirements.yml lists only tarball names and versions, because the download command deliberately does not fetch detached signatures. An air-gapped bundle therefore carries checksums and no identity at all. Mirror the signature files yourself and point at them with the signatures: key in requirements.yml if you need identity on the far side.
None of this inspects behaviour either. A correctly pinned, correctly signed, entirely authentic release can still be hostile, because a hijacked maintainer account produces genuine signatures. That is why moving a pin should mean reading the diff and rehearsing against a staging inventory with --check --diff before the fleet ever sees it. The next rung up is freezing the whole set, ansible-core and your pinned collections together, into an execution environment image (a container built with ansible-builder that carries the exact automation stack) so every run on every machine starts from one artifact you can point a finger at.
--keyring ~/.ansible/galaxy-keyring.kbx --required-valid-signature-count all, and a collection carrying no signature at all installs without a murmur. Why?--offline controls whether distribution servers are contacted and has nothing to do with GPG checks.ansible-galaxy collection install -r requirements.yml -p ./collections, exits 0, and reports two collections installed. The play then fails with "the role geerlingguy.docker was not found", even though that role is pinned in the same requirements.yml. What happened?