Ansible in CI/CD
Lint, check mode, and safe automation.
A playbook is a list of instructions you hand to a program that can become root on every server you own, and it does not ask twice. Push it from a laptop and the only review it ever got was your own eyes, scrolling fast, at the end of a long day. A pipeline (the build robot your code host runs on every push, also called continuous integration or CI) is the review. Before a change touches production it gets read by a linter, rehearsed on paper, then applied for real to throwaway machines that are destroyed straight afterwards. The point of all of it is to meet the broken YAML (YAML Ain't Markup Language, the indented text format playbooks are written in), the deprecated module, and the task that restarts sshd (the OpenSSH server daemon that answers every remote login) on every single run inside a build log instead of inside an incident channel.
The Gate That Never Touches a Host
Start with the check that costs nothing. A proofreader can tell you a letter is grammatical without knowing whether the address on the envelope is real, and that is exactly the trade you want first: no SSH (Secure Shell, the encrypted remote-login protocol Ansible rides on) keys, no inventory, no target machines, nothing at risk. ansible-playbook --syntax-check loads your playbook and every file it pulls in at parse time, builds the play and task objects in memory, and throws them away. If a task has two module calls glued onto it, or a role path is wrong, or an editor turned a tab into a landmine, you find out in about a second. Run it on every push, before anything else, because everything downstream is slower and needs credentials.
# Parse the whole play tree. No SSH, no inventory, no host contacted.ansible-playbook site.yml --syntax-check
[WARNING]: No inventory was parsed, only implicit localhost is availableERROR! conflicting action statements: ansible.builtin.template, ansible.builtin.serviceThe error appears to be in '/srv/infra/roles/sshd/tasks/main.yml': line 9, column 3, but maybe elsewhere in the file depending on the exact syntax problem.The offending line appears to be:- name: Render sshd_config^ here
That check is shallow by design, and it has a blind spot worth knowing about. It only sees what is stitched together at parse time: import_tasks, import_playbook, and the roles: list. Anything behind include_tasks is resolved while the play is running, so a broken file behind a dynamic include sails straight through this gate. The syntax check also does not care whether ansible.builtin.template (the fully qualified collection name, or FQCN, which is the modern way to write a module: namespace, collection, module) was handed an option that module has never heard of, and it will not mention that a template task with no mode: is about to write a config file with whatever permissions the process umask (the default-permission mask new files inherit) happened to allow. That is the linter's job. ansible-lint is the second reader, the one who knows the house style and the fire code: deprecated syntax, tasks with no name so nobody can read the output, a bare command with no changed_when (the task reports that it changed the machine on every run, forever, whether it did or not), risky file permissions, and module options that do not exist in that module's argument spec. It ships separately from ansible-core, so pin it in the same place you pin everything else.
# Style and safety rules, still zero hosts and zero credentials.ansible-lint --profile productionecho "exit=$?"
WARNING Listing 3 violation(s) that are fatalrisky-file-permissions: File permissions unset or incorrect.roles/app/tasks/main.yml:21 Task/Handler: ansible.builtin.template src=app.conf.j2 dest=/etc/app/app.confno-changed-when: Commands should not change things if nothing needs doing.roles/sshd/tasks/main.yml:8 Task/Handler: ansible.builtin.command cmd=/usr/local/bin/refresh-krlname[missing]: All tasks should be named.roles/sshd/tasks/main.yml:14 Task/Handler: ansible.builtin.service name=sshd state=restartedRule Violations Summarycount tag profile rule associated tags1 name[missing] basic idiom1 no-changed-when shared command-shell, idempotency1 risky-file-permissions safety unpredictabilityFailed: 3 failure(s), 0 warning(s) on 12 files. Last profile that met the validation criteria was 'min'.exit=2
The rules arrive in stacked profiles: min, basic, moderate, safety, shared, production, each one carrying everything below it. Pin the profile in a .ansible-lint file at the repo root so the ruleset is version controlled, argued about in pull requests, and identical on a laptop and on a runner. Watch the last line of that output. It names the strictest profile your repo currently passes, which turns "we should tidy the roles up sometime" into a number you can move. Exit code 2 means violations were found, so the stage fails on its own with no shell plumbing. Two flags earn their keep. ansible-lint --fix rewrites the mechanically fixable violations in one noisy commit instead of dribbling them through reviews, and --sarif-file lint.sarif writes findings as SARIF (Static Analysis Results Interchange Format), the file format code-scanning dashboards read, so playbook problems land next to your container and dependency findings rather than in a log nobody opens.
# Repo root. Reviewed like any other code.profile: production # min < basic < moderate < safety < shared < productionoffline: true # no Galaxy calls mid-lint; deps come from one auditable stepexclude_paths:- .cache/- molecule/warn_list:- experimental # new rules arrive as warnings before they can fail a buildskip_list: [] # every entry here needs a comment and a date it disappears
Pin the tools as well. A fresh ansible-lint release adds rules, and a Tuesday morning upgrade turns a green repo red on code nobody touched. The drift that bites harder is on the collection side. ansible-galaxy install -r requirements.yml with nothing pinned downloads whichever tarball happens to be newest on Ansible Galaxy (the public sharing site for roles and collections) at build time, and your pipeline then runs that code as root across the fleet. Pin exact versions. For internal roles pulled from Git, pin a commit hash rather than a tag, because a tag is a label somebody can move to point at different code, and a hash is the code.
# Installed in CI with: ansible-galaxy install -r requirements.yml# (plain `install` reads both keys; the role/collection subcommands read only one)collections:- name: ansible.posixversion: 1.5.4- name: community.cryptoversion: 2.22.3roles:- name: baselinesrc: git+ssh://[email protected]/infra/ansible-baseline.gitversion: 4f2c9a1b8e5d3c7a0b1f6e2d9c4a8b3f1e7d5c2a # a commit, not a tag
Check Mode Is a Rehearsal, Not a Promise
Check mode is reading the recipe out loud in an empty kitchen. You catch the missing eggs. You do not find out the oven is broken. With --check (short flag -C) every module that supports it reports what it would have done and writes nothing. Add --diff (-D) and you get the exact lines that would change in every managed file and rendered template. Ansible still connects, still gathers facts, still evaluates your conditionals against the live machine. That is the whole value of it: the preview is computed from the host's real state, not from a guess about the host's state.
# Rehearse against staging: connect, evaluate, show the diff, write nothing.ansible-playbook -i inventories/staging site.yml --check --diff --limit web
PLAY [Harden the web tier] *****************************************************TASK [Gathering Facts] *********************************************************ok: [web-01.stg]TASK [sshd : Read the effective sshd configuration] ****************************ok: [web-01.stg]TASK [sshd : Render sshd_config] ***********************************************--- before: /etc/ssh/sshd_config+++ after: /home/runner/.ansible/tmp/ansible-local-9142f3ktu1qz/tmp8vn2ac/sshd_config.j2@@ -31,7 +31,7 @@PubkeyAuthentication yes-PermitRootLogin yes+PermitRootLogin noPasswordAuthentication nochanged: [web-01.stg]TASK [sshd : Apply the pending schema migration] *******************************skipping: [web-01.stg]RUNNING HANDLER [sshd : Restart sshd] ******************************************changed: [web-01.stg]PLAY RECAP *********************************************************************web-01.stg : ok=4 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0
Read that recap the way a defender would. Two changes on a staging host you believed was already converged means either the host drifted since the last run or the change is bigger than the pull request claimed. Both are worth ten seconds before you continue. Notice that the handler fired too. A handler is a task that sits idle until something notifies it, like a note on the fridge saying restart sshd when you are done, and notified handlers do run in check mode. So you get to see that this change costs an sshd restart before you decide to spend that restart on two hundred machines at once.
Two task keywords give you the sharp edges here. check_mode: false forces a task to execute for real even during a dry run, which is what you want on a read-only command whose output later tasks branch on. check_mode: true does the opposite and pins a task to simulation on every run, including a genuine apply, which is how you keep something irreversible (a schema migration, a fleet-wide reboot) on a permanent leash until a human deliberately deletes that line. Pair either with changed_when: false on reads so a lookup never inflates your change count. And put validate: on file-writing tasks. The command runs against the candidate file while it is still a temporary file, and Ansible moves it into place only if that command exits zero, so a template that renders an sshd config the daemon refuses to parse never reaches disk.
- name: Read the effective sshd configurationansible.builtin.command: /usr/sbin/sshd -Tregister: sshd_effectivebecome: truecheck_mode: false # must really run, even under --checkchanged_when: false # reading something is not changing it- name: Render sshd_configansible.builtin.template:src: sshd_config.j2dest: /etc/ssh/sshd_configowner: rootgroup: rootmode: "0600"validate: /usr/sbin/sshd -t -f %s # a config sshd rejects never landsbecome: truenotify: Restart sshd- name: Apply the pending schema migrationansible.builtin.command: /opt/app/migrate.shcheck_mode: true # simulated on every run, including real applies
command and shell tasks come back as skipping without telling you a thing about what they would have done, so a task that would have failed on the real apply looks cheerful. Worse, check mode breaks chains: if task A creates a directory and task B writes into it, A creates nothing during the rehearsal, so B looks at a host missing that directory and either fails or returns a result that means nothing. Your validate: safety net has nothing to validate either, because no candidate file is ever written. Treat --check --diff as a rehearsal that catches obvious mistakes on an already-converged host. It is not proof that the real run succeeds.Molecule Applies the Role Twice and Dares It to Change Anything
Linting proves a role parses. Molecule proves it runs. Picture a test kitchen you burn down after every service: Molecule builds a disposable target (a Podman or Docker container, or a real cloud instance), applies your role to it with converge, applies the identical role a second time, runs your assertions with verify, then destroys the whole thing. It installs separately from ansible-core, alongside a driver plugin for whichever kind of target you use. The second apply is the part worth paying for. Molecule watches that run and fails the build if any task reports changed, which is a direct automated test of the promise every well-written module makes: it is idempotent (run it twice and the second run changes nothing).
# Full lifecycle in CI: create -> converge -> apply again -> verify -> destroymolecule test# Fast local loop: the container stays up between runsmolecule converge # apply the rolemolecule verify # rerun only the assertionsmolecule login # shell into the instance and look around
INFO default scenario test matrix: dependency, cleanup, destroy, syntax, create,prepare, converge, idempotence, side_effect, verify, cleanup, destroyINFO Running default > createINFO Running default > prepareINFO Running default > convergePLAY RECAP *********************************************************************instance : ok=11 changed=6 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0INFO Running default > idempotencePLAY RECAP *********************************************************************instance : ok=11 changed=1 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0CRITICAL Idempotence test failed because of the following tasks:* [instance] => sshd : Refresh the revoked key list
Molecule has named the task that lied. ansible.builtin.command runs a program and has no idea whether that program changed anything, so it reports changed every single time. Left alone, that one task notifies a handler on every run, and a routine playbook that should do nothing at all starts restarting sshd across the fleet at two in the morning, dropping sessions and filling logs with noise. It also ruins your cheapest detection: if a run is always changed=6, nobody can spot the night it should have been zero and was not. The fix is a changed_when that reads the command's real output, or a module that tracks state itself instead of a shell script that cannot.
verify is where you write down what working means, in assertions, so the next person cannot quietly undo it. Assert against effective state rather than against the file you wrote, because the file is your input and the daemon's parsed config is the truth. sshd -T prints exactly what the server will enforce, in lowercase, including defaults you never typed, which is a far stronger claim than grepping your own template. One limit on that: without a -C connection spec, sshd -T does not apply Match blocks, so a block that re-enables password logins for one group will not appear in the output you are asserting on. Container testing has two honest costs too. To exercise ansible.builtin.systemd_service tasks properly the container has to run systemd, which needs an init process and the SYS_ADMIN capability (the Linux permission covering mounts and most other administrative syscalls), so keep those containers on ephemeral CI runners rather than on the workstation where you also read email. And a minimal image ships no SSH host keys, so sshd -t fails there until Molecule's prepare step generates them with ssh-keygen -A.
- name: Verifyhosts: allgather_facts: falsebecome: truetasks:- name: Ask sshd to parse its own config fileansible.builtin.command: /usr/sbin/sshd -tchanged_when: false- name: Read what sshd will actually enforceansible.builtin.command: /usr/sbin/sshd -Tregister: effectivechanged_when: false- name: Root login and password auth must be offansible.builtin.assert:that:- "'permitrootlogin no' in effective.stdout"- "'passwordauthentication no' in effective.stdout"fail_msg: "sshd would still accept root or password logins"
Handing the Vault Password to a Robot
Every gate past the linter needs the Ansible Vault password. Vault is the built-in encryption for files you want to keep in Git anyway (variable files, keys, certificates), scrambled with AES-256 behind a passphrase, and a build runner is not a person you can prompt for one. --vault-password-file takes a path, with a detail that changes everything: if the file is executable, Ansible runs it and reads the password from its standard output. So the "file" becomes a short script that fetches the secret from your secrets manager at the moment it is needed and never writes it to disk. Set ANSIBLE_VAULT_PASSWORD_FILE in the job environment and nobody has to remember the flag. A password sitting in a file on the runner is the key under the doormat; this is a keyholder who hands it over and takes it straight back. What you are avoiding shows up in every leaked-pipeline post-mortem: echoing the password into a temp file, where it outlives the step, gets swept up by cache and artifact uploads, and sits readable on a shared runner for the next job to find.
#!/usr/bin/env bash# chmod 0700. Ansible runs this because it is executable and reads stdout.# Prints the password and nothing else: no banners, no debug lines.set -euo pipefailaws secretsmanager get-secret-value \--secret-id ci/ansible-vault \--query SecretString --output text
--diff renders the file it is about to write and prints the changed lines. If that template contains a database password, an API token, or a private key that Vault decrypted a moment earlier, the plaintext lands in the job log, which is readable by everyone with access to the project, copied into artifacts, and forwarded to whatever log service your CI vendor uses. Ansible does not redact it for you. Put no_log: true on tasks that render or receive secrets, which suppresses the task arguments and the diff together, or drop --diff on the plays that touch credentials. Be aware that no_log is blunt: it also hides the diff you wanted to review, and it does nothing about the same secret being written to the target's disk by a task whose mode: is too loose.The Last Gate Is a Human
Automate everything up to the apply, then stop and make a person press the button. The approval is not ceremony. The reviewer is looking at one specific artifact, the --check --diff output from staging, and answering one question: does this diff match what the pull request said it would do. After approval, shrink the blast radius (how much of the fleet a bad change can reach before you notice and stop it) on the way in. --list-hosts prints the exact target set without running anything and costs nothing, so print it and read it. Send the first real pass at one machine with --limit web-01.prod, then let the play roll through the rest in batches.
- name: Harden the web tierhosts: webbecome: trueserial: "10%" # ten percent of the tier at a time, not all of itmax_fail_percentage: 0 # any failure in a batch stops the rolloutroles:- sshd
serial turns one big run into a queue of small ones, so a change that breaks logins breaks ten percent of the tier while you still have shells open on the rest. max_fail_percentage: 0 is the brake: Ansible aborts the play when the share of failed hosts in a batch is greater than that number, and greater than zero means one host. Leave it out and the default behaviour is to carry on with whatever hosts are still alive, walking the same breakage across every machine you own and telling you at the end.
Then point the same playbook back at production on a schedule, in check mode, and treat any reported change as an alarm. On a fleet Ansible owns, a changed result at 3am is a night watchman finding a door open: a managed host was edited by hand, by a well-meaning engineer or by somebody else. One catch. ansible-playbook --check exits 0 even when every task reports a change, because from Ansible's point of view nothing failed. If you want drift to fail the job, you have to say so yourself.
# Nightly drift job. Production is managed, so nothing should come back changed.set -euo pipefailexport ANSIBLE_VAULT_PASSWORD_FILE=ci/vault-pass.shansible-playbook -i inventories/prod site.yml --check --diff | tee drift.logif grep -qE 'changed=[1-9]' drift.log; thenecho "Drift on production: a managed host was edited outside Ansible"exit 1fi
TASK [sshd : Render sshd_config] ***********************************************--- before: /etc/ssh/sshd_config+++ after: /home/runner/.ansible/tmp/ansible-local-3318qk9d0war/tmpz1c7ye/sshd_config.j2@@ -31,7 +31,7 @@PubkeyAuthentication yes-PermitRootLogin yes+PermitRootLogin noPasswordAuthentication nochanged: [db-02.prod]RUNNING HANDLER [sshd : Restart sshd] ******************************************changed: [db-02.prod]PLAY RECAP *********************************************************************db-02.prod : ok=4 changed=2 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0web-01.prod : ok=3 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0web-02.prod : ok=3 changed=0 unreachable=0 failed=0 skipped=1 rescued=0 ignored=0Drift on production: a managed host was edited outside Ansible
That output is a complete finding on its own: the host, the file, the exact line, and a timestamp, arriving before anyone opens a ticket. Keep drift.log as a build artifact so you can diff last night against tonight, and remember what --diff does before you save it, because any play in that run which renders a secret has now written it into the artifact. The first time this job catches PermitRootLogin yes reappearing on a production database server, every hour you spent wiring up the linter and the check-mode stage has already paid for itself.
ansible-playbook site.yml --syntax-check actually prove?ansible.builtin.command: /usr/sbin/sshd -T, registers the output, and later tasks branch on what it found. What keeps that working during a --check --diff run?molecule test clears converge, then prints CRITICAL Idempotence test failed because of the following tasks: * [instance] => sshd : Refresh the revoked key list. That task is ansible.builtin.command: /usr/local/bin/refresh-krl. What is happening?Try this
Run ansible-playbook site.yml --syntax-check on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: a green --check is not a promise. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.