Test Kitchen & InSpec
Test cookbooks before they ship.
A cookbook is a pile of instructions you hand to a program that runs as root (the Linux account allowed to do anything) on every machine that receives it. Get one file mode wrong, the permission bits that decide who may read a file, and you have quietly published a shared credential to every local account on four thousand hosts. Get one platform branch wrong and half the fleet comes back from a reboot with no web tier. So you do what any kitchen does before a new dish reaches the menu: cook it once, somewhere disposable, and taste it. Test Kitchen is named after exactly that.
Two tools split the work. Test Kitchen builds a throwaway machine (a container, a virtual machine, or a cloud instance), copies your cookbook onto it, runs Chef Infra Client, and then throws the machine away. InSpec is the taster. It connects to the finished machine and asks it questions in near-English: is nginx installed, is port 80 listening, is this file mode 0640 and owned by root. Kitchen produces the evidence. InSpec judges it. Each is useful on its own. Together they are the gate you put in front of every cookbook change.
Describing the machines you test on
Everything Kitchen does comes out of one file, kitchen.yml, sitting in the root of your cookbook. (YAML is indented plain text. Read it like a form with labelled boxes.) Six boxes matter. The driver decides what kind of machine to build. The transport decides how to talk to that machine once it exists. The provisioner decides how Chef gets run on it. The verifier decides who inspects the result afterwards. The platforms list the operating systems to build, and the suites list the combinations of run list (the ordered set of recipes Chef applies) and attributes (the settings those recipes read) you want tested. Kitchen multiplies platforms by suites, so two platforms and one suite gives you two instances. Older cookbooks call this file .kitchen.yml with a leading dot. That still loads, but kitchen.yml is the current name.
---driver:name: dokken # Docker-backed instances, the fast optionprivileged: true # systemd needs it (see the warning below)chef_version: "18" # pin the client version the fleet actually runstransport:name: dokkenprovisioner:name: dokken # inherits every chef_infra settingpolicyfile: Policyfile.rb # test the exact locked versions you shipdeprecations_as_errors: true # a deprecation warning fails the runchef_license: accept-no-persistverifier:name: inspecplatforms:- name: ubuntu-22.04driver:image: dokken/ubuntu-22.04pid_one_command: /bin/systemd- name: rockylinux-9driver:image: dokken/rockylinux-9pid_one_command: /usr/lib/systemd/systemdsuites:- name: default# The Policyfile owns the run list, so a run_list written here is ignored.# To exercise a different entry point, name one in Policyfile.rb and add:# provisioner:# named_run_list: smokeattributes:web:upstream_host: api.internalverifier:inspec_tests:- test/integration/default # optional: this is the default path anyway
dokken is the Docker-backed driver that ships with Chef Workstation, and it is the fast one. Instead of installing Chef Infra Client into the container on every run, it mounts the client from a separate image and converges against a base image that boots systemd, so a create-plus-converge cycle costs seconds rather than the minutes a full virtual machine needs. The dokken provisioner inherits from Chef's own provisioner, so every chef_infra setting works here too. (chef_infra is the current name for what used to be called chef_zero. The old name still loads as an alias, and you will meet it in older cookbooks.) Pointing policyfile at your Policyfile.rb makes Kitchen resolve the same locked cookbook versions your fleet runs, rather than whatever happens to be sitting in a local cache. deprecations_as_errors turns a deprecation warning into a failed run, which is how you hear about a removed feature now instead of during the next major upgrade. And chef_license stops the client pausing to ask a human to accept a licence in a pipeline where no human is watching.
One trap arrives with that Policyfile. Once Kitchen is resolving a policy, the policy owns the run list, and a run_list you write in the suite is ignored. If you want a suite to exercise a different entry point, define a named run list inside Policyfile.rb and point the suite's provisioner at it with named_run_list. Attributes written in the suite still apply, so a suite is still how you test the same recipes with production-shaped settings.
$ kitchen list
Instance Driver Provisioner Verifier Transport Last Action Last Errordefault-ubuntu-2204 Dokken Dokken Inspec Dokken <Not Created> <None>default-rockylinux-9 Dokken Dokken Inspec Dokken <Not Created> <None>
Look at the instance names. Kitchen builds each one from suite plus platform and strips the dots, so the ubuntu-22.04 platform becomes default-ubuntu-2204. Every other kitchen command takes one of those names, or a Ruby regular expression that matches several of them, which is how you iterate on a single platform without waiting for the rest of the matrix. When the YAML is not behaving the way you read it, kitchen diagnose --all prints the fully merged configuration Kitchen actually ended up with, including all the defaults you inherited and never wrote down.
Converge, then converge again
$ kitchen converge default-ubuntu-2204
-----> Starting Test Kitchen (v3.7.0)-----> Creating <default-ubuntu-2204>...Creating kitchen sandbox at /home/dev/.dokken/sandbox/9f3c1a7eCreating verifier sandbox at /home/dev/.dokken/verifier/9f3c1a7eBuilding work image..Creating container default-ubuntu-2204Finished creating <default-ubuntu-2204> (0m4.11s).-----> Converging <default-ubuntu-2204>...Policyfile found at /home/dev/cookbooks/web/Policyfile.rb, using it to resolve dependenciesPreparing dna.jsonPreparing validation.pemPreparing client.rb+---------------------------------------------+✔ 2 product licenses accepted.+---------------------------------------------+Starting Chef Infra Client, version 18.4.12Patents: https://www.chef.io/patentsInfra Phase startingUsing policy 'web' at revision '9f3c1a7e2b6d0c4a'Synchronizing cookbooks:- web (0.4.0)Installing cookbook gem dependencies:Compiling cookbooks...Loading Chef InSpec profile files:Loading Chef InSpec input files:Loading Chef InSpec waiver files:Converging 4 resourcesRecipe: web::default* apt_package[nginx] action install- install version 1.18.0-6ubuntu14.6 of package nginx* template[/etc/nginx/conf.d/upstream.conf] action create- create new file /etc/nginx/conf.d/upstream.conf- update content in file /etc/nginx/conf.d/upstream.conf from none to 6b1f2e(suppressed sensitive resource)* file[/etc/nginx/sites-enabled/default] action delete- delete file /etc/nginx/sites-enabled/default* service[nginx] action enable (up to date)* service[nginx] action start- start service service[nginx]Running handlers:Running handlers completeInfra Phase complete, 4/4 resources updated in 12 secondsFinished converging <default-ubuntu-2204> (0m41.02s).-----> Test Kitchen is finished. (0m46.88s)
That is a real Chef Infra Client run, not a simulation of one. Kitchen copied the cookbook and the resolved policy onto the instance, then ran the client in local mode, which spins up a tiny Chef server inside the client's own memory (the chef-zero server) and converges against it. Same machinery a production node uses, minus the Chef Infra Server. All four resources changed, which is exactly what you want on a clean box the first time through.
Now run it again. Do not destroy anything in between. This is the highest-value test in the whole stack and hardly anybody writes it down.
$ kitchen converge default-ubuntu-2204 # again, with no destroy in between
-----> Converging <default-ubuntu-2204>...Starting Chef Infra Client, version 18.4.12Infra Phase startingUsing policy 'web' at revision '9f3c1a7e2b6d0c4a'Converging 4 resourcesRecipe: web::default* apt_package[nginx] action install (up to date)* template[/etc/nginx/conf.d/upstream.conf] action create (up to date)* file[/etc/nginx/sites-enabled/default] action delete (up to date)* service[nginx] action enable (up to date)* service[nginx] action start (up to date)Running handlers:Running handlers completeInfra Phase complete, 0/4 resources updated in 03 secondsFinished converging <default-ubuntu-2204> (0m9.44s).
Zero of four. That is idempotency (running the same thing twice changes nothing the second time), and it works like a light switch: flipping it on when it is already on should do nothing at all. The difference matters more than it sounds. A cookbook that is idempotent can run every thirty minutes forever. One that is not bounces nginx on a schedule. A non-idempotent resource in a fleet-wide cookbook is a slow-motion, self-inflicted outage: every node restarts a service on every converge, notifications fire endlessly, and you find out during an incident, at the exact moment you can no longer tell routine churn from an attacker's changes. If the second converge reports anything other than 0 of N, you have a bug, even though nothing failed.
Writing the InSpec checks
InSpec is a language for interrogating a machine, and the shape of it is a clipboard with a checklist on it. Each item on the clipboard is a control: a named group of assertions carrying a severity and a description. Inside a control you use resources (package, service, port, file, processes, command, and a few hundred more), and each resource hands you matchers, the short phrases like should be_installed that do the comparing. The whole file is Ruby. That stops mattering right up until two platforms disagree, and then it matters a great deal.
# Kitchen picks this up on its own: test/integration/<suite name>/title 'web cookbook: service, secret file, and default-vhost hardening'# Plain Ruby. It runs where InSpec runs, but `os` describes the box under test.web_user = os.debian? ? 'www-data' : 'nginx'control 'web-01' doimpact 1.0title 'nginx is installed, enabled at boot, and listening on 80'desc 'A web tier that is running but not enabled comes back dead after a reboot.'describe package('nginx') doit { should be_installed }enddescribe service('nginx') doit { should be_enabled }it { should be_running }enddescribe port(80) doit { should be_listening }its('protocols') { should include 'tcp' }enddescribe processes('nginx') doits('users') { should include web_user } # workers must drop privilegeendendcontrol 'web-02' doimpact 1.0title 'The upstream config holding the backend token is not world-readable'tag 'secrets'ref 'CIS Benchmarks', url: 'https://www.cisecurity.org/cis-benchmarks'describe file('/etc/nginx/conf.d/upstream.conf') doit { should be_file }its('owner') { should eq 'root' }its('group') { should eq web_user }its('mode') { should cmp '0640' }it { should_not be_more_permissive_than('0640') }endendcontrol 'web-03' doimpact 0.7title 'The stock welcome vhost is gone'desc 'nginx -T prints the effective config, so this checks reality, not a file path.'describe command('nginx -T 2>/dev/null') doits('stdout') { should_not match(%r{root\s+/var/www/html}) } # Debian defaultits('stdout') { should_not match(%r{root\s+/usr/share/nginx/html}) } # RHEL defaultendend
Several things in there earn their keep. impact is a number from 0.0 to 1.0 saying how much you care, and reporting tools rank findings by it. cmp is InSpec's forgiving comparison. A file's mode comes back as a plain number, 416, which is what 0640 means once you read it as octal (base 8, the way Unix permissions have always been written). cmp knows that '0640', 0640, and 416 are the same thing, so you sidestep the classic type mismatch that eq would fail on. be_more_permissive_than is the matcher to reach for on anything holding a credential, because it fails 0644 and 0666 while accepting 0600, without you listing every acceptable mode by hand. And os.debian? handles the awkward fact that Debian's nginx runs its workers as www-data while Red Hat's runs them as nginx.
$ kitchen verify default-ubuntu-2204
-----> Verifying <default-ubuntu-2204>...Loaded tests from {:path=>"/home/dev/cookbooks/web/test/integration/default"}Profile: tests from {:path=>"/home/dev/cookbooks/web/test/integration/default"}Version: (not specified)Target: docker://f3c9a41d8e02Target ID: 2f8b0c1e-4d17-4a9b-9c3e-6a1b8d0f2c55✔ web-01: nginx is installed, enabled at boot, and listening on 80✔ System Package nginx is expected to be installed✔ Service nginx is expected to be enabled✔ Service nginx is expected to be running✔ Port 80 is expected to be listening✔ Port 80 protocols is expected to include "tcp"✔ Processes nginx users is expected to include "www-data"× web-02: The upstream config holding the backend token is not world-readable (2 failed)✔ File /etc/nginx/conf.d/upstream.conf is expected to be file✔ File /etc/nginx/conf.d/upstream.conf owner is expected to eq "root"✔ File /etc/nginx/conf.d/upstream.conf group is expected to eq "www-data"× File /etc/nginx/conf.d/upstream.conf mode is expected to cmp == "0640"expected: 0640got: 0644(compared using `cmp` matcher)× File /etc/nginx/conf.d/upstream.conf is expected not to be more permissive than "0640"✔ web-03: The stock welcome vhost is gone✔ Command: `nginx -T 2>/dev/null` stdout is expected not to match /root\s+\/var\/www\/html/✔ Command: `nginx -T 2>/dev/null` stdout is expected not to match /root\s+\/usr\/share\/nginx\/html/Profile Summary: 2 successful controls, 1 control failure, 0 controls skippedTest Summary: 11 successful, 2 failures, 0 skipped>>>>>> ------Exception------->>>>>> Class: Kitchen::ActionFailed>>>>>> Message: 1 actions failed.>>>>>> Verify failed on instance <default-ubuntu-2204>. Please see .kitchen/logs/default-ubuntu-2204.log for more details>>>>>> ----------------------
Read what that caught. Somebody edited the template resource and dropped the mode '0640' line, so Chef created the file under the client's default umask (the mask that decides permissions on new files when nobody says otherwise) and it landed on 0644. Nothing errored. The converge was green. nginx started and served traffic like any other day. But the upstream token in that file is now readable by every local account on the box, including the unprivileged nginx worker, so any file-read bug in the web application hands an attacker your backend credential. Go back and look at the converge output. The template block printed a content update and no mode line at all. That absence was the only hint, and nobody reads converge logs that closely. A converge tells you Chef did what the recipe said. Verify tells you whether what the recipe said was right.
The full loop, on every platform
kitchen test runs the entire cycle in one command: destroy any leftover instance, create, converge, set up the verifier, verify, destroy. It does that for every instance in the matrix, so a single command answers whether your change works on every operating system the cookbook claims to support. Add -c to run instances in parallel. By default it destroys an instance only when that instance passed, so a failure leaves the box standing and kitchen login drops you into a shell on it, still broken, for you to poke at. Pass --destroy=never if you want survivors either way, and remember to run kitchen destroy afterwards when the driver bills by the hour.
The mode bug is fixed now. Here is the same change across the whole matrix.
$ kitchen test -c 2
-----> Starting Test Kitchen (v3.7.0)-----> Cleaning up any prior instances of <default-ubuntu-2204>-----> Cleaning up any prior instances of <default-rockylinux-9>-----> Testing <default-ubuntu-2204>-----> Testing <default-rockylinux-9>...(create and converge on both; interleaved output trimmed here)...[default-ubuntu-2204] Infra Phase complete, 4/4 resources updated in 12 seconds[default-rockylinux-9] Infra Phase complete, 3/4 resources updated in 17 seconds[default-ubuntu-2204] Profile Summary: 3 successful controls, 0 control failures, 0 controls skipped[default-rockylinux-9] Profile Summary: 2 successful controls, 1 control failure, 0 controls skipped× web-03: The stock welcome vhost is gone (1 failed)✔ Command: `nginx -T 2>/dev/null` stdout is expected not to match /root\s+\/var\/www\/html/× Command: `nginx -T 2>/dev/null` stdout is expected not to match /root\s+\/usr\/share\/nginx\/html/expected "# configuration file /etc/nginx/nginx.conf:\n..." not to match /root\s+\/usr\/share\/nginx\/html/Diff:@@ -1 +1 @@-/root\s+\/usr\/share\/nginx\/html/+" root /usr/share/nginx/html;"-----> Destroying <default-ubuntu-2204>...Finished destroying <default-ubuntu-2204> (0m2.19s).Finished testing <default-ubuntu-2204> (2m04.88s).-----> Test Kitchen is finished. (2m41.55s)>>>>>> ------Exception------->>>>>> Class: Kitchen::ActionFailed>>>>>> Message: 1 actions failed.>>>>>> Verify failed on instance <default-rockylinux-9>. Please see .kitchen/logs/default-rockylinux-9.log for more details>>>>>> ----------------------
This is the break the whole exercise exists to catch. Both converges succeeded. Rocky's did leave a clue, if anyone had been counting: 3 of 4 resources updated against Ubuntu's 4 of 4. The missing one is file '/etc/nginx/sites-enabled/default' with action :delete, because deleting a path that was never there is a successful no-op that reports up to date. On Debian and Ubuntu that line removes the stock welcome vhost. On Rocky Linux that directory does not exist, since Red Hat keeps its default server block inside nginx.conf, so the recipe congratulated itself and left the default site serving a directory nobody owns. Ubuntu was green. Rocky was quietly wrong. Asking nginx -T for the effective config, rather than checking whether a file path is gone, is what made the difference visible.
The two cheaper layers
Kitchen is the slow layer. Two cheaper ones run first, and the three together work like checking a letter before you post it: a spellchecker, a colleague reading it back to you, then actually posting it and seeing what comes back. cookstyle is the spellchecker, a linter (a tool that reads code without running it) that ships with Chef Workstation, built as a stripped-down RuboCop, the standard Ruby style checker, carrying several hundred Chef-specific rules. ChefSpec is the colleague reading it back. It runs the converge entirely in memory against fake node data from Fauxhai, so it can tell you what resources the recipe would create on Rocky Linux without ever building a Rocky Linux machine.
$ cookstyle . # add -a to autocorrect the safe offences in place
Inspecting 11 files..R......R.Offenses:metadata.rb:5:9: R: [Correctable] Chef/Sharing/InvalidLicenseString: Cookbook metadata.rb does not use a SPDX compliant license string or "all rights reserved". See https://spdx.org/licenses/ for a complete list of license identifiers.license 'Apache 2.0'^^^^^^^^^^^^recipes/default.rb:9:1: R: [Correctable] Chef/Modernize/ExecuteAptUpdate: Use the apt_update resource instead of the execute resource to run an apt-get update package cache update.execute 'apt-get update' do^^^^^^^^^^^^^^^^^^^^^^^^^^^11 files inspected, 2 offenses detected, 2 offenses autocorrectable
# spec/spec_helper.rb does: require 'chefspec' and require 'chefspec/policyfile'require 'spec_helper'describe 'web::default' docontext 'on Ubuntu 22.04' doplatform 'ubuntu', '22.04' # fake node data from Fauxhai, no machine bootedit { is_expected.to install_apt_package('nginx') }it { is_expected.to enable_service('nginx') }it 'writes the upstream config locked down' dois_expected.to create_template('/etc/nginx/conf.d/upstream.conf').with(owner: 'root', group: 'www-data', mode: '0640', sensitive: true)endendcontext 'on Rocky Linux 9' doplatform 'rocky', '9'it { is_expected.to install_dnf_package('nginx') }endend
$ chef exec rspec
..F.Failures:1) web::default on Ubuntu 22.04 writes the upstream config locked downFailure/Error:is_expected.to create_template('/etc/nginx/conf.d/upstream.conf').with(owner: 'root', group: 'www-data', mode: '0640', sensitive: true)expected "template[/etc/nginx/conf.d/upstream.conf]" to have parameters:mode "0640", found nil# ./spec/unit/recipes/default_spec.rb:11:in `block (3 levels) in <top (required)>'Finished in 3.44 seconds (files took 6.21 seconds to load)4 examples, 1 failureFailed examples:rspec ./spec/unit/recipes/default_spec.rb:11 # web::default on Ubuntu 22.04 writes the upstream config locked down
The split here is worth being precise about. ChefSpec asserts on the resource collection, which is what Chef intends to do. InSpec asserts on the machine, which is what actually happened. Look at that rspec failure again: it is the same missing mode that Kitchen needed two minutes and a booted container to find, caught in three seconds with nothing booted at all. ChefSpec still cannot tell you whether nginx starts, and InSpec cannot tell you what the recipe meant to do on a platform you never booted, so you keep both. Be honest about cookstyle's limits too. It checks style, deprecations, and known-bad patterns. It will never tell you a config file is world-readable.
The same profile runs against production
The best thing about InSpec is that the profile is not tied to Kitchen at all. That same directory of controls runs against a production host over SSH (secure shell, the standard encrypted remote login), against a container, or against a cloud API, because the target is a command-line flag. Add an inspec.yml alongside the controls with a name, a version, and a supports block, and the directory becomes a versioned artifact: something you can publish, depend on from other profiles, and run on a schedule. The checklist that gates your pull request becomes the scan that watches production, which is where drift shows up. Drift is reality quietly walking away from what your code says, and it is almost always a person, in a hurry, at three in the morning.
$ CHEF_LICENSE=accept-no-persist inspec exec test/integration/default \-t ssh://[email protected] -i ~/.ssh/id_ed25519 --sudo \--reporter cli junit2:reports/web1.xml
Profile: web baseline (web-baseline)Version: 1.2.0Target: ssh://[email protected]:22Target ID: 9c2d4f61-8a03-4c7a-b1f0-7d5e2a1c3b44✔ web-01: nginx is installed, enabled at boot, and listening on 80✔ System Package nginx is expected to be installed✔ Service nginx is expected to be enabled✔ Service nginx is expected to be running✔ Port 80 is expected to be listening✔ Port 80 protocols is expected to include "tcp"✔ Processes nginx users is expected to include "www-data"× web-02: The upstream config holding the backend token is not world-readable (2 failed)✔ File /etc/nginx/conf.d/upstream.conf is expected to be file✔ File /etc/nginx/conf.d/upstream.conf owner is expected to eq "root"✔ File /etc/nginx/conf.d/upstream.conf group is expected to eq "www-data"× File /etc/nginx/conf.d/upstream.conf mode is expected to cmp == "0640"expected: 0640got: 0664(compared using `cmp` matcher)× File /etc/nginx/conf.d/upstream.conf is expected not to be more permissive than "0640"✔ web-03: The stock welcome vhost is gone✔ Command: `nginx -T 2>/dev/null` stdout is expected not to match /root\s+\/var\/www\/html/✔ Command: `nginx -T 2>/dev/null` stdout is expected not to match /root\s+\/usr\/share\/nginx\/html/Profile Summary: 2 successful controls, 1 control failure, 0 controls skippedTest Summary: 11 successful, 2 failures, 0 skipped$ echo $?100
That box is not a Kitchen container. Mode 0664 on a file Chef writes as 0640 means a human edited it by hand, most likely during an incident, and the next scheduled converge will put it back and erase the evidence. Running InSpec on a schedule is how you learn it happened at all. The extra bits are not cosmetic either: 0664 gives the whole group write access and everyone else read access, so a compromised deploy account can now rewrite the upstream token instead of merely reading it.
Exit code 100 means at least one test failed, 101 means controls were only skipped, and 0 means clean, so your pipeline can branch on the result without parsing text. The junit2 reporter writes the same run as XML your CI (continuous integration, the system that runs checks on every push) already knows how to draw, and you can emit several reporters in one invocation. Ready-made profiles exist too: the dev-sec baselines cover Linux, SSH, and nginx hardening without you writing a line of Ruby. If the Progress licence on the official binaries is a problem, CINC is the community rebuild of the same source code, where cinc-auditor is InSpec and cinc-client is Chef Infra Client. In a pipeline, export CHEF_LICENSE=accept-no-persist so a run never stalls at a prompt nobody is there to answer.
Be clear-eyed about what a green kitchen test proves. It proves your recipe logic holds together on a clean base image with the attributes you wrote into the suite. It does not prove the cookbook survives a machine carrying ten years of accumulated state, real secrets pulled from a data bag (Chef's server-side store for shared values), a firewall, SELinux in enforcing mode, or the node attributes your Chef Infra Server actually hands out. Containers are not machines either. Kernel modules, mounts, and reboots all behave differently inside one. Close the gap where you can. Point the provisioner at the same Policyfile.rb you ship. Add a second suite whose attributes mirror production instead of the defaults. Then run the identical InSpec profile against one canary node after its first real converge, and read that report before you let the change reach the other three thousand nine hundred and ninety-nine.
Try this
Run kitchen list on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: kitchen containers run privileged, on your kernel. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.