The chef-client run & convergence

Compile, converge, idempotency.

You read the whole recipe card before you touch a pan. Every ingredient, every step, so you know what you are signing up for. Then you cook. And you do not re-salt a soup that already tastes right; you taste first, and add salt only if it needs it. A chef-client run works in exactly those two passes over your code. The passes are called compile and converge, and the taste-first habit has a name of its own: idempotency (run the same thing twice and the second run changes nothing).

Getting this straight matters more here than with a push tool like Ansible, because of who is holding the steering wheel. Chef Infra Client is an agent, a program that lives on the box and starts itself on a timer (a systemd timer these days, created by the built-in chef_client_systemd_timer resource). It wakes up, takes inventory of the machine with Ohai (Chef's fact gatherer: hostname, platform, network, memory, disks), pulls whatever cookbooks its run-list names, and enforces them as root with nobody watching. A run-list is the ordered set of recipes this machine is supposed to have. If you cannot say which of your code runs in which pass, you cannot say which lines execute before your hardening resources have done a single thing, and you cannot say what a preview flag really protects you from. Both of those become security questions before the end of this lesson.

Two passes: build the list, then walk it

Compile is Chef reading. Picture a picker walking a warehouse with an order form, writing down what to fetch and touching nothing on the shelves. Chef loads the cookbooks for the run-list in a fixed order (libraries first, then any Ohai plugins the cookbooks ship, then attribute files, then custom resources and providers, then old-style definitions, and last of all recipes) and evaluates the Ruby in each recipe from the first line to the last. A resource block you wrote is not carried out there. It is turned into an object, filled in with the property values you gave it, and pushed onto an ordered in-memory list called the resource collection. include_recipe splices another recipe's resources into that same list at the exact spot where you called it. When compile ends, Chef holds a complete plan and has changed nothing you declared.

Read that wording again: nothing you declared. Plain Ruby is not a declaration. A backtick shell-out, a ::File.read, an HTTP call to a secrets service, a .run_action on a resource, all of it fires on the spot during compile, while the machine is still in its old state. Hold onto that. It is the single biggest source of Chef bugs, and the reason preview mode is thinner than it looks.

Converge is Chef doing. It walks the collection front to back and hands each resource to a provider, the platform-specific code that knows how to make that one kind of thing true (apt or dnf for packages, systemd for services). A provider behaves like a decorator who checks the wall before opening the paint: its first job is to look, not act. It loads the current state of that thing on this machine. Is the package installed, and at what version? Does the file exist with that content, owner and mode? Is the service enabled? Then it compares. Same as declared? It prints (up to date) and moves on. Different? It performs the smallest action that closes the gap, and prints exactly what it did. Delayed notifications pile up during this pass and fire in one batch at the end of it, deduplicated, which is how a template that genuinely changed reloads a service once in the same run. After the last resource, Chef saves the node object (the server's record of this machine's facts, attributes and run-list) back to the Chef Infra Server, and only then do report and exception handlers run.

A real run, line by line

Here is a four-resource recipe. Read it as a plan rather than a script: install the package, own the drop-in directory, render the config, keep the service enabled and running, and reload it if (and only if) that config changed. In production you would never hand a run-list to the command line like this, because the run-list and the exact cookbook versions come from the node's Policyfile, a single file that names the run-list and pins every cookbook version. chef install resolves it into Policyfile.lock.json, and chef push publishes that lock to a policy group, so what compiles is precisely the versions you pinned. Roles and environments used to do this job and are now legacy. On a workstation, though, the command line is the fastest way to watch a run happen.

package 'nginx'
 
directory '/etc/nginx/conf.d' do
  owner 'root'
  group 'root'
  mode  '0755'
end
 
template '/etc/nginx/conf.d/myapp.conf' do
  source 'myapp.conf.erb'      # ERB = Embedded Ruby, Chef's templating format
  mode   '0644'
  notifies :reload, 'service[nginx]', :delayed
end
 
service 'nginx' do
  action [:enable, :start]
end

# run from the directory that holds cookbooks/
$ sudo chef-client --local-mode --override-runlist 'recipe[myapp::default]'
 
# the same thing in short flags; -z starts chef-zero, a throwaway in-memory Chef server on this host
#   sudo chef-client -z -o 'recipe[myapp::default]'
# a brand new host also wants --chef-license accept the first time
# add -l debug (long form --log_level debug) to watch cookbook loading and provider decisions

Chef Infra Client, version 18.4.12
Patents: https://www.chef.io/patents
Infra Phase starting
Resolving cookbooks for run list: ["myapp::default"]
Synchronizing cookbooks:
  - myapp (0.1.0)
Installing cookbook gem dependencies:
Compiling cookbooks...
Loading Chef InSpec profile files:
Loading Chef InSpec input files:
Loading Chef InSpec waiver files:
Converging 4 resources
Recipe: myapp::default
  * apt_package[nginx] action install
    - install version 1.18.0-6ubuntu14.4 of package nginx
  * directory[/etc/nginx/conf.d] action create (up to date)
  * template[/etc/nginx/conf.d/myapp.conf] action create
    - create new file /etc/nginx/conf.d/myapp.conf
    - update content in file /etc/nginx/conf.d/myapp.conf from none to 8fd21c
    --- /etc/nginx/conf.d/myapp.conf  2026-07-21 09:14:02.118441000 +0000
    +++ /etc/nginx/conf.d/.chef-myapp20260721-3312-1qk9ru.conf  2026-07-21 09:14:02.114441000 +0000
    @@ -0,0 +1,5 @@
    +server {
    +  listen 8080;
    +  server_name web01.acme.internal;
    +  root /srv/myapp/public;
    +}
    - change mode from '' to '0644'
  * service[nginx] action enable (up to date)
  * service[nginx] action start (up to date)
  * service[nginx] action reload
    - reload service service[nginx]
 
[2026-07-21T09:14:13+00:00] WARN: Skipping final node save because override_runlist was given
 
Running handlers:
Running handlers complete
Infra Phase complete, 3/6 resources updated in 11 seconds

Four things in that output are worth knowing by heart. Converging 4 resources is the size of the collection compile built, which tells you the plan was complete before anything moved. The (up to date) lines are idempotency showing its work: the directory already existed with the right mode, because installing the package created it moments earlier in the same pass, and Ubuntu's nginx package enables and starts the service for you, so Chef left all three alone. The unified diff is Chef stating exactly what it changed inside a file, which is the artifact you want in an audit trail. And the reload sits at the very bottom, after every other resource, because it arrived as a delayed notification rather than as a line in the recipe.

The last line is the one to read first when you are triaging at 3am. Three actions did work, out of six that Chef checked. Six does not match the four in Converging 4 resources, and that gap is not a bug. The tally counts actions, not resource blocks. The service block carries two actions (:enable and :start), and the notified :reload is a third action on the same resource, so one block put three entries into the count. The WARN line above it is not an error either: it is Chef telling you that a run driven by --override-runlist deliberately does not write the node object back to the server. That detail comes back to bite in a minute.

The second run should be boring

# run it again immediately, with nothing else touching the box in between
$ sudo chef-client -z -o 'recipe[myapp::default]' | sed -n '/^Converging/,$p'

Converging 4 resources
Recipe: myapp::default
  * apt_package[nginx] action install (up to date)
  * directory[/etc/nginx/conf.d] action create (up to date)
  * template[/etc/nginx/conf.d/myapp.conf] action create (up to date)
  * service[nginx] action enable (up to date)
  * service[nginx] action start (up to date)
 
[2026-07-21T09:15:41+00:00] WARN: Skipping final node save because override_runlist was given
 
Running handlers:
Running handlers complete
Infra Phase complete, 0/5 resources updated in 04 seconds

Zero out of five. That is the promise of configuration management in one line: the first run did the work, the second found nothing left to do. Two details in that number repay a look. The total dropped from six to five because the reload does not exist this time, since a delayed notification is only queued when the notifying resource actually changed, and the template did not. And every resource reported (up to date) after inspecting the machine, not after consulting a cache of what Chef did last time. Chef re-derives current state on every run. That is exactly why it corrects drift, meaning the machine quietly wandering away from what your code says, even drift it did not cause.

That tally is a free drift detector

A settled node prints 0/N every time. So a non-zero number on a node nobody deployed to is a signal: something changed the box between runs and Chef put it back. Sometimes that is a package upgrade nudging a config file. Sometimes it is a colleague editing sshd_config over SSH at midnight. Sometimes it is someone who has no business being there at all. Chef fixes it quietly either way, and unless you are shipping run data off the host, the only record is a log line on the very machine that got tampered with.

The mirror image of that signal is a node that stops reporting at all. knife, the command-line tool your workstation uses to talk to the Chef Infra Server, has a roll-call for exactly this. knife status prints every node with how long ago it last checked in, and a check-in is the moment a successful run saved the node object back to the server.

$ knife status --run-list

9 days ago, db03, db03.acme.internal, 10.0.3.7, ubuntu 20.04, run_list: recipe[base], recipe[postgres].
19 minutes ago, web02, web02.acme.internal, 10.0.2.12, ubuntu 22.04, run_list: recipe[base], recipe[myapp].
14 minutes ago, web01, web01.acme.internal, 10.0.2.11, ubuntu 22.04, run_list: recipe[base], recipe[myapp].

Nine days is not a node that is behaving itself. It is a node that is dead, firewalled off, or running with its agent stopped, and stopping chef-client is the standard way both tired engineers and intruders keep a hand-made change alive. Now the wrinkle that WARN line was hinting at: a run started with --override-runlist (-o) never saves the node object, so ad-hoc runs like the ones above leave the check-in timestamp untouched. Use -o for testing, and read nothing into a timestamp it never refreshed.

Why-run mode, and how far it protects you

The --why-run flag (short form -W) is Chef's dry run: a walkthrough with a clipboard, not a rehearsal. The converge pass still walks every resource, and every provider still inspects current state, but instead of acting, each one reports what it would have done, with the word Would in front of every line it prints. You get the same diffs, a different final tally, and no node save. Here it is against a host where somebody hand-edited the listen port.

$ sudo chef-client -z --why-run -o 'recipe[myapp::default]' | sed -n '/^Converging/,$p'

Converging 4 resources
Recipe: myapp::default
  * apt_package[nginx] action install (up to date)
  * directory[/etc/nginx/conf.d] action create (up to date)
  * template[/etc/nginx/conf.d/myapp.conf] action create
    - Would update content in file /etc/nginx/conf.d/myapp.conf from 3c0a91 to 8fd21c
    --- /etc/nginx/conf.d/myapp.conf  2026-07-21 03:02:55.402118000 +0000
    +++ /etc/nginx/conf.d/.chef-myapp20260721-9931-6t2wqe.conf  2026-07-21 09:41:18.220118000 +0000
    @@ -1,5 +1,5 @@
     server {
    -  listen 8081;
    +  listen 8080;
       server_name web01.acme.internal;
       root /srv/myapp/public;
     }
  * service[nginx] action enable (up to date)
  * service[nginx] action start (up to date)
  * service[nginx] action reload
    - Would reload service service[nginx]
 
[2026-07-21T09:41:18+00:00] WARN: Skipping final node save because override_runlist was given
 
Running handlers:
Running handlers complete
Infra Phase complete, 2/6 resources would have been updated

Two things about that preview surprise people. The reload shows up even here, because why-run still marks the template as updated internally, so delayed notifications still queue and still get reported. And there is no node save at all, for two reasons stacked on top of each other: why-run never writes the node back, and the -o flag would not have either.

Now the limits, and you have to hold both in your head. First, why-run is best-effort, provider by provider. A resource whose behaviour depends on something an earlier resource would have done can report nonsense or fail outright, because the earlier thing never happened. An execute block reports the command it would run, which tells you intent, not consequence. Second, and this is the part that matters to defenders, why-run does nothing whatsoever about the compile pass.

Where this bites you on a fresh host

Here is the classic failure, and it usually reaches production because it worked fine on the machine where it was written. The recipe drops an API key file, then reads that file back to feed a config template. Before you copy it, note that keeping a key in a node attribute is its own problem: node attributes are stored on the Chef Infra Server in the clear, and any node holding a valid client key can search for them. Read this one as a bug demo, not as a secrets pattern.

directory '/etc/myapp'
 
file '/etc/myapp/api.key' do
  content node['myapp']['api_key']
  mode '0600'
  sensitive true
end
 
# WRONG: plain Ruby, so this runs during compile
api_key = ::File.read('/etc/myapp/api.key').chomp
 
template '/etc/myapp/config.toml' do
  source 'config.toml.erb'
  variables(api_key: api_key)
  mode '0640'
end

$ sudo chef-client -z -o 'recipe[myapp::secret]' 2>&1 | sed -n '/^Compiling/,$p'

Compiling cookbooks...
 
================================================================================
Recipe Compile Error in /root/.chef/local-mode-cache/cache/cookbooks/myapp/recipes/secret.rb
================================================================================
 
Errno::ENOENT
-------------
No such file or directory @ rb_sysopen - /etc/myapp/api.key
 
Cookbook Trace: (most recent call first)
----------------------------------------
  /root/.chef/local-mode-cache/cache/cookbooks/myapp/recipes/secret.rb:10:in `from_file'
 
Relevant File Content:
----------------------
/root/.chef/local-mode-cache/cache/cookbooks/myapp/recipes/secret.rb:
 
  3:  file '/etc/myapp/api.key' do
  4:    content node['myapp']['api_key']
  5:    mode '0600'
  6:    sensitive true
  7:  end
  8:
  9:  # WRONG: plain Ruby, so this runs during compile
 10>>  api_key = ::File.read('/etc/myapp/api.key').chomp
 11:
 12:  template '/etc/myapp/config.toml' do
 
System Info:
------------
chef_version=18.4.12
platform=ubuntu
platform_version=22.04
ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
program_name=/opt/chef/bin/chef-client
executable=/opt/chef/bin/chef-client
 
Running handlers:
[2026-07-21T09:31:07+00:00] ERROR: Running exception handlers
Running handlers complete
[2026-07-21T09:31:07+00:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 02 seconds
[2026-07-21T09:31:07+00:00] FATAL: Stacktrace dumped to /root/.chef/local-mode-cache/cache/chef-stacktrace.out
[2026-07-21T09:31:07+00:00] FATAL: ---------------------------------------------------------------------------------------
[2026-07-21T09:31:07+00:00] FATAL: PLEASE PROVIDE THE CONTENTS OF THE stacktrace.out FILE (above) IF YOU FILE A BUG REPORT
[2026-07-21T09:31:07+00:00] FATAL: ---------------------------------------------------------------------------------------
[2026-07-21T09:31:07+00:00] FATAL: Errno::ENOENT: No such file or directory @ rb_sysopen - /etc/myapp/api.key

Notice what Chef called it. A Recipe Compile Error, raised before the converge pass ever started, so not one resource ran. Line 10 tried to read a file that line 3 promises to create, and line 3 has created nothing yet. The path in the trace is the local-mode cache copy rather than your working tree, because even a local run synchronises cookbooks into a cache first. On your laptop, or on any host where an earlier run already left that file behind, this recipe works perfectly, which is exactly how it survives review and then dies on the first fresh node. Reordering the recipe does not save you, because both lines are read during compile and only one of them waits.

Pushing work into the converge pass

Two tools move work out of compile. lazy { } wraps a property value in a note that says "ask me again later", so Chef works the value out at the moment that resource converges instead of when it is built. ruby_block is a resource whose body is Ruby, so it takes its turn in the collection like everything else, and node.run_state is a scratch-pad hash that lives for exactly one run, handy for handing a value from one resource to another. You need both here. The ruby_block reads the file at the right time, and the template's variables property still has to be lazy, because property values are assigned during compile no matter what runs later. If only this one template wanted the value, you could drop the ruby_block and put the read straight inside the lazy block. Keep the ruby_block when several resources want the same value.

directory '/etc/myapp'
 
file '/etc/myapp/api.key' do
  content node['myapp']['api_key']
  mode '0600'
  sensitive true            # keeps the contents out of the run log and the report handler
end
 
ruby_block 'read api key' do
  block do
    # ::File with the leading colons is Ruby's own File class. Inside a custom
    # resource or a library, bare File resolves to Chef::Resource::File instead,
    # so write ::File everywhere and never have to remember where you are.
    node.run_state['api_key'] = ::File.read('/etc/myapp/api.key').chomp
  end
end
 
template '/etc/myapp/config.toml' do
  source 'config.toml.erb'
  variables(lazy { { api_key: node.run_state['api_key'] } })
  mode '0640'
  sensitive true
end

# same fresh host, nothing pre-created this time
$ sudo chef-client -z -o 'recipe[myapp::secret]' | sed -n '/^Converging/,$p'

Converging 4 resources
Recipe: myapp::secret
  * directory[/etc/myapp] action create
    - create new directory /etc/myapp
  * file[/etc/myapp/api.key] action create
    - create new file /etc/myapp/api.key
    - update content in file /etc/myapp/api.key from none to 6b86b2
    (suppressed sensitive resource)
    - change mode from '' to '0600'
  * ruby_block[read api key] action run
    - execute the ruby block read api key
  * template[/etc/myapp/config.toml] action create
    - create new file /etc/myapp/config.toml
    - update content in file /etc/myapp/config.toml from none to c2f4a8
    (suppressed sensitive resource)
    - change mode from '' to '0640'
 
[2026-07-21T09:52:04+00:00] WARN: Skipping final node save because override_runlist was given
 
Running handlers:
Running handlers complete
Infra Phase complete, 4/4 resources updated in 03 seconds

sensitive true earns its place on both resources. Without it, Chef prints the full unified diff of every file it changes into the run log and ships that same diff to whatever report handler you have wired up, so the first run of a key file publishes the key straight into your logging stack, where it is indexed, replicated and kept for a year. With it you get the checksum line and (suppressed sensitive resource), and you still get told the resource changed. That is all the drift signal you actually needed.

Prove it converges to zero

The check that catches non-idempotent code is cheap to the point of embarrassment. Run it twice, demand a zero. Any resource that reports updated on the second run is doing work on every run, which usually means an execute with no guard, a template rendering a timestamp, or content computed fresh each time. Across a thousand nodes that is a thousand needless service restarts a day, and it wrecks the drift signal, because you can no longer tell a real change from your own noise.

$ sudo chef-client -z -o 'recipe[myapp::default]' > /dev/null        # run 1: do the work
$ sudo chef-client -z -o 'recipe[myapp::default]' 2>&1 | tee /tmp/run2.log | tail -n 1
$ grep -qE 'complete, 0/[0-9]+ resources updated' /tmp/run2.log \
    && echo 'idempotent' \
    || { echo 'NOT idempotent: something acts on every run'; exit 1; }

Infra Phase complete, 0/5 resources updated in 04 seconds
idempotent

Test Kitchen, the tool that spins up a disposable virtual machine or container, runs your cookbook on it and destroys it afterwards, gives you the same gate: kitchen converge twice against the same instance, then kitchen verify to run your InSpec checks against the converged box. InSpec is Chef's language for writing "is this machine actually like this?" tests, and running it after the converge confirms the state you wanted is the state you got. Watch the detail that kitchen test converges only once, so add the second converge yourself if you want idempotency enforced in continuous integration rather than discovered in production. And if you are on CINC, the community rebuild of Chef from the same source without the trademarks or the licence prompt, every command in this lesson is identical with cinc-client in place of chef-client.

Try this

Run sudo chef-client --local-mode --override-runlist 'recipe[myapp::default]' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a converge overwrites your evidence. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related