Nodes, run-lists & roles

What a node should become.

A run-list is the work order taped to a machine's door. It names what the machine should end up being, and the order the pieces go on in. It says nothing at all about how to get there. Chef Infra Client, the agent that runs on the machine itself, reads that order at the start of every run, pulls down the cookbooks it names, and pushes the box toward the state the order describes. A role is the same work order with a name on it, so you can hang it on two hundred doors at once instead of writing it out two hundred times.

This is the layer where cookbooks stop being code on your laptop and become a fleet. It is also the layer where one careless upload reconfigures four hundred machines. Knowing exactly how Chef turns a short list into an ordered set of recipes pays for itself the first time something goes sideways.

The Node Object and Its Run-List

The Chef Infra Server keeps one index card per machine, and that card is called the node object. It is a JSON document (JavaScript Object Notation, a plain text format for structured data) holding the machine's name, its environment, the facts Ohai reported, a record of what the last run resolved, and one field you genuinely manage by hand: run_list. Ohai is Chef's built-in inventory collector; it gathers hostname, IP addresses, platform and a few hundred other details at the start of every run. (If you use CINC, the free rebuild of Chef that ships without the trademarks, everything below is identical apart from the branding: the agent is called cinc-client.)

Run-list items come in two shapes. recipe[cookbook::recipe] names exactly one recipe. role[name] names a role, which is a run-list of its own. Order is literal and it matters: the recipe that adds an internal package repository has to sit above the recipe that installs from it, or the install fails. A bare recipe[frontend] is shorthand for recipe[frontend::default]. You rarely edit that JSON by hand. knife, the command line tool on your workstation that talks to the server, does it for you and writes the change straight through.

# What the Chef Infra Server knows about
knife node list
 
# Everything it knows about one machine
knife node show web-01.example.com

build-07.example.com
db-01.example.com
legacy-nfs-01.example.com
web-01.example.com
web-02.example.com
web-03.example.com
 
Node Name:   web-01.example.com
Environment: production
FQDN:        web-01.example.com
IP:          10.0.4.21
Run List:    role[base], recipe[frontend::apache]
Roles:       base
Recipes:     agent, agent::default, timesync, timesync::default, hardening::ssh, frontend::apache
Platform:    ubuntu 22.04
Tags:

Read the last few lines carefully. Run List is what you asked for. Roles and Recipes are what the previous run actually resolved and executed. They are related. They are not identical, and the gap between them is where most of the surprises live. The doubling in Recipes is deliberate, not a display bug: for every recipe ending in ::default, Chef records the short name too, so a search for agent and a search for agent::default both find this node.

# Append an item to the end of the list
knife node run_list add web-01.example.com 'recipe[myapp::deploy]'
 
# Place an item exactly, instead of at the end
knife node run_list add web-01.example.com 'recipe[repo::internal]' \
  --before 'recipe[frontend::apache]'
 
# Take one back off
knife node run_list remove web-01.example.com 'recipe[myapp::deploy]'

web-01.example.com:
  run_list:
    role[base]
    recipe[frontend::apache]
    recipe[myapp::deploy]
 
web-01.example.com:
  run_list:
    role[base]
    recipe[repo::internal]
    recipe[frontend::apache]
    recipe[myapp::deploy]
 
web-01.example.com:
  run_list:
    role[base]
    recipe[repo::internal]
    recipe[frontend::apache]

knife node run_list add appends to the end by default. --before ITEM and --after ITEM drop an item into a precise slot instead, and the item you name has to match an existing entry exactly. That matters more than it sounds. Append a repository recipe to the bottom of a list and it lands after everything that needed it. There is a third subcommand, knife node run_list set, which replaces the whole list rather than adding to it. It is the quickest way to delete role[base] from a production node without noticing, because it succeeds quietly and prints a perfectly reasonable looking result.

You can also hand a machine its work order at the moment it joins the fleet, so it knows what to become before anyone logs into it. Bootstrapping is that one-time step: knife installs the agent over SSH, creates the client and node records on the server, and kicks off the first run.

knife bootstrap ssh://[email protected] \
  --node-name web-03.example.com \
  --ssh-identity-file ~/.ssh/id_ed25519 \
  --sudo \
  --run-list 'role[base],recipe[frontend::apache]'

Connecting to 10.0.4.23
Creating new client for web-03.example.com
Creating new node for web-03.example.com
Bootstrapping 10.0.4.23
 [10.0.4.23] Installing Chef Infra Client 18
 [10.0.4.23] Thank you for installing Chef Infra Client!
 [10.0.4.23] Chef Infra Client, version 18.11.11
 [10.0.4.23] Patents: https://www.chef.io/patents
 [10.0.4.23] Infra Phase starting
 [10.0.4.23] Resolving cookbooks for run list: ["agent::default", "timesync::default", "hardening::ssh", "frontend::apache"]
 [10.0.4.23] Converging 41 resources
 [10.0.4.23] Infra Phase complete, 27/41 resources updated in 01 minutes 12 seconds

Roles: A Run-List With a Name

Copying the same six item list onto fifty nodes is how fleets drift apart. A role is the job description you write once and pin to every machine that does that job: a named, reusable run-list plus two optional blocks of attributes. Write it as a Ruby file (JSON works too), upload it, and from then on role[web] is a single item standing in for the whole list. Roles nest, so one small base role can sit inside every job specific role you write.

# roles/web.rb - the job description. Lives in git, applied from the pipeline.
name 'web'
description 'Public-facing web servers'
 
# Ordered. role[base] is spliced in right here, at this exact position.
run_list(
  'role[base]',
  'recipe[repo::internal]',
  'recipe[frontend::apache]', # our wrapper cookbook: the community apache2
                              # cookbook ships resources, not recipes
  'recipe[myapp::deploy]'
)
 
# Beats cookbook and environment defaults. Loses to node normal and to every override.
default_attributes(
  'frontend' => { 'listen_ports' => %w{80 443} }
)
 
# Beats ordinary cookbook overrides. Loses to environment overrides and to Ohai.
override_attributes(
  'frontend' => { 'keepalive' => 'On' }
)

Those two attribute blocks are where roles bite people. Chef stacks every attribute value in ten fixed layers, and a role owns exactly two of them. default_attributes sits above cookbook defaults and environment defaults, and below force_default, below anything set as normal on the node, and below every flavour of override. A value you set confidently in the role loses to a node attribute somebody stamped in during a bootstrap two years ago. override_attributes sits above ordinary cookbook overrides, and below environment overrides, below force_override, and below the automatic facts Ohai collects. Neither block is the top of the ladder and neither is the bottom (ch-attributes walks the whole order). If the value can live in a cookbook attribute file instead, put it there, where it is versioned alongside the code that reads it.

# Push the role to the Chef Infra Server
knife role from file roles/web.rb
 
# One item now stands for the whole list
knife node run_list set web-01.example.com 'role[web]'

Updated Role web!
 
web-01.example.com:
  run_list:
    role[web]

What Expansion Actually Does

At the start of every run the client walks the node's run-list and asks the server for each role by name, a plain HTTP request against /roles/NAME. The splicing then happens on the machine, not on the server. Every role[...] is replaced, in place, by that role's own run-list, and the replacement repeats until nothing but recipes are left. Three rules decide the result, and all three have caused real incidents.

First, splicing is positional. A nested role's items land exactly where the role reference sat, never at the front and never at the end. Second, everything is de-duplicated. A role that has already been applied is skipped, which is also what stops two roles that both pull in role[base] from looping forever. A recipe that shows up twice survives only at its first position, and the second occurrence disappears with no log line and no error. The one duplicate that does stop the run is a version clash: two items pinning different versions of the same recipe raise Chef::Exceptions::CookbookVersionConflict. Third, attributes from every role in the expansion are deep merged in expansion order, so a later role wins ties at the same precedence level. A role's attributes still apply even when every one of its recipes was de-duplicated away by an earlier occurrence.

The client prints the finished, flattened list on every single run. It is the line beginning Resolving cookbooks for run list:, and it is the most useful line in the whole log. Here is the base role, then the same node resolved in why-run mode, a dry run that works out everything it would do and changes nothing.

# roles/base.rb - every machine in the fleet carries this
name 'base'
description 'Baseline: agent schedule, time sync, SSH hardening'
 
run_list(
  'recipe[agent::default]',    # wraps the built-in chef_client_systemd_timer
                               # resource; the old community chef-client
                               # cookbook is retired
  'recipe[timesync::default]', # chrony on anything modern
  'recipe[hardening::ssh]'
)

# On the node itself: resolve and report, change nothing
sudo chef-client --why-run

Chef Infra Client, version 18.11.11
Patents: https://www.chef.io/patents
Infra Phase starting
Resolving cookbooks for run list: ["agent::default", "timesync::default", "hardening::ssh", "repo::internal", "frontend::apache", "myapp::deploy"]
Synchronizing cookbooks:
  - agent (2.4.0)
  - timesync (1.2.0)
  - hardening (3.1.0)
  - repo (1.4.0)
  - frontend (2.0.3)
  - apache2 (9.3.11)
  - myapp (0.4.2)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 31 resources
Recipe: hardening::ssh
  * template[/etc/ssh/sshd_config] action create (up to date)
  * service[ssh] action enable (up to date)
Recipe: myapp::deploy
  * apt_package[myapp] action install
    - install version 1.8.3 of package myapp
 
[2026-07-21T09:14:52+00:00] WARN: In why-run mode, so NOT performing node save.
 
Running handlers:
Running handlers complete
Infra Phase complete, 3/31 resources would have been updated

Proving What Actually Ran

knife node show web-01.example.com -r prints the assigned run-list with roles left folded up. You see role[web] and nothing underneath it. That is intent. To see the outcome, read the automatic attributes the client writes back after a successful run: roles (every role that got expanded), recipes (the flat ordered recipe list, plus the short form of every default recipe), and expanded_run_list (the same recipes written out in full, with @version appended to any that were pinned).

# Intent: roles are NOT expanded here
knife node show web-01.example.com -r
 
# Outcome: what the last successful run actually resolved
knife node show web-01.example.com -a recipes -a roles

web-01.example.com:
  run_list:
    role[web]
 
web-01.example.com:
  recipes:
    agent
    agent::default
    timesync
    timesync::default
    hardening::ssh
    repo::internal
    frontend::apache
    myapp::deploy
  roles:
    web
    base

The server indexes all of that, which turns it into fleet wide questions you can answer in one command. Two details will save you a bad afternoon. Colons are reserved characters in the query language, so hardening::ssh has to be written hardening\:\:ssh. And the singular and plural field names are different fields holding different truths. role:web and recipe:frontend match the node's run-list as written, which is intent, and they only see top level items. roles:web and recipes:hardening\:\:ssh match the automatic attributes the last successful run wrote back, which is outcome, and those do include everything a nested role dragged in. For a compliance question you always want the plural.

# Who is carrying the web job description?
knife search node 'roles:web' -i
 
# Who actually converged the SSH hardening recipe?
knife search node 'recipes:hardening\:\:ssh' -i
 
# The query that matters: who did not?
knife search node '(NOT recipes:hardening\:\:ssh)' -i
 
# And who has stopped checking in (silent for over an hour)?
knife status --hide-by-mins 60

3 items found
 
web-01.example.com
web-02.example.com
web-03.example.com
 
4 items found
 
db-01.example.com
web-01.example.com
web-02.example.com
web-03.example.com
 
2 items found
 
build-07.example.com
legacy-nfs-01.example.com
 
31 hours ago, db-01.example.com, db-01.example.com, 10.0.4.9, ubuntu 22.04.
216 hours ago, legacy-nfs-01.example.com, legacy-nfs-01.example.com, 10.0.3.4, centos 7.9.

The third query belongs in a runbook. "Which hosts are supposed to be hardened and did not run the hardening recipe" has an exact answer, and getting it costs one command instead of an afternoon with a spreadsheet. It catches drift nobody meant to cause: a node whose run-list was hand edited during an outage and never put back, a box bootstrapped without role[base] because someone copied the wrong command. The sneaky case is the one the search cannot see, which is why knife status is stapled to it. A node that stops converging keeps its old recipes attribute forever. Look at db-01: it sits in the hardened list, and it has not checked in for 31 hours, so it is reporting last Tuesday's configuration as though it were today's. Note also that knife always counts in hours once a node passes the hour mark, so 216 hours is nine days of silence.

When you want to try one recipe on one machine without changing what the node claims to be, override the run-list for a single run.

# Run once, ignore the interval, ignore the node's real run-list
sudo chef-client --once --override-runlist 'recipe[myapp::deploy]'

Chef Infra Client, version 18.11.11
Patents: https://www.chef.io/patents
Infra Phase starting
Resolving cookbooks for run list: ["myapp::deploy"]
Synchronizing cookbooks:
  - myapp (0.4.2)
  - frontend (2.0.3)
  - apache2 (9.3.11)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 9 resources
Recipe: myapp::deploy
  * apt_package[myapp] action install
    - install version 1.8.3 of package myapp
  * service[myapp] action restart
    - restart service service[myapp]
 
[2026-07-21T09:31:07+00:00] WARN: Skipping final node save because override_runlist was given
 
Running handlers:
Running handlers complete
Infra Phase complete, 2/9 resources updated in 08 seconds

Two things are true about that run. It converged myapp::deploy and nothing else, ignoring the node's real list entirely. And because the run-list was overridden, the client deliberately skips the final node save, which it tells you in plain language. So knife node show -a recipes still reports the previous run's list. Handy while you are testing. Worth remembering when you read node data as evidence during an investigation: recipes records the last ordinary run, not necessarily the last thing Chef executed on that host.

Staging a Role Change

Roles do have one built in concession to staged rollout: per environment run-lists. Instead of a single run_list, a role can carry env_run_lists, a map from environment name to its own ordered list. Nodes in staging get one list, nodes in production get another, and a change can bake for a week before you touch the production line.

# roles/web.rb - one role, a different list per environment
name 'web'
description 'Public-facing web servers'
 
# '_default' is mandatory. Every environment without its own
# entry falls back to it.
env_run_lists(
  '_default'   => ['role[base]', 'recipe[repo::internal]', 'recipe[frontend::apache]'],
  'staging'    => ['role[base]', 'recipe[repo::internal]', 'recipe[frontend::apache]', 'recipe[myapp::canary]'],
  'production' => ['role[base]', 'recipe[repo::internal]', 'recipe[frontend::apache]', 'recipe[myapp::deploy]']
)

That _default key really is mandatory. Leave it out and the upload dies with Chef::Exceptions::InvalidEnvironmentRunListSpecification: _default key is required in env_run_lists. There is a tidy reason behind the rule. Inside a role, run_list and env_run_lists['_default'] are the same object, so the plain run_list in the earlier version of this file was always the _default entry wearing a different name. Without it, nodes in an unlisted environment would quietly receive nothing from this role and drift into an unconfigured state that nothing alerts on. If you would rather edit one environment's slice on the live role than rewrite the file, knife role env_run_list add web staging 'recipe[myapp::canary]' does that in place.

# Push the role from the pipeline
knife role from file roles/web.rb

Updated Role web!

Then go and prove the split from the node side, on a box that lives in staging. Same role on the node, different flattened list in the log, and that log line is the only evidence that settles the argument.

# On web-05.example.com, whose chef_environment is 'staging'
sudo chef-client --once

Chef Infra Client, version 18.11.11
Patents: https://www.chef.io/patents
Infra Phase starting
Resolving cookbooks for run list: ["agent::default", "timesync::default", "hardening::ssh", "repo::internal", "frontend::apache", "myapp::canary"]
Synchronizing cookbooks:
  - agent (2.4.0)
  - timesync (1.2.0)
  - hardening (3.1.0)
  - repo (1.4.0)
  - frontend (2.0.3)
  - apache2 (9.3.11)
  - myapp (0.4.2)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 28 resources
Recipe: myapp::canary
  * apt_package[myapp] action install
    - install version 1.9.0 of package myapp
 
Running handlers:
Running handlers complete
Infra Phase complete, 1/28 resources updated in 11 seconds

That is as far as roles go, and the honest answer is that it is not very far. The attribute blocks stay global across every environment. There is still no version number and no checksum. Promoting staging to production is still a person editing a file and pushing it to a live object that every node reads on its next run. When you want the run-list itself to be an artifact you build, sign off on and ship, that is a Policyfile (ch-policyfiles). It locks the run-list together with the exact resolved cookbook versions into one lockfile, which you push to a named policy group, so promotion becomes publishing a build instead of mutating shared state. A Policyfile run-list is a flat list of recipes and composition happens through include_policy, which retires the expansion surprises above along with the roles themselves.

Until you get there, three habits carry most of the weight. Keep every role file in git and apply it only from the pipeline. Run knife search node 'roles:web' -i before you touch a role, so the blast radius is a number on your screen instead of a guess. Then prove the change on one host with sudo chef-client --once and read the Resolving cookbooks for run list: line with your own eyes, before the interval reads it for four hundred machines.

Try this

Run knife node list on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: why-run is a preview, not a promise. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related