What Chef is & the pull model

Client/server, agent-based convergence.

A restaurant chain with two hundred branches has two ways to keep every kitchen matching the current standard. Head office can send an inspector round to each branch, which works right up until the inspector is off sick and half the branches drift. Or every branch keeps the standard pinned to the wall, and the manager walks the kitchen against it every half hour, fixing whatever has slipped since the last walk. Chef picks the second one. Every machine you manage runs a small program that wakes on a timer, calls head office, asks what it is supposed to look like, and repairs itself on the spot.

That program is Chef Infra Client, and you invoke it as chef-client. Head office is the Chef Infra Server. The standard pinned to the wall is a set of cookbooks: versioned bundles of Ruby code, files and templates that you write on a third machine, your workstation, and upload to the server. The shape has a name. It is the pull model, because the managed machine starts the conversation and pulls its instructions down. The server never dials out to a node. It stores, it answers questions, and it records what each node reported the last time it called in.

The second idea to get straight is convergence. A recipe is closer to a shopping list than to driving directions. Directions run in order and strand you if you start halfway through. A shopping list you check against the cupboard, buying only what is missing. Everything you declare in a recipe (a package installed, a file holding exactly this content, a service switched on at boot) inspects the machine first and acts only where the machine disagrees. The node converges toward the state you described, then stops. Run the same recipe five minutes later and it should do nothing at all, a property called idempotence (running it a second time changes nothing). That property is what turns a half-hourly program running as root (root, the administrator account that can change anything on the box) into a safe idea rather than a terrifying one.

Three machines, one direction of travel

You will touch three kinds of machine. Your workstation carries the Chef Workstation tools: chef for cookbook and policy work, knife for talking to the server, kitchen for spinning up throwaway virtual machines to test on, inspec for writing checks that prove a box really ended up hardened, and cookstyle for linting your Ruby (linting, an automated read-through that flags sloppy or deprecated code before a human sees it). The Chef Infra Server is a PostgreSQL database, a search index and a file store sitting behind an HTTPS API (application programming interface, the machine-readable front door a program talks to instead of a web page; HTTPS is the same encrypted web protocol your browser uses). It holds cookbooks and node records, and it answers searches. It does not execute a single line of your recipe code. The nodes do that, locally, as root.

Direction of travel matters more than it sounds. Every connection is outbound: node to server, port 443. No inbound firewall rule pointing at a node, no jump host (a bastion machine you log into first because the target cannot be reached directly), no extra port listening on the machine being managed. A node behind NAT (network address translation, where a pile of machines share one outward address and cannot be dialled from outside) behaves exactly like one with a public address. A push tool has to reach in. Chef nodes reach out. The cost is a credential sitting on every single box, which is the security story at the end of this lesson.

# read by chef-client at the start of every run
chef_server_url   'https://chef.acme.internal/organizations/acme'
node_name         'web01.acme.internal'
client_key        '/etc/chef/client.pem'
 
policy_name       'web'      # which Policyfile this node runs
policy_group      'prod'     # which published revision of it
 
ssl_verify_mode   :verify_peer
trusted_certs_dir '/etc/chef/trusted_certs'
file_cache_path   '/var/chef/cache'
log_location      '/var/log/chef/client.log'

Read that file as the node's entire worldview. chef_server_url ends in /organizations/acme because one Chef Infra Server hosts many organizations, each with its own cookbooks, nodes and users, walled off from the rest. node_name is the identity this machine authenticates as, and it defaults to the FQDN (fully qualified domain name, the machine's full name in DNS, the domain name system that turns names into addresses). client_key points at the private key the node signs its requests with. policy_name and policy_group say which Policyfile this node runs and which shelf to take it from: you lock the exact cookbook versions with chef install, publish them to a group with chef push prod, and every node in that group then runs the identical set. Roles and environments do a similar job and are the legacy path. ssl_verify_mode :verify_peer means the client checks the server's TLS certificate (TLS, transport layer security, the encryption underneath HTTPS) and refuses to talk to anything it cannot verify.

How a node gets an identity

A fresh machine is a stranger at the door: no key, no record on the server. knife bootstrap hands it a badge. From your workstation it opens an SSH session (secure shell, the encrypted remote login), installs the agent, asks the server to create a client record for this machine using your own credentials, writes the resulting key onto the box, and runs the agent once.

$ knife bootstrap web01.acme.internal \
    --connection-user ubuntu \
    --ssh-identity-file ~/.ssh/acme.pem \
    --sudo \
    --node-name web01.acme.internal \
    --policy-name web \
    --policy-group prod

Connecting to web01.acme.internal using ssh
Creating new client for web01.acme.internal
Creating new node for web01.acme.internal
Bootstrapping web01.acme.internal
web01.acme.internal -----> Installing Chef Infra Client...
web01.acme.internal Getting information for chef stable 18 for ubuntu...
web01.acme.internal downloading https://omnitruck.chef.io/stable/chef/metadata?v=18&p=ubuntu&pv=24.04&m=x86_64
web01.acme.internal version 18.4.12
web01.acme.internal Thank you for installing Chef Infra Client!
web01.acme.internal Starting the first Chef Infra Client run...
web01.acme.internal Chef Infra Client, version 18.4.12
web01.acme.internal Infra Phase starting
web01.acme.internal Using policy 'web' at revision '0b7d5c31e4a9f26d38b70c15ae9284fd631c07b2a48e5d9061f3ac72be40851d'
web01.acme.internal Resolving cookbooks for run list: []
web01.acme.internal Converging 0 resources
web01.acme.internal Infra Phase complete, 0/0 resources updated in 03 seconds

Two lines there carry the security of the whole model. "Creating new client" is the server minting an identity for this machine, and it happened because your knife credentials were allowed to ask for one. The private key comes back to knife and lands on the node as /etc/chef/client.pem, mode 0600, readable by root and nobody else. From then on the node signs every request with that key and the server checks the signature against the matching public half it kept, the way a lock recognises a key without ever holding a copy of it. The run itself did nothing because the web policy had an empty run list at that moment (the run list being the ordered set of recipes a node is meant to apply), which is deliberate here: identity first, configuration second.

The older path to identity is the validation key, a single organization-wide credential copied onto unregistered machines so they can create their own client record. It still works and you should not use it, because anyone holding a copy can register a node under any name they like and start pulling cookbooks. After a bootstrap, check that /etc/chef/validation.pem is not sitting on the box.

What one run actually does

Now give the policy something to say. Here is a small recipe from a cookbook called baseline. It installs fail2ban (a service that watches auth logs and blocks addresses after repeated failed logins), writes an SSH hardening file, and makes sure the SSH daemon is switched on and running. Three declarations. No commands.

package 'fail2ban' do
  action :install
end
 
template '/etc/ssh/sshd_config.d/50-hardening.conf' do
  source 'sshd_hardening.erb'
  owner  'root'
  group  'root'
  mode   '0600'
  variables(allow_groups: node['baseline']['ssh_allow_groups'])
  notifies :restart, 'service[ssh]', :delayed
end
 
service 'ssh' do
  action [:enable, :start]
end

Add that cookbook to the web policy, lock the versions with chef install, publish with chef push prod, then trigger the agent by hand rather than waiting for the timer.

$ sudo chef-client

Chef Infra Client, version 18.4.12
Patents: https://www.chef.io/patents
Infra Phase starting
Using policy 'web' at revision '2f8c4a1d9b7e3056c1a84f2d6b90e7c35a1f8d24906bce7f13a5d820c4e69b17'
Resolving cookbooks for run list: ["baseline::default"]
Synchronizing cookbooks:
  - baseline (0.4.1)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 3 resources
Recipe: baseline::default
  * apt_package[fail2ban] action install
    - install version 1.0.2-3ubuntu0.1 of package fail2ban
  * template[/etc/ssh/sshd_config.d/50-hardening.conf] action create
    - create new file /etc/ssh/sshd_config.d/50-hardening.conf
    - update content in file /etc/ssh/sshd_config.d/50-hardening.conf from none to 4c9f1e
    - change mode from '' to '0600'
  * service[ssh] action enable (up to date)
  * service[ssh] action start (up to date)
  * service[ssh] action restart
    - restart service service[ssh]
 
Running handlers:
Running handlers complete
Infra Phase complete, 3/3 resources updated in 18 seconds

Walk the phases in order, because each one fails differently. Resolving works out the exact cookbook versions this node is entitled to, which with a Policyfile is one pinned revision rather than whatever is newest. Synchronizing downloads the cookbook files, and only the ones whose checksums (short fingerprints of the file contents) are missing from /var/chef/cache, so a run with no code change moves almost no data. Compiling reads your Ruby from top to bottom and should touch nothing on the machine: it builds an ordered list of resources, the individual things you declared. Converging is the part that acts, walking that list and asking each resource whether reality already matches. apt_package[fail2ban] had work to do. service[ssh] was already enabled and already running, so it reported "up to date" twice and did nothing. The restart at the bottom is the template's delayed notification firing after everything else has finished.

That compile step is where Chef parts company with the tools that describe everything in YAML (a plain-text format for structured data, readable by people and deliberately not a programming language). A recipe is real Ruby with a DSL (domain specific language, a small vocabulary bolted onto a general language for one job) layered on top, so you can loop over a list of users or branch on the platform. The temptation that arrives with the power is writing procedure instead of state: shelling out through execute blocks, or doing real work during compile where nothing checks whether it was needed. Both break idempotence. A resource that acts on every run destroys the signal the next section rests on. Use a real resource wherever one exists, keep genuine logic in custom resources and libraries, and read an execute block as an admission that something is missing.

The second run is the one that proves it

# nothing edited on the box, nothing changed in the cookbook
$ sudo chef-client

Chef Infra Client, version 18.4.12
Patents: https://www.chef.io/patents
Infra Phase starting
Using policy 'web' at revision '2f8c4a1d9b7e3056c1a84f2d6b90e7c35a1f8d24906bce7f13a5d820c4e69b17'
Resolving cookbooks for run list: ["baseline::default"]
Synchronizing cookbooks:
  - baseline (0.4.1)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 3 resources
Recipe: baseline::default
  * apt_package[fail2ban] action install (up to date)
  * template[/etc/ssh/sshd_config.d/50-hardening.conf] action create (up to date)
  * service[ssh] action enable (up to date)
  * service[ssh] action start (up to date)
 
Running handlers:
Running handlers complete
Infra Phase complete, 0/3 resources updated in 04 seconds

0/3 is the number to care about. The machine already matched the code, so the run cost four seconds of checking and wrote nothing. A recipe that reports updates every single time is broken rather than busy, and it will bury the updates that mean something. Once quiet runs are normal, any resource that reports a change is telling you the machine moved away from the code since the last run. That is an alarm worth wiring up.

Watch what that buys you. At 02:40 someone with root on web01 edits the hardening file to turn root logins back on, chasing a stuck deploy. Nobody writes it down. Twenty minutes later the agent runs again, whether that is the timer or somebody's hand.

# 03:01, the first run after somebody edited that file by hand
$ sudo chef-client

Chef Infra Client, version 18.4.12
Patents: https://www.chef.io/patents
Infra Phase starting
Using policy 'web' at revision '2f8c4a1d9b7e3056c1a84f2d6b90e7c35a1f8d24906bce7f13a5d820c4e69b17'
Resolving cookbooks for run list: ["baseline::default"]
Synchronizing cookbooks:
  - baseline (0.4.1)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 3 resources
Recipe: baseline::default
  * apt_package[fail2ban] action install (up to date)
  * template[/etc/ssh/sshd_config.d/50-hardening.conf] action create
    - update content in file /etc/ssh/sshd_config.d/50-hardening.conf from 8b31f7 to 4c9f1e
    --- /etc/ssh/sshd_config.d/50-hardening.conf	2026-07-21 02:41:09.412000000 +0000
    +++ /etc/ssh/sshd_config.d/.chef-50-hardening20260721-3182-1kx8gq.conf	2026-07-21 03:01:14.880000000 +0000
    @@ -1,5 +1,5 @@
     # Managed by Chef. Local edits are reverted on the next run.
    -PermitRootLogin yes
    +PermitRootLogin no
     PasswordAuthentication no
     KbdInteractiveAuthentication no
     AllowGroups sre deploy
  * service[ssh] action enable (up to date)
  * service[ssh] action start (up to date)
  * service[ssh] action restart
    - restart service service[ssh]
 
Running handlers:
Running handlers complete
Infra Phase complete, 2/3 resources updated in 09 seconds

Drift (the slow slide away from the standard, one undocumented edit at a time) was detected and repaired in a single pass, with a unified diff in the run log naming the exact lines that moved, and you bought no extra product to get it. The exposure window is bounded by the interval: worst case that weakened config stood for about half an hour. Two of three resources updated, because the template rewrote the file and the service was restarted by the notification. Be honest about the other edge. Chef overwrote a human's emergency change without asking, because the cookbook is the source of truth by design. If the 02:40 edit was right, it belonged in the cookbook before 03:01, or it dies quietly.

Who fires the run, and how often

Nothing so far says what wakes the agent. Chef Infra Client can run as a daemon (a program that stays resident in the background), but the current answer on Linux is a systemd timer, systemd being the thing that starts and supervises services on every mainstream distribution now. Chef Infra Client 18 ships a resource that writes the timer unit for you. Put it in the baseline cookbook and the fleet manages its own schedule.

chef_client_systemd_timer 'Run Chef Infra Client every 30 minutes' do
  interval            '30min'
  splay               '5min'
  delay_after_boot    '1min'
  accept_chef_license true
  action :add
end

$ systemctl list-timers chef-client.timer

NEXT                        LEFT   LAST                        PASSED  UNIT              ACTIVATES
Tue 2026-07-21 09:41:07 UTC 18min  Tue 2026-07-21 09:11:07 UTC 11min   chef-client.timer chef-client.service
 
1 timers listed.

splay is the property people delete first and regret second. It becomes RandomizedDelaySec in the timer unit, shifting each node's start by a random amount up to five minutes, so a fleet that all booted together does not hit the server in the same second and flatten it. Staggered shift starts, so the whole factory is not queuing at one door. accept_chef_license matters for a different reason. Chef Infra Client 18 wants its licence agreement accepted on the first run, and an unattended timer has nobody to answer the prompt, so the run fails and that node quietly stops converging. Set it in the resource, or put CHEF_LICENSE=accept in the unit's environment.

When you need a change now rather than within the half hour, you go and get it. knife ssh 'name:web*' 'sudo chef-client' runs the agent on every node matching a search. Notice what that admits. The pull model gives you no ordering across machines. If a change has to land on the database before the web tier, or one node at a time behind a load balancer, a timer cannot express that and you are driving from outside again. Chef Workstation also ships chef-run for genuine one-offs against a single host, pushing over SSH with no server involved.

The failure mode is silence

A push run tells you loudly when a host is unreachable, because it was on the list and it did not answer. A pull fleet fails the other way round, like a night watchman who stops phoning in. A node that stops calling produces no error anywhere. It sits there holding whatever configuration it had on the day it went quiet, missing every change you have shipped since, and looking perfectly healthy from the outside. So ask the server who has gone silent.

$ knife status --hide-by-mins 60

2 days ago, web07.acme.internal, ubuntu 24.04, web07.acme.internal, 10.24.8.17.
9 days ago, db02.acme.internal, ubuntu 24.04, db02.acme.internal, 10.24.9.4.

--hide-by-mins 60 hides every node that checked in during the last hour, so what remains is your problem list. Those two boxes have applied nothing recently: web07 for two days, db02 for nine. A completed run always saves the node object (the server's record of that machine, holding its attributes, its policy and the time it last checked in) back to the server, so a stale timestamp means runs are not finishing, not that there was nothing to do. Check in this order: the timer is disabled or its unit is failing, the node cannot reach the server, the licence was never accepted, the client key no longer matches what the server holds. Then turn that query into an alert. A node nobody watches is a node whose hardening quietly expired.

Every box carries a key

The pull model puts a credential on every machine, so be precise about what that credential reaches. /etc/chef/client.pem authenticates as one node. Under the default permissions of a Chef organization that is enough to read every cookbook in the organization including ones this node never runs, to read any data bag that has not been encrypted (a data bag is a small file of shared settings kept on the server in JSON, a plain-text data format, and it works like the fleet's shared address book), and to write to its own node record.

That last one is the sharp edge. Node attributes (the facts a machine reports about itself, gathered by Ohai and by your recipes) flow back up to the server, and other machines read them. A load balancer recipe that searches for web nodes and writes their addresses into a backend pool is trusting whatever those nodes said about themselves. Root on one unimportant node therefore becomes attacker-controlled content in a config file on a machine the attacker never touched. Treat node attributes as claims made by a host, not as facts. Set anything security-relevant as an override attribute in the cookbook or the Policyfile, because override precedence beats the normal attributes a node is able to save about itself.

The honest trade-off

The comparison people reach for is Ansible, the inspector from the opening: no agent on the host, work pushed over SSH from a control node, nothing happening at all unless a person or a pipeline starts it. Pull earns its keep on estates that are large, long-lived, and full of machines you did not personally start. Nothing to schedule, no inbound access, no control node holding keys to the whole fleet, and a machine that was offline for a week fixes itself when it comes back, without anyone noticing it was gone. The bill is real. You run an agent as root on every host, a Ruby runtime and a few hundred megabytes under /opt/chef, and that agent is a program that fetches code over the network and executes it. You run a server too: database, search index, backups, certificates, and somebody on call for all of it.

One claim you will see repeated deserves correcting. A node that cannot reach the server does not carry on converging from its cache. It holds the state it last applied and retries at the next interval, so a broken Chef Infra Server gives you a fleet that quietly stops changing rather than a fleet that breaks. Two escape hatches are worth knowing early. chef-client --local-mode runs the same cookbooks with no server at all, using a throwaway in-memory stand-in called Chef Zero, which is how you test on your laptop and bake machine images. And if the licensing on Progress's builds does not suit you (a licence agreement accepted on first run, plus commercial terms worth reading before you scale), CINC is a rebuild of the same Apache-2.0 source with the trademarks stripped out. cinc-client speaks the same API to the same server and runs the same cookbooks.

Try this

Run sudo chef-client on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: never turn off certificate verification. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related