Chef, Puppet & Salt

The config-management landscape.

A fleet of servers behaves a lot like a block of rented flats. You can keep them all in order two ways. Drive round with a master key and fix each one yourself, or give every flat a caretaker who reads the house rules every half hour and quietly puts things back where they belong. Both work. They fail differently, and they hand an intruder very different opportunities.

That is configuration management in one sentence: write down what a machine should look like, then make the machine match. Ansible is the master key. It is agentless, meaning nothing gets installed on the machines it manages, and push-based: a control node opens an SSH connection (Secure Shell, the encrypted remote-login protocol) and runs the tasks, at the moment you tell it to. Puppet, Chef and Salt hire caretakers. Each machine runs a daemon (a background program that keeps running with nobody logged in) holding the right to change anything on that box as root, the all-powerful administrator account on Linux. Puppet and Chef wake on a timer and go and fetch their instructions. Salt keeps a line open to its master and waits to be told. Ansible wins most new work. The other three run enormous estates, and sooner or later you inherit one.

Puppet: the node asks, the server answers

Puppet is the oldest of the four and the strictest about the loop, which makes it the clearest one to learn from. The agent behaves like a tenant filling in a form. A tool called Facter collects facts about the host (operating system, hostname, addresses, whether it is a virtual machine) and posts them to the Puppet server over TCP (Transmission Control Protocol, the reliable connection layer nearly everything on a network runs over) port 8140. The server takes those facts, reads your manifests (files written in Puppet's own declarative language) and compiles a catalog: a list, built for this one node, of every resource that should exist and the state it should be in. The agent applies the catalog, reports back, sleeps, and does the whole thing again thirty minutes later.

Trust between agent and server is not a password. It is a client certificate, which works like an ID card stamped by the building manager. The node makes a key pair, sends a signing request, and an operator signs it using the Puppet server's own certificate authority (the component that issues those ID cards and vouches for them). That signature is what says "this box really is web-01, so hand it web-01's catalog." Hold on to that, because it is where the interesting attacks live.

$ sudo puppet config print server runinterval certname
$ sudo puppet agent --test --noop ; echo "exit=$?"

server = puppet.acme.internal
runinterval = 1800
certname = web-01.acme.internal
 
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for web-01.acme.internal
Info: Applying configuration version '1784626863'
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/sshd_config]/content:
--- /etc/ssh/sshd_config	2026-07-21 09:14:22.113000000 +0000
+++ /tmp/puppet-file20260721-4471-1qk3ba	2026-07-21 09:41:03.221000000 +0000
@@ -30,7 +30,7 @@
 # Authentication:
 
 #LoginGraceTime 2m
-PermitRootLogin yes
+PermitRootLogin no
 #StrictModes yes
 #MaxAuthTries 6
 #MaxSessions 10
 
Notice: /Stage[main]/Ssh::Server/File[/etc/ssh/sshd_config]/content: current_value '{md5}0f2b8c9e4a17d3b0c5e81a2f6d94b073', should be '{md5}a41c9d02f7be6538cc1470ad9e3b25f1' (noop)
Notice: Class[Ssh::Server]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 3.42 seconds
exit=2

noop is short for "no operation": work everything out, change nothing, print the diff. --test switches on the bundle of flags you want while debugging, and one of them is --detailed-exitcodes. That is the part to memorise. Exit 0 means the node already matched its catalog. Exit 2 means Puppet would have changed something. 4 means something failed, 6 means both. A timer that runs puppet agent --test --noop and alerts on exit code 2 is a drift detector you can build this afternoon. Schedule it away from the agent's own run, though, because both take the same lock file and the second one to arrive is turned away without doing any work.

Every run report is a drift alarm

Puppet separates two reasons a resource changed, and hardly anyone uses the distinction. A shop doing a stock check knows the difference in its bones. Head office sends a new price list and you re-label the shelves: that is an intentional change, and you were expecting it. The price list is identical to last week but the labels have moved anyway: someone has been in the shop overnight. Puppet calls the second kind a corrective change. The catalog did not change, the machine drifted away from it, and the agent hauled it back.

$ sudo puppet config print lastrunfile
$ grep -A9 '^resources:' /opt/puppetlabs/puppet/public/last_run_summary.yaml

/opt/puppetlabs/puppet/public/last_run_summary.yaml
resources:
  changed: 2
  corrective_change: 2
  failed: 0
  failed_to_restart: 0
  out_of_sync: 2
  restarted: 1
  scheduled: 0
  skipped: 0
  total: 84

That summary sits under public on purpose. It is world readable so monitoring can collect it without root. And corrective_change: 2 next to an unchanged catalog version is a flat statement about your environment: something modified this machine outside the pipeline in the last half hour. That is exactly what an intruder does. Append a key to /root/.ssh/authorized_keys (the file listing which keys may log in as root with no password), drop a file into /etc/sudoers.d/ to hand an account full rights, flip PermitRootLogin back to yes, install a systemd unit (systemd is the program that starts and supervises services on most Linux systems) that calls home every minute. If Puppet manages those paths, the next run puts them back and stamps a corrective change into the report. One caveat worth knowing before you rely on it: Puppet ignores files it was never told about, so a brand new file in /etc/sudoers.d/ only disappears if you manage that directory with recurse => true and purge => true. Ship reports to PuppetDB (Puppet's store of node facts, catalogs and reports) or into your log pipeline, then alert on corrective changes to security-relevant resources. The agent already does the work. The detection costs you nothing.

Chef: the same loop, written in Ruby

Chef runs the same loop with different vocabulary and a real programming language underneath. Configuration lives in recipes, written in a dialect of Ruby (a general-purpose scripting language), and recipes are grouped into cookbooks. Every node carries a run list naming which recipes apply to it. The client program is chef-client. It talks to the Chef Infra Server over HTTPS (HTTP Secure, the encrypted web protocol) on port 443, and signs every request with a private key kept on the node. Its dry run is spelled --why-run.

chef_server_url  'https://chef.acme.internal/organizations/acme'
node_name        'web-01.acme.internal'
client_key       '/etc/chef/client.pem'       # this node's identity
validation_key   '/etc/chef/validation.pem'   # org-wide bootstrap key
ssl_verify_mode  :verify_peer                 # never set this to :verify_none
log_level        :info

Two files named in there decide how bad a stolen box gets. client.pem is this node's identity, a PEM file (a plain-text wrapper for keys and certificates) holding its private key. Copy it and you are web-01 as far as the server is concerned: pull down every cookbook the organisation has and read the data bags (Chef's shared key/value store, very often used to hand out credentials), because the default permissions let any registered client read both. validation.pem is worse. It is the organisation-wide bootstrap key that lets a brand new machine register itself, a spare fob that cuts new keys, and it should be deleted the moment the first run finishes. Go and look.

$ sudo ls -l /etc/chef/client.pem \
              /etc/chef/encrypted_data_bag_secret \
              /etc/chef/validation.pem

-rw------- 1 root root 1674 Mar  3 11:02 /etc/chef/client.pem
-rw-r--r-- 1 root root  684 Mar  3 11:02 /etc/chef/encrypted_data_bag_secret
-rw-r--r-- 1 root root 1679 Mar  3 11:02 /etc/chef/validation.pem

Three lines, two findings. validation.pem survived the build and it is readable by everyone on the box, so any local account can register a fresh node under a name of its own choosing and then read the whole cookbook tree and the data bags with it. encrypted_data_bag_secret is mode 0644 as well, and that is the shared key which unlocks Chef's encrypted data bags. On its own that second finding is only half a break-in, because you still need the ciphertext and the server will not hand it to an unregistered client. Chained to the first, it is the whole thing: register, fetch, decrypt. Ten seconds a host, and it turns something up depressingly often.

Salt: a live bus, and the scar it left

Salt is the fast one. Puppet and Chef poll, which is the caretaker reading the noticeboard on the half hour, so an urgent change waits for the next round. Salt leaves the radio on instead. Every minion (Salt's word for its agent) holds an open subscription to the master on TCP port 4505 and sends results back on 4506, over ZeroMQ (a lightweight messaging library built for speed). Fire one command and thousands of machines answer in seconds. Desired state is written in YAML (a plain-text data format designed to be easy for people to read), per-node facts are called grains, and secrets come from the pillar. Salt runs agentless over SSH too, with salt-ssh.

$ sudo salt-key -L
$ sudo salt '*' test.ping

Accepted Keys:
web-01
web-02
Denied Keys:
Unaccepted Keys:
build-99
Rejected Keys:
 
web-01:
    True
web-02:
    True

Both of those run on the master, and trust works the way Puppet's does. The minion offers its public key, an operator accepts it with salt-key -a web-03, and only an accepted minion gets work. Read that Unaccepted Keys list every time you audit. A machine called build-99 that you did not build, sitting there waiting, is either sloppy provisioning or somebody finding out whether your master hands configuration to anyone who asks.

Salt also carries the sharpest security lesson in this family. At the end of April 2020, two flaws in the salt-master process went public. CVE-2020-11651 (CVE stands for Common Vulnerabilities and Exposures, the public catalogue of known security flaws) was an authentication bypass: anyone who could reach port 4506 could ask the master for its root key and be handed it. CVE-2020-11652 was a directory traversal in the same service, which let an attacker read and write files outside the directory the code intended. Chain the two and you had code execution as root on the master, and the master runs commands on every minion it owns. Mass exploitation began within days, hitting LineageOS, Ghost and DigiCert among others. Fixes shipped in 2019.2.4 and 3000.2.

Patching was the easy part. The lesson that outlives it is that a configuration master is the best foothold in any network, because it already holds the right to run anything anywhere. From a node, that relationship is visible on the wire.

$ sudo ss -tnp state established '( dport = :4505 or dport = :4506 )'

Recv-Q Send-Q Local Address:Port   Peer Address:Port  Process
0      0      10.0.3.21:47812     10.0.1.10:4505     users:(("salt-minion",pid=812,fd=17))
0      0      10.0.3.21:53106     10.0.1.10:4506     users:(("salt-minion",pid=812,fd=21))

One long-lived outbound connection on 4505, with 4506 beside it while a job returns. Puppet's equivalent is a short burst to 8140 once a run interval. Chef's is an HTTPS request to 443. Those are fingerprints. Use them to write firewall rules, and to hunt for an agent that nobody admits to installing.

Where the keys live

Strip the vocabulary away and all four tools are the same object: one box that can run arbitrary code as root across your whole estate. The real questions are which box, and which way the connection travels. With push, the control node holds the SSH private keys and the sudo rights for everything, so taking it hands over the fleet. The attacker still has to run something to use it, and every node has to accept inbound SSH, which is a door you then have to defend. With pull, the master holds the code, and taking it hands over the fleet on its own, because every node comes and asks for instructions and runs them as root without a human pressing anything. Nodes need no inbound ports at all: better for exposure, worse for how fast a bad change spreads.

So classify the server properly. It belongs in the tier of a domain controller or a cloud organisation root account: separate credentials, admin access through a bastion (one hardened jump host that every admin session has to pass through), its own logging, and changes only through reviewed pull requests. If a merge to the manifests repository can reach production without a second pair of eyes, one compromised developer account is root on every server you own within thirty minutes, and the only trace is a change event that looks like normal work. Whichever tool you inherit, the first thing to learn is how to make it change nothing.

# Puppet: compile the catalog, apply nothing, show diffs. Exit 2 = drift found.
sudo puppet agent --test --noop
 
# Chef: same idea, Chef's spelling.
sudo chef-client --why-run
 
# Salt, from the master: test=True is the dry run.
sudo salt 'web-*' state.apply test=True
 
# Ansible, from the control node: --check plans, --diff shows file changes.
ansible-playbook -i inventory.ini site.yml --check --diff

Choosing one, and what changed recently

For new work, pick Ansible unless you have a specific reason not to. Nothing to install, no certificates to sign, and a colleague can read a playbook in an afternoon. Puppet and Chef earn their keep on estates of thousands of long-lived servers, where continuous enforcement and per-node reporting are the whole point. Salt earns its keep when you need answers from ten thousand machines before your coffee goes cold.

Two things changed since that advice was first written. Ownership moved, for a start. Chef went to Progress in 2020, SaltStack to VMware the same year and on to Broadcom in 2023, Puppet to Perforce in 2022. Perforce then announced in late 2024 that Puppet development would move behind a private repository with commercially licensed builds, and the community forked the open version as OpenVox. Chef's binaries from Progress carry a commercial licence too, with a free rebuild called CINC. Read the licence on the packages you install, because "the project is open source" and "the vendor's build is free to use" stopped being the same sentence.

The other change is direction of travel. Configuring machines in place is losing ground to immutable infrastructure: bake a finished image with Packer or a Dockerfile, deploy it, and change anything by building a new image and replacing the machine. Nothing mutates, so nothing drifts, and a host that gets replaced on every deploy gives an attacker's foothold a very short life. Configuration management is not going away, because databases, network gear, engineer laptops and twelve-year-old application servers do not get rebuilt on a Tuesday afternoon. For anything new on cloud, though, the shape is usually Terraform provisioning immutable images rather than Ansible configuring mutable servers.

Next time you land on an unfamiliar Linux box and need to know who is actually in charge of it, two commands settle the question.

$ systemctl list-units --type=service --state=running | grep -E 'puppet|chef|salt'
$ ls -d /etc/puppetlabs /etc/chef /etc/salt 2>/dev/null

  puppet.service      loaded active running Puppet agent
/etc/puppetlabs

A running puppet.service with /etc/puppetlabs on disk means your hand-edit to /etc/ssh/sshd_config has a half-life of under thirty minutes. That box is not yours to edit. The fix belongs in the repository the server compiles from, and if you make it on the machine instead, you get to watch it vanish while you are still on the call.

Try this

Run sudo puppet config print server runinterval certname on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: self-healing is not incident response. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related