Securing the Chef server & clients

Keys, RBAC, and the pull model’s risks.

Every thirty minutes, a program running as root on every machine you own phones a central server and asks what it should be. Whatever answer comes back, it applies. Nobody approves it first. That is the pull model, and it is the whole security story of Chef compressed into three sentences.

The Chef Infra Server works like the key desk in a large office building. It cuts a key for each new machine, checks that key on every visit, and keeps a ledger of which doors each key opens. Securing Chef comes down to three questions about that desk. Who gets a key. What each key opens. And what you do about the fact that the agent sitting in every room trusts the desk completely. Encrypting the secrets themselves belongs to the data bags lesson, and pinning what a node runs belongs to the Policyfiles lesson. This one is identity, authorization, and blast radius.

What A Node Actually Proves

Nodes do not log in with a password. When Chef Infra Client starts, it reads a private RSA key (Rivest-Shamir-Adleman, the classic public-key algorithm) out of /etc/chef/client.pem and signs every request it sends. Signing is the digital version of a wax seal. The node runs a calculation that only the private half of the key can run, and the server checks the result against the public half it stored when the node enrolled. The secret never crosses the wire. There is no password to phish, reuse, or leave sitting in a log file.

The signature covers a good deal more than the address you asked for. Chef builds one canonical string out of the HTTP method, a hash of the path, a hash of the request body carried in the X-Ops-Content-Hash header, a timestamp carried in X-Ops-Timestamp, and the client name, then signs that whole string. So a captured request cannot be replayed an hour later, and it cannot be quietly edited in flight. The Chef Infra Server also refuses any request whose timestamp sits more than fifteen minutes away from its own clock. Hold on to that number. It explains an authentication failure that otherwise eats an afternoon.

# The node's identity, and the terms on which it trusts the server.
 
node_name          'web01.internal'
client_key         '/etc/chef/client.pem'      # private half, mode 0600, never leaves this box
chef_server_url    'https://chef.example.com/organizations/acme'
 
ssl_verify_mode    :verify_peer                # refuse a server we cannot verify (already the default)
trusted_certs_dir  '/etc/chef/trusted_certs'   # extra certificate authorities this node will accept
 
# This node is enrolled, so it has no business holding an enrolment key.
# Chef Infra Client only ever reads validation_key when client_key is missing,
# so the real fix is deleting /etc/chef/validation.pem. Pointing the setting at a
# path that does not exist stops anyone putting one back without you noticing.
validation_key     '/etc/chef/no-such-validator.pem'
 
log_level          :info
log_location       '/var/log/chef/client.log'

Two of those lines do security work rather than configuration work. ssl_verify_mode :verify_peer makes the node refuse a server whose certificate it cannot verify, and it is already the default, so the only reason to write it down is to stop a future colleague helpfully turning it off. The validation_key line is belt and braces. On an enrolled machine that file is dead weight, because Chef reads the setting only when client.pem is missing, so delete the file first and treat the config line as a tripwire.

sudo ls -l /etc/chef/
sudo openssl rsa -in /etc/chef/client.pem -noout -text | head -2

total 16
-rw------- 1 root root 1678 Jun  2 11:14 client.pem
-rw-r--r-- 1 root root  712 Jul 21 09:02 client.rb
-rw-r--r-- 1 root root  312 Jun  2 11:14 first-boot.json
drwxr-xr-x 2 root root 4096 Jun  2 11:14 trusted_certs
Private-Key: (2048 bit, 2 primes)
modulus:

One file, mode 0600, owned by root. That is the node's entire identity. Copy it and you are that node. Break it and the node cannot talk to the server at all, which looks like this.

sudo chef-client

Chef Infra Client, version 18.11.11
Patents: https://www.chef.io/patents
Infra Phase starting
 
================================================================================
Chef encountered an error attempting to load the node data for "web01.internal"
================================================================================
 
Authentication Error:
---------------------
Failed to authenticate to the chef server (http 401).
 
Server Response:
----------------
Invalid signature for user or client 'web01.internal'
 
Relevant Config Settings:
-------------------------
chef_server_url   "https://chef.example.com/organizations/acme"
node_name         "web01.internal"
client_key        "/etc/chef/client.pem"
 
If these settings are correct, your client_key may be invalid, or
you may have a chef user with the same client name as this node.
 
Running handlers:
[2026-07-21T09:44:02+00:00] ERROR: Running exception handlers
Running handlers complete
[2026-07-21T09:44:02+00:00] ERROR: Exception handlers complete
Infra Phase failed. 0 resources updated in 01 seconds
[2026-07-21T09:44:02+00:00] FATAL: Net::HTTPClientException: 401 "Unauthorized"

Two different problems produce exactly that block, and telling them apart fast is a skill worth having. Either the key on disk no longer matches the public key the server holds, because somebody rotated one side and not the other. Or two machines are claiming the same node_name, which is what happens when /etc/chef/client.pem survives into a machine image: every clone then authenticates as the same client, and they fight over a single node object, overwriting each other's attributes on every converge.

A third cause hides behind a different block. If the node's clock has drifted outside that fifteen minute window, the server answers with a body containing "Synchronize the clock on your host", and Chef Infra Client spots that phrase and replaces the whole Server Response section with a plain-English hint instead: the request failed because your clock has drifted by more than 15 minutes, sync it against an NTP time source. NTP is the Network Time Protocol, the background service that keeps machine clocks honest. Read the block you actually got before you touch any keys, because that one is a time fix, and a new key will fail the same way ten minutes later.

So strip /etc/chef/client.pem and /etc/chef/client.rb out of anything you snapshot. Bootstrap should mint a fresh identity on first boot, every single time.

Enrol Without Handing Out A Master Key

Older Chef deployments enrolled machines with a shared organization validator key, a file named something like acme-validator.pem that got copied onto every new box. The node used it once to register itself, then switched to its own key. It behaves exactly like a building contractor's master key: fine while the contractor is careful, awkward forever afterwards, because it opens everything and nobody can list every place it was copied to.

Validatorless bootstrap takes it out of the picture entirely. knife bootstrap connects to the target over SSH (Secure Shell, the encrypted remote-login protocol), creates the client and node objects on the server using your own workstation key, writes the resulting private key onto the node, installs Chef Infra Client, and runs it once. No shared secret is copied anywhere, and the enrolment is attributable to a named human rather than to a file that has been in circulation since 2019.

knife bootstrap 10.0.0.11 \
  --connection-user ubuntu \
  --ssh-identity-file ~/.ssh/bootstrap_ed25519 \
  --sudo \
  --node-name web01.internal \
  --policy-name web-server \
  --policy-group production \
  --bootstrap-version 18.11.11

Connecting to 10.0.0.11 using ssh
Creating new client for web01.internal
Creating new node for web01.internal
Bootstrapping 10.0.0.11
 [10.0.0.11] -----> Installing Chef Omnibus (18.11.11)
 [10.0.0.11] Chef Infra Client, version 18.11.11
 [10.0.0.11] Patents: https://www.chef.io/patents
 [10.0.0.11] Infra Phase starting
 [10.0.0.11] Using Policyfile 'web-server' at revision '5f1c9a3d0e7b41c8...'
 [10.0.0.11] Synchronizing cookbooks:
 [10.0.0.11] Converging 24 resources
 [10.0.0.11] Running handlers:
 [10.0.0.11] Running handlers complete
 [10.0.0.11] Infra Phase complete, 11/24 resources updated in 38 seconds

The two lines worth checking are "Creating new client" and "Creating new node". They appear only when knife is doing the registration itself with your credentials. If a validator key is in play, knife hands over and the node registers itself, and the run prints a very different line: "Creating a new client identity for web01.internal using the validator key." That sentence in a bootstrap log is your cue that a shared secret is still in circulation. The --policy-name and --policy-group pair is the Policyfile-native way to say what this machine runs, which is what you want on Chef 18. A plain --run-list still works and still pulls roles into the picture, and roles are the legacy path.

# Is the skeleton key still sitting in this organization?
knife client list | grep validator
 
knife client delete acme-validator --yes > /dev/null
knife client list | grep validator || echo 'no validator client in this org'

acme-validator
no validator client in this org

Rotate A Key Before You Have To

A client object on the server can hold several public keys at once, the way a hotel door accepts both the old card and the new one during a changeover week. That is what makes rotation survivable. Add the new key, roll it out, prove it works, then revoke the old one. Do it in the other order and you lock the node out, and a node locked out of the server stops converging quietly, which is the kind of failure nobody notices for six weeks.

knife client key list web01.internal
 
# Cut a second key that stops working on its own. The umask is not decoration:
# knife opens the file with a plain File.open, so your shell's default mode wins.
umask 077
knife client key create web01.internal \
  --key-name 2026-q3 \
  --expiration-date 2026-10-19T00:00:00Z \
  --file /tmp/web01-2026-q3.pem
 
ls -l /tmp/web01-2026-q3.pem
knife client key list web01.internal --with-details

default
Created key: 2026-q3
-rw------- 1 sre sre 1678 Jul 21 10:02 /tmp/web01-2026-q3.pem
default: https://chef.example.com/organizations/acme/clients/web01.internal/keys/default
2026-q3: https://chef.example.com/organizations/acme/clients/web01.internal/keys/2026-q3

The --expiration-date flag is the part teams skip and the part that changes behaviour. A key with no expiry is a decision you made once and never revisited. A key that stops working on 19 October 2026 turns rotation into something that either happens on time or fails loudly, and both of those beat a five-year-old credential nobody remembers issuing. Dates are ISO 8601, meaning year first with a trailing Z for UTC. Run the list with --with-details later and any expired key is marked as such, which makes it a fine thing to graph. The same subcommands exist for people, as knife user key create and knife user key list, and your admins deserve them more than your nodes do.

# On the node: swap the key in, then prove the server accepts it
sudo install -m 0600 -o root -g root /tmp/web01-2026-q3.pem /etc/chef/client.pem
sudo chef-client --once | tail -3

Running handlers:
Running handlers complete
Infra Phase complete, 0/24 resources updated in 06 seconds

Zero of twenty-four resources updated is what a healthy converge on an unchanged machine looks like, and the run got far enough to load the node object, which means the server accepted the new signature. Now, and only now, revoke the old key with knife client key delete web01.internal default. List the keys once more and you should see one name where there used to be two.

Least Privilege On The Server

Authorization on the Chef Infra Server is a filing room. An organization is the room. Containers are the labelled cabinets inside it: clients, containers, cookbooks, data, environments, groups, nodes, policies, policy_groups and roles. Groups are staff lists, and every node in the organization is added to the list called clients automatically the moment it registers. An access control list, usually shortened to ACL, is the sticky note on a cabinet, or on one individual file inside a cabinet, saying which staff lists may do what to it.

There are five permissions: create, read, update, delete, and grant. Four of them behave the way you expect. grant is permission to edit permissions, so handing out grant is handing out administrator with extra typing. The knife acl and knife group subcommands ship with Chef Workstation, and before you change anything, look at what you already have. This is the most useful command on the page.

knife acl show containers data -F json

{
  "create": {
    "actors": [],
    "groups": [
      "admins",
      "users"
    ],
    "users": [],
    "clients": []
  },
  "read": {
    "actors": [],
    "groups": [
      "admins",
      "users",
      "clients"
    ],
    "users": [],
    "clients": []
  },
  "update": {
    "actors": [],
    "groups": [
      "admins",
      "users"
    ],
    "users": [],
    "clients": []
  },
  "delete": {
    "actors": [],
    "groups": [
      "admins",
      "users"
    ],
    "users": [],
    "clients": []
  },
  "grant": {
    "actors": [],
    "groups": [
      "admins"
    ],
    "users": [],
    "clients": []
  }
}

Four keys under each permission, because knife asks the server for the granular view. actors is a compatibility leftover and stays empty; the names you care about land in groups, users and clients. Now read the second block again. The clients group holds read on the data container, and every node in the organization belongs to clients. Any machine that can authenticate may fetch any data bag in the org. The nodes container tells the same story with create and read, so any node can read every other node's attributes: internal hostnames, address maps, which box runs payments, and whatever a colleague once dropped into a plain attribute because it was faster than setting up encryption. One compromised web server reads the entire map of your estate.

The answer is two moves in this order. Encryption first, because in practice an access list is not what stands between a node and a secret. Then scoping: build narrow groups and point them at exactly the objects they need.

# On the Chef Infra Server itself:
chef-server-ctl org-user-add acme alice
 
# On your workstation. A person gets access by joining a group, never directly.
knife group create auditors
knife group add user alice auditors
knife group list

admins
auditors
billing-admins
clients
public_key_read_access
users

The five names you did not create are the defaults every organization ships with, and clients is the one to keep an eye on, because it grows by itself every time somebody bootstraps a machine. Notice that knife group add happily took a user. knife acl add will not, and that catches people out constantly.

# knife acl add MEMBER_TYPE MEMBER_NAME OBJECT_TYPE OBJECT_NAME PERMS
knife acl add user alice containers nodes read

FATAL: ERROR: To enforce best practice, knife-acl can only add a client or a group to an ACL.
FATAL:        See the knife-acl README for more information.

You cannot grant a permission to a named person at all. Put them in a group and grant the group, which is the habit you wanted anyway. So point the new group at the objects it needs instead.

knife acl add group auditors containers nodes read    # template for nodes created later
knife acl bulk add group auditors nodes '.*' read     # the 400 nodes already there

Adding 'auditors' to 'read' ACE of 'nodes'
The ACL of the following nodes will be modified:
 
app01.internal    db02.internal     web01.internal
cache01.internal  edge01.internal   web02.internal
 
Are you sure you want to modify the ACL of these nodes?? (Y/N) Y
Adding 'auditors' to 'read' ACE of 'app01.internal'
Adding 'auditors' to 'read' ACE of 'cache01.internal'
Adding 'auditors' to 'read' ACE of 'db02.internal'
Adding 'auditors' to 'read' ACE of 'edge01.internal'
Adding 'auditors' to 'read' ACE of 'web01.internal'
Adding 'auditors' to 'read' ACE of 'web02.internal'

The doubled question mark is knife-acl's, not a typo of mine. The bulk command lists everything it matched and waits for a yes before touching anything, which makes it a free preview; knife's global -y flag skips the prompt once you trust the match. It also flatly refuses to bulk-edit containers or groups, so you cannot flatten your whole authorization model with one careless regular expression.

Those two commands do genuinely different jobs, and this is the sharpest edge on the page. A container access list is a template. It governs listing, and it is stamped onto objects created after you set it. It never reaches backwards onto the four hundred node objects already sitting there. Set only the container and your auditor gets a list of names with nothing readable behind them, then files a bug against you. knife acl bulk add walks the existing objects and edits each one, matching the regular expression against object names, so '.*' means all of them and '^web' means the web tier. Check your work on a single object.

knife acl show nodes web01.internal -F json | jq '.read'

{
  "actors": [],
  "groups": [
    "admins",
    "users",
    "clients",
    "auditors"
  ],
  "users": [],
  "clients": [
    "web01.internal"
  ]
}

auditors is on the groups line, so the bulk pass landed. The node's own client name sits under clients, because whoever creates an object on the Chef Infra Server gets full rights to it, and web01 created its own node object during bootstrap.

The Server Is A Fleet-Wide Root Oracle

Now the uncomfortable part. Because nodes pull, the Chef Infra Server decides what root does on every machine you have, and so does anything that can write to that server: your build pipeline, the laptop of anyone in the admins group, the host itself. There is no per-change approval on the node side. Two clamps are worth the effort.

The first is verifying the server rather than merely reaching it. knife ssl check runs a real certificate validation from your workstation and tells you the truth about the result.

knife ssl check
knife ssl check https://chef-staging.example.com

Connecting to host chef.example.com:443
Successfully verified certificates from `chef.example.com'
 
Connecting to host chef-staging.example.com:443
ERROR: The SSL certificate of chef-staging.example.com could not be verified
Certificate issuer data: /C=US/ST=WA/L=Seattle/O=Acme/OU=Ops/CN=chef-staging.example.com
 
Configuration Info:
 
OpenSSL Configuration:
* Version: OpenSSL 3.0.15 3 Sep 2024
* Certificate file: /opt/chef-workstation/embedded/ssl/cert.pem
* Certificate directory: /opt/chef-workstation/embedded/ssl/certs
Chef Infra Client SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/home/sre/.chef/trusted_certs"
 
TO FIX THIS ERROR:
[...]

knife ssl fetch is the escape hatch, and it deserves suspicion. It downloads whatever certificate the host happens to present and writes it into trusted_certs_dir. That is trust on first use, the same bet you make the first time you SSH somewhere new and type yes. Once, on a laptop, over a network you trust, it is defensible. Baked into a provisioning script that runs on every new node forever, it is a standing invitation: whoever can answer for chef.example.com during that one window owns root on that machine permanently, and ssl_verify_mode :verify_peer will not save you, because you told the node to trust the attacker's certificate.

The second clamp is shrinking how much of each machine ends up readable by every other machine. Ohai, the tool that inventories the host at the start of every run, collects a great deal, and all of it is saved to the node object, which we established a moment ago that every client can read. You can filter what gets written back. These settings used to be called whitelists and blacklists; Chef 16 renamed them, and the names below are the current ones.

# Stop shipping the entire machine to a server that every node can read.
#   filesystem          every mount, device and UUID (unique volume label) on the box
#   network/interfaces  every address on every interface
#   packages            the full installed-package inventory
blocked_automatic_attributes %w{filesystem network/interfaces packages}
 
# Or invert it, and send only what your cookbooks and searches genuinely use.
# Pick one direction per attribute type, not both:
# allowed_automatic_attributes %w{fqdn os platform platform_version ipaddress}

Be precise about what this does. The list filters what is written back to the Chef Infra Server. It does not stop Ohai collecting the data, and recipes can still read node['filesystem'] during the same run that collected it. What changes is what sits on the server between runs, waiting for anyone with a client key to fetch it. The trade-off is real: knife search queries and cookbooks that look up another node's interfaces will stop finding what they expect, so block one category, converge one node, then check your searches before rolling it out.

# On the workstation, before the change
knife node show web01.internal -l -F json | wc -c
 
# ...ship the new client.rb, converge that node once, then look again
knife node show web01.internal -l -F json | wc -c
knife node show web01.internal -a filesystem

418442
26105
web01.internal:

Four hundred kilobytes down to twenty-six, and filesystem comes back empty because it is no longer on the server. That is a verification rather than a hope, and it is the shape of check you want after every hardening change in this lesson.

The last piece is the agent itself. Chef Infra Client 18 manages its own schedule with a built-in resource, so the thing that configures everything else gets configured the same way. Every time value here is a systemd time span written as a string rather than a number of seconds, and systemd is the service manager that starts and schedules things on a modern Linux box. Passing splay 300 as a bare integer fails property validation before anything reaches the machine.

chef_client_systemd_timer 'converge every 30 minutes' do
  interval            '30min'
  splay               '5min'   # random offset so 2000 nodes do not all arrive together
  delay_after_boot    '2min'   # let the network settle before the first run
  accept_chef_license true
end

systemctl list-timers 'chef-client*'
systemctl cat chef-client.service

NEXT                        LEFT       LAST                        PASSED   UNIT              ACTIVATES
Tue 2026-07-21 10:34:11 UTC 24min left Tue 2026-07-21 10:01:52 UTC 7min ago chef-client.timer chef-client.service
 
1 timers listed.
 
# /etc/systemd/system/chef-client.service
[Unit]
Description=Chef Infra Client periodic execution
After=network.target auditd.service
 
[Service]
Type=oneshot
ExecStart=/opt/chef/bin/chef-client --chef-license accept -c /etc/chef/client.rb
SuccessExitStatus=3
SuccessExitStatus=213
SuccessExitStatus=35
SuccessExitStatus=37
SuccessExitStatus=41
 
[Install]
WantedBy=multi-user.target

Look at what is missing. There is no User= line, and a system service without one runs as root, so this converges as root every thirty minutes on every machine, with no terminal and nobody watching. Worth knowing before you try to change it: the resource does expose a user property that defaults to root, but Chef 18 never writes it into the generated unit, so setting it to anything else has no effect. Read the unit file, not the property list. What you get in exchange is a fleet that repairs itself and cannot drift, which is a genuinely good deal. What you also get is one server able to run arbitrary code as root on two thousand machines, guarded by a key file and a table of permissions. Running CINC, the trademark-free community rebuild of Chef Infra, changes none of this beyond the names: cinc-client, cinc-server-ctl, and no license flag to accept.

Two commands are worth running before you close this page. knife acl show containers data -F json tells you whether every node in your organization can read every data bag you own. knife client list | grep validator tells you whether the skeleton key is still lying around. Both take a second, and both have a habit of returning something you did not expect.

Try this

Run sudo ls -l /etc/chef/ on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the validator is a skeleton key that never expires. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related