The tools & workstation setup

chef-client, knife, Workstation.

Chef is three programs living on three different machines, and treating it as one program is the fastest way to stay confused for a week. Your laptop gets Chef Workstation, the bench where you write and test cookbooks (a cookbook is a folder of Chef code describing how a machine should be set up). Somewhere in the middle sits the Chef Infra Server, a records office holding the authoritative copy of every cookbook, every machine's stored data, and who is allowed to touch what. On each managed machine runs chef-client, a small agent that wakes on a schedule, asks the office what this machine is supposed to look like, and makes it true. That last step has a name you will see everywhere: a converge. This lesson builds the bench and wires it to the office. What the office stores, and what the agent does once a run starts, are the lessons on either side of this one.

knife is the remote control sitting on that bench. It is a CLI tool (command-line interface, meaning you type commands instead of clicking) and almost every knife command is a single signed HTTPS request (HTTP carried inside an encrypted connection) to the Chef Infra Server. It does not log into your servers. It does not scan your network. The one exception is knife bootstrap, which opens an SSH session (Secure Shell, the encrypted remote-login protocol), and we get to that at the end. Hold onto that fact. Most knife failures people bring you are identity or trust problems wearing a networking costume.

What the Workstation package actually contains

Chef Workstation is one download that drops an entire toolbox into /opt/chef-workstation, bringing its own Ruby so it never argues with the Ruby your operating system already has. Inside the box: the chef command (generators and Policyfile handling), knife (the server remote control), a full copy of Chef Infra Client 18 so you can converge your own laptop or a throwaway container, Test Kitchen (spins up disposable virtual machines or containers and runs your cookbook on them for real), Chef InSpec (a testing tool that asserts what is actually true on a machine, rather than what you meant to configure), Cookstyle (a linter, meaning a program that reads your code and complains about it, tuned to catch Chef syntax that has been deprecated), and chef-run (fires a single change at a host over SSH with no server involved at all).

It replaced ChefDK, the old Chef Development Kit, completely. ChefDK is end of life, and any blog post telling you to install it is stale. Never run both. Each ships its own Ruby and each drops shims into /usr/bin, so whichever one wins the PATH race silently decides which Chef Infra Client version your local test run uses. That produces bugs with no sensible explanation: the same recipe passing on your machine and failing on your colleague's.

# macOS / Linux: the Omnitruck installer, stable channel, version pinned
curl -L https://omnitruck.chef.io/install.sh -o install.sh
less install.sh                       # read it: the next line runs it as root
sudo bash install.sh -P chef-workstation -c stable -v 24.12.1073
 
# Windows (PowerShell, via Chocolatey)
choco install chef-workstation --version 24.12.1073

ubuntu 24.04 x86_64
Getting information for chef-workstation stable 24.12.1073 for ubuntu...
downloading https://omnitruck.chef.io/stable/chef-workstation/metadata?v=24.12.1073&p=ubuntu&pv=24.04&m=x86_64
  to file /tmp/install.sh.3821/metadata.txt
trying wget...
sha1	98e4eed6c037582c22bb0e2fd7e27ad0fd4e823b
sha256	7ea1d81f7afcb7e6b6293a5d2b7130725533e29ce9aeeaf6c99a2da4850b4a14
url	https://packages.chef.io/files/stable/chef-workstation/24.12.1073/ubuntu/22.04/chef-workstation_24.12.1073-1_amd64.deb
version	24.12.1073
downloaded metadata file looks valid...
downloading https://packages.chef.io/files/stable/chef-workstation/24.12.1073/ubuntu/22.04/chef-workstation_24.12.1073-1_amd64.deb
  to file /tmp/install.sh.3821/chef-workstation_24.12.1073-1_amd64.deb
trying wget...
Comparing checksum with sha256sum...
Installing Chef Workstation 24.12.1073
installing with dpkg...
Selecting previously unselected package chef-workstation.
(Reading database ... 74213 files and directories currently installed.)
Preparing to unpack .../chef-workstation_24.12.1073-1_amd64.deb ...
Unpacking chef-workstation (24.12.1073-1) ...
Setting up chef-workstation (24.12.1073-1) ...
Thank you for installing Chef Workstation!

Look at the url line in that output. You asked for Ubuntu 24.04 and Omnitruck handed back the Ubuntu 22.04 package. That is deliberate, not a bug: Chef maps a newer platform release onto the nearest build it actually ships, and the 22.04 binary runs fine on 24.04. Knowing that saves you a support ticket the first time somebody spots it.

Every Chef install guide on the internet shows you curl piped straight into sudo bash. Look at the deal you are accepting: root on your workstation, handed to whatever bytes come back from a URL, unread. The script does compare the downloaded package against a SHA-256 checksum (a short fingerprint computed from the file's contents, where changing one byte of the file changes the fingerprint completely), and you can watch it do that above. But it fetches that checksum from the same host over the same connection, so it catches a corrupted download and not a poisoned source. Reading the script first costs ten seconds. If you would rather handle the package yourself, the Omnitruck metadata endpoint hands you the exact download URL and checksum, and you fetch and verify on your own terms.

$ curl -s 'https://omnitruck.chef.io/stable/chef-workstation/metadata?p=ubuntu&pv=24.04&m=x86_64&v=24.12.1073'

sha1	98e4eed6c037582c22bb0e2fd7e27ad0fd4e823b
sha256	7ea1d81f7afcb7e6b6293a5d2b7130725533e29ce9aeeaf6c99a2da4850b4a14
url	https://packages.chef.io/files/stable/chef-workstation/24.12.1073/ubuntu/22.04/chef-workstation_24.12.1073-1_amd64.deb
version	24.12.1073

One command confirms the whole toolbox landed, and it answers the question you will be asked constantly: which Chef Infra Client version does my workstation actually test with? The Workstation version number and the Infra Client version number are unrelated, which trips up everyone exactly once.

$ chef --version

Chef Workstation version: 24.12.1073
Chef Infra Client version: 18.6.2
Chef InSpec version: 5.22.65
Chef CLI version: 5.6.16
Chef Habitat version: 1.6.1243
Test Kitchen version: 3.7.0
Cookstyle version: 7.32.8

Your first Chef command stops and asks you to accept a licence. That is not a formality. Chef's compiled packages have shipped under a commercial EULA (end user licence agreement, the contract you click through when installing software) since 2019, even though the source code stays Apache 2.0. Your answer is recorded under ~/.chef/accepted_licenses/, so you are asked once per product. On a CI runner (continuous integration, the pipeline that builds and tests your code automatically) nobody is sitting there to type yes, and the prompt will hang the build until it times out. Set CHEF_LICENSE=accept in the environment, or pass --chef-license accept on the command.

That licence split is exactly why CINC exists. CINC (a recursive joke standing for "CINC Is Not Chef") is a community rebuild of the same Apache-2.0 source with Chef's trademarks and branding stripped out. It ships cinc-client, cinc-workstation and cinc-auditor in place of chef-client, Chef Workstation and InSpec. Same code, same behaviour, no licence prompt, no commercial terms. Everything in this course works on either one. Wherever a command or a path says chef, the CINC build says cinc.

Scaffold a repo, then read what it made

A Chef repo is an ordinary directory of files, not a magic workspace. The generator saves you from arguing about layout: chef generate repo builds the top level, and chef generate cookbook builds a working cookbook inside it, test harness included. Both run git init for you, and the repo generator writes a .gitignore before you have typed a line of code, which matters more than it sounds, because that directory is about to hold a private key. One default worth knowing: the repo generator now assumes Policyfiles and creates a policyfiles/ directory. Pass --roles if you genuinely need the legacy roles/ and environments/ layout.

$ chef generate repo ~/chef-repo && cd ~/chef-repo && chef generate cookbook cookbooks/base

Generating Chef Infra repo chef-repo
- Ensuring correct Chef Infra repo file content
 
Your new Chef Infra repo is ready! Type `cd chef-repo` to enter it.
 
Generating cookbook base
- Ensuring correct cookbook content
- Committing cookbook files to git
 
Your cookbook is ready. Type `cd cookbooks/base` to enter it.

$ tree cookbooks/base

cookbooks/base
├── CHANGELOG.md
├── LICENSE
├── Policyfile.rb
├── README.md
├── chefignore
├── compliance
│   ├── README.md
│   ├── inputs
│   ├── profiles
│   └── waivers
├── kitchen.yml
├── metadata.rb
├── recipes
│   └── default.rb
├── spec
│   ├── spec_helper.rb
│   └── unit
│       └── recipes
│           └── default_spec.rb
└── test
    └── integration
        └── default
            └── default_test.rb
 
11 directories, 12 files

Five things in there carry real weight. metadata.rb names the cookbook, sets its version, and lists what it depends on. Policyfile.rb is the modern way to lock the exact cookbook versions a node is allowed to run; roles and environments still work but are legacy, and every generated cookbook now assumes Policyfiles. kitchen.yml is YAML (a plain, indentation-based config format) telling Test Kitchen which platforms to build a throwaway machine for. test/integration/default/default_test.rb is an InSpec test that runs against that throwaway machine after the converge, checking the outcome rather than the intent. And compliance/ is where InSpec profiles, waivers and inputs go if you want the client to audit a machine in the same run that configures it. Cookstyle reads the Ruby files and tells you whether any of them use syntax Chef 18 has deprecated.

$ cookstyle cookbooks/base

Inspecting 6 files
......
 
6 files inspected, no offenses detected

Teaching knife who you are

Every knife request is signed like a cheque: your name on it, a signature proving the name is yours, and the branch address it goes to. knife needs those same three facts before it can say a word to the server. They live in .chef/config.rb inside your repo, in a directory the generator does not create, so you make it yourself. Older documentation calls this file knife.rb. Same file, renamed: knife looks for config.rb first and falls back to knife.rb when that is what it finds. The key itself is a .pem file (Privacy Enhanced Mail, which despite the name is only the text wrapper format that keys and certificates are stored in).

# ~/chef-repo/.chef/config.rb  (read by knife AND by the chef CLI)
current_dir = File.dirname(__FILE__)
 
node_name         'sachin'                       # your Chef Infra Server USER, not a node
client_key        "#{current_dir}/sachin.pem"    # that user's private key
chef_server_url   'https://chef.example.com/organizations/secopslog'
cookbook_path     ["#{current_dir}/../cookbooks"]
trusted_certs_dir "#{current_dir}/trusted_certs"
ssl_verify_mode   :verify_peer                   # the default; never set :verify_none

Read node_name twice. It is the name of your human account on the Chef Infra Server, the person logging in. It is not the name of any machine. Every managed machine, which Chef calls a node, has a separate identity called a client, with its own private key at /etc/chef/client.pem. Mixing the two up is the most common cause of a baffling 401 (the HTTP status code for "I do not accept who you claim to be"), because the server is honestly telling you that the name you signed as does not match the key you signed with.

If you deal with more than one Chef Infra Server, a credentials file gives you named profiles instead of a drawer full of config files. It is TOML (Tom's Obvious Minimal Language, a plain key-and-value config format), it lives at ~/.chef/credentials, and you pick a profile with --profile or the CHEF_PROFILE environment variable.

[default]
client_name     = "sachin"
client_key      = "/home/sachin/.chef/sachin.pem"
chef_server_url = "https://chef.example.com/organizations/secopslog"
 
[prod]
client_name     = "sachin"
client_key      = "/home/sachin/.chef/sachin-prod.pem"
chef_server_url = "https://chef-prod.example.com/organizations/secopslog"

Here is the detail that bites people daily. knife finds its configuration by walking up the directory tree from wherever your shell happens to be, looking for a .chef folder, and only falls back to ~/.chef when the walk finds nothing. Run knife from your home directory and it may quietly pick a different identity and a different server than the one you were reading about thirty seconds earlier in your repo. When anything surprises you, ask knife what it thinks it is doing before you start editing files. One caveat on the command below: it reads ~/.chef/credentials and nothing else, so a repo whose identity comes from .chef/config.rb will not show up in the list at all.

$ knife config list-profiles

 Profile  Client  Key                                 Server
------------------------------------------------------------------------------------------------------------
*default  sachin  /home/sachin/.chef/sachin.pem       https://chef.example.com/organizations/secopslog
 prod     sachin  /home/sachin/.chef/sachin-prod.pem  https://chef-prod.example.com/organizations/secopslog

$ ls -l .chef/sachin.pem
$ git check-ignore -v .chef/sachin.pem

-rw------- 1 sachin sachin 1678 Jul 21 09:14 .chef/sachin.pem
.gitignore:12:.chef	.chef/sachin.pem

Those two lines turn "probably safe" into "checked". Mode 600, which ls prints as -rw-------, means only you can read the key. The check-ignore output names the exact rule, on the exact line of .gitignore, that will stop you committing it, and here that rule is the whole .chef directory rather than a pattern for .pem files. Silence from check-ignore means the file is not ignored, and one absent-minded git add -A later your fleet credential is in a repository, in a mirror, in someone's laptop clone, forever.

A directory-wide ignore rule comes with a trap. Git never looks inside a directory it has already excluded, so adding !.chef/config.rb underneath it does nothing at all, and your config stays untracked however loudly you negate it. If you want the config in version control and the keys out, change the rule from .chef to .chef/* first. Then the negation works, and git check-ignore will show you the new rule winning.

Trusting the server, and the five seconds where you can be robbed

A stock Chef Infra Server presents a self-signed TLS certificate. TLS (Transport Layer Security) is the lock on HTTPS, and a certificate is the ID card the server shows when you connect. Self-signed means the server wrote its own ID card instead of a recognised certificate authority (CA, an organisation your machine already trusts to vouch for others) writing it, rather like a stranger's handwritten note vouching for the stranger. Your workstation has no reason to believe it, so your first knife command fails with a wall of text. That wall is genuinely helpful. Read it instead of reaching for the nearest fix.

$ knife ssl check

Connecting to host chef.example.com:443
ERROR: The SSL certificate of chef.example.com could not be verified
Certificate issuer data: /C=US/ST=WA/L=Seattle/O=Chef Software/OU=Operations/CN=chef.example.com
 
Configuration Info:
 
OpenSSL Configuration:
* Version: OpenSSL 3.0.13 30 Jan 2024
* Certificate file: /opt/chef-workstation/embedded/ssl/cert.pem
* Certificate directory: /opt/chef-workstation/embedded/ssl/certs
Chef SSL Configuration:
* ssl_ca_path: nil
* ssl_ca_file: nil
* trusted_certs_dir: "/home/sachin/chef-repo/.chef/trusted_certs"
 
TO FIX THIS ERROR:
 
If the server you are connecting to uses a self-signed certificate, you must
configure chef to trust that server's certificate.
 
By default, the certificate is stored in the following location on the host
where your chef-server runs:
 
  /var/opt/opscode/nginx/ca/SERVER_HOSTNAME.crt
 
Copy that file to your trusted_certs_dir (currently:
/home/sachin/chef-repo/.chef/trusted_certs) using SSH/SCP or some other secure
method, then re-run this command to confirm that the server's certificate is
now trusted.

Notice where trusted_certs_dir points. It sits inside the repo, next to the config file knife loaded, not in your home directory. Trust is per-repo, which is good design, because a certificate you accepted for a lab server never leaks into your production repo. It is also a surprise the first time you change directories and lose it. Notice too that the error names the exact path of the certificate on the server. That is the copy you want, and SCP (secure copy, file transfer riding on an SSH connection) is how you would fetch it by hand.

The shortcut everyone reaches for is knife ssl fetch. It works, it is one command, and it prints a warning you should actually read.

$ knife ssl fetch

WARNING: Certificates from chef.example.com will be fetched and placed in your trusted_cert
directory (/home/sachin/chef-repo/.chef/trusted_certs).
 
Knife has no means to verify these are the correct certificates. You should
verify the authenticity of these certificates after downloading.
 
Adding certificate for chef_example_com in /home/sachin/chef-repo/.chef/trusted_certs/chef_example_com.crt

"Knife has no means to verify these are the correct certificates" is the whole security story in one line. So do the verification the message asks for. Compare the fingerprint of what you downloaded against the fingerprint of the real file on the server, read over a channel you already trust. The fingerprint here is a SHA-256 hash of the certificate, a short string that changes completely if a single byte of the certificate changes. An SSH session you already have open to the Chef server counts as a trusted channel. A screenshot pasted into a group chat does not.

# on the Chef Infra Server, in a session you already trust
$ sudo openssl x509 -noout -fingerprint -sha256 -in /var/opt/opscode/nginx/ca/chef.example.com.crt
 
# on your workstation, against what knife saved
$ openssl x509 -noout -fingerprint -sha256 -in .chef/trusted_certs/chef_example_com.crt

sha256 Fingerprint=3D:9F:0A:57:C1:64:E8:2B:7A:11:D0:5E:93:46:F2:8C:B5:20:7D:6A:E4:31:C9:08:AF:52:1B:73:D8:6C:90:E7
 
sha256 Fingerprint=3D:9F:0A:57:C1:64:E8:2B:7A:11:D0:5E:93:46:F2:8C:B5:20:7D:6A:E4:31:C9:08:AF:52:1B:73:D8:6C:90:E7

$ knife ssl check
$ knife client list

Connecting to host chef.example.com:443
Successfully verified certificates from `chef.example.com'
secopslog-validator

One client already exists in a brand new organisation, and its name ends in -validator. Hold that thought. The bootstrap section explains why you would rather it stayed unused.

Putting chef-client onto a node

chef-client has to exist on every machine Chef manages, and you rarely install it by hand. knife bootstrap is the locksmith's visit that turns an unmanaged box into a managed one, and it works in a fixed order: connect over SSH (or WinRM, Windows Remote Management, the equivalent for Windows targets), download and install the Chef Infra Client version you asked for, create a client record and a fresh key pair on the server, write /etc/chef/client.rb and /etc/chef/client.pem onto the machine, then run chef-client once so the node reports for duty.

One thing has to happen first. A policy group is a label on the server, something like production, pointing at one exact revision of one policy. Nothing can be bootstrapped into a group that does not exist yet. So from the cookbook directory you run chef install, which resolves dependencies and writes Policyfile.lock.json, then chef push production, which uploads that lock file and attaches it to the production group. Only then does the command below have something to hand the new node.

$ knife bootstrap 10.0.0.5 \
    --connection-user ubuntu \
    --ssh-identity-file ~/.ssh/id_ed25519 \
    --sudo \
    --node-name web01 \
    --policy-name base --policy-group production \
    --bootstrap-version 18.6.2

Connecting to 10.0.0.5 using ssh
Creating new client for web01
Creating new node for web01
Bootstrapping 10.0.0.5
 [10.0.0.5] -----> Installing Chef Omnibus (stable/18.6.2)
 [10.0.0.5] ubuntu 24.04 x86_64
 [10.0.0.5] Getting information for chef stable 18.6.2 for ubuntu...
 [10.0.0.5] trying wget...
 [10.0.0.5] Installing chef 18.6.2
 [10.0.0.5] installing with dpkg...
 [10.0.0.5] Thank you for installing Chef Infra Client!
 [10.0.0.5] Chef Infra Client, version 18.6.2
 [10.0.0.5] Patents: https://www.chef.io/patents
 [10.0.0.5] Infra Phase starting
 [10.0.0.5] Resolving cookbooks for run list: ["base::default"]
 [10.0.0.5] Synchronizing cookbooks:
 [10.0.0.5]   - base (0.1.0)
 [10.0.0.5] Installing cookbook gem dependencies:
 [10.0.0.5] Compiling cookbooks...
 [10.0.0.5] Converging 4 resources
 [10.0.0.5] Running handlers:
 [10.0.0.5] Running handlers complete
 [10.0.0.5] Infra Phase complete, 3/4 resources updated in 18 seconds

Two things in that command deserve a second look. --policy-name and --policy-group hand the node a Policyfile instead of a run-list, which is the current way to pin exactly which cookbook versions it may run; you cannot pass both a policy and --run-list, so pick one. --bootstrap-version pins what gets installed, because without it bootstrap fetches the newest client available, and the version that lands on each machine then depends on the date you happened to build it. On a network with no route to the internet, --bootstrap-url points the installer at your own mirror of the install script instead of omnitruck.chef.io.

Notice what did not happen: no shared secret touched that machine. Modern knife bootstrap signs the client-creation request with your own user key, which Chef calls a validatorless bootstrap. The old way used validation.pem, one organisation-wide key copied onto every new machine, able to register anything as anything, and it had a habit of sitting forgotten inside a golden image or a Packer template for years. That is the extra -validator client you saw a moment ago. If your organisation still uses it, deleting it is a free security win.

$ knife node show web01
$ knife status

Node Name:   web01
Policy Name: base
Policy Group: production
FQDN:        web01.example.internal
IP:          10.0.0.5
Run List:
Recipes:     base::default
Platform:    ubuntu 24.04
Tags:
 
1 minute ago, web01, web01.example.internal, 10.0.0.5, ubuntu 24.04.

The empty Run List line is correct rather than broken. This node is driven by a Policyfile, so knife prints Policy Name and Policy Group where a run-list-driven node would show Environment, and the recipes the policy expands to appear under Recipes instead. knife status is the heartbeat of your fleet and the report worth alerting on. A node that last checked in nine days ago is either switched off or broken, and broken usually means an expired key, a wrong server URL, or a certificate nobody trusts any more. Silence looks exactly like health if you never ask.

You can learn all of this without a server

Standing up a Chef Infra Server to try one recipe is a lot of ceremony, and Chef Workstation ships two ways around it. chef-run packages a resource or a recipe on your laptop, pushes it to a host over SSH, installs a temporary client if the host needs one, and applies it. No server, no node record, nothing left behind on the records office side. It is the closest Chef gets to an ad-hoc command, and it earns its keep on one-off fixes as much as on learning.

$ chef-run ssh://[email protected] --identity-file ~/.ssh/id_ed25519 package nginx action=install

[✔] Packaging cookbook... done!
[✔] Generating local policyfile... exporting... done!
[✔] Applying package[nginx] to [email protected]... done!

The other way never leaves your own machine. chef-client --local-mode, or -z after the tiny chef-zero server it starts, spins up an in-memory Chef server, reads cookbooks from the cookbooks directory beside where you run it, converges the local machine, then throws the server away. Run it with sudo, because installing a package needs root and the run will fail halfway through without it. The recipe code executing under -z on your laptop is the same code that will execute on web01. What changes is whose keys signed for it, and who has to live with the result.

$ sudo chef-client --local-mode --runlist 'recipe[base::default]'

Chef Infra Client, version 18.6.2
Patents: https://www.chef.io/patents
Infra Phase starting
Resolving cookbooks for run list: ["base::default"]
Synchronizing cookbooks:
  - base (0.1.0)
Installing cookbook gem dependencies:
Compiling cookbooks...
Converging 2 resources
Recipe: base::default
  * apt_package[htop] action install
    - install version 3.3.0-4build1 of package htop
  * directory[/etc/secopslog] action create (up to date)
 
Running handlers:
Running handlers complete
Infra Phase complete, 1/2 resources updated in 06 seconds

Try this

Run curl -L https://omnitruck.chef.io/install.sh -o install.sh on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: that .pem is a root shell on every node. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related