Cookbook structure & dependencies

Berkshelf, metadata, versioning.

A cookbook is a shipping crate. Everything one job needs travels inside it: recipes, templates, attribute files, custom resources, small Ruby helper libraries. Taped to the lid is a packing list. It names the crate, stamps a version on it, and names every other crate that has to arrive at the same time. Chef Infra Client, the agent that runs on each of your machines and drags that machine into line with your code, reads the packing list before a single line of your recipe runs. Get the list wrong and the failure is rarely quiet in a useful way. Either the machine refuses to start, or it finishes perfectly happily using a version of somebody else's cookbook that you have never read. This lesson is about the packing list, the numbers stamped on it, and how Berkshelf turns a wish list into a locked, uploaded set.

Let the generator lay down the crate

Building a cookbook directory by hand is how you end up with something that works on your laptop and nowhere else. The chef generate cookbook command, which ships with Chef Workstation, writes a complete one in a single shot: metadata, a default recipe, a chefignore file, a Test Kitchen config (Test Kitchen builds a throwaway virtual machine or container, runs your cookbook on it, then bins it), unit and integration test stubs, and a git repository already initialized with the first commit made. One default catches people out. The current generator writes a Policyfile.rb, not a Berksfile. Those are two different ways of pinning dependencies, and a cookbook picks one or the other, so pass --berks when you want the Berkshelf layout this lesson covers.

chef --version
chef generate cookbook web_app --berks

Chef Workstation version: 24.4.1064
Chef Infra Client version: 18.4.12
Chef InSpec version: 5.22.55
Chef CLI version: 5.6.16
Chef Habitat version: 1.6.1013
Test Kitchen version: 3.5.0
Cookstyle version: 7.32.8
 
Generating cookbook web_app
- Ensuring correct cookbook content
- Committing cookbook files to git
 
Your cookbook is ready. Type `cd web_app` to enter it.
 
There are several commands you can run to get started locally developing and testing your cookbook.
 
Why not start by writing an InSpec test? Tests for the default recipe are stored at:
 
test/integration/default/default_test.rb

cd web_app
tree -a -L 1 --dirsfirst

.
├── .git
├── recipes
├── spec
├── test
├── .gitignore
├── Berksfile
├── CHANGELOG.md
├── chefignore
├── kitchen.yml
├── LICENSE
├── metadata.rb
└── README.md
 
4 directories, 8 files

Only recipes/ gets scaffolded. The other well known directories are conventions that Chef finds by name, and you add them the day you need them. attributes/ holds default values for a machine. templates/ holds ERB files (Embedded Ruby, a text file with small pieces of Ruby baked in, so one template can produce different config on different machines). files/ holds static drops. resources/ holds custom resources. libraries/ holds plain Ruby helpers. There is also compliance/, which the generator does not create for you, and which Chef Infra Client 17 and later scan for InSpec profiles, inputs and waivers. InSpec is Chef's test language for asking a machine a direct question: is this actually true about you?

Read chefignore before you commit anything, because it carries far more weight than the name suggests. Every file in this directory that chefignore does not exclude is uploaded to the Chef Infra Server, then downloaded onto every machine that uses the cookbook, into /var/chef/cache/cookbooks. A stray .env. A test fixture holding a real password. A private key you dropped in while debugging. All of it ships, to every node, in the clear. chefignore is the filter, and for uploads it is the only one you get.

metadata.rb, the packing list

metadata.rb is the file every machine reads. It names the cookbook, stamps the version, declares which Chef Infra Client releases the code is fit for, and lists dependencies with depends. Nothing else in the cookbook overrides it. No amount of setup on your workstation stands in for it. If a recipe reaches for a resource or a recipe that lives in a cookbook missing from depends, the run does not degrade politely. It stops.

name             'web_app'
maintainer       'Platform Team'
maintainer_email '[email protected]'
license          'Apache-2.0'
description      'Installs and configures the web tier'
version          '2.3.1'                 # MAJOR.MINOR.PATCH, bump on every change
 
source_url       'https://github.com/acme/web_app'
issues_url       'https://github.com/acme/web_app/issues'
 
# refuse to run on Infra Client releases this code was never tested against
chef_version     '>= 17.0', '< 19.0'
 
supports 'ubuntu', '>= 20.04'
supports 'redhat', '>= 8.0'
 
# exactly one constraint per line; this list is what a node actually loads
depends 'nginx',    '~> 12.0'   # >= 12.0.0 and < 13.0.0
depends 'firewall', '>= 2.7'    # any 2.7 or newer, including 6.x
depends 'internal_base'         # unconstrained: whatever solves today
 
# Chef Infra Client 17+ can skip auto-loading every file in libraries/
# eager_load_libraries false

The version string is MAJOR.MINOR.PATCH, and other people write constraints against it, so treat it as a promise you are making to them. Bump PATCH for a fix that changes no interface. Bump MINOR for a backward compatible addition, like a new attribute with a sensible default. Bump MAJOR when you rename an attribute, delete a recipe, or change what a resource does by default. Bump on every change that leaves your machine, with no exceptions. A version number that sits still while the content moves is the most expensive habit in Chef, and much of the rest of this lesson is about why.

chef_version fences off client releases you have never tested against. A machine running an older client stops during compile, the phase where Chef reads all of your Ruby before it touches anything on disk, and raises Chef::Exceptions::CookbookChefVersionMismatch with both the constraint and the running client version printed in the message. That is a far better outcome than a half finished run on a Ruby your code does not understand. Two details are worth carrying around. chef_version accepts several constraints in one call and all of them have to hold, while writing the line twice gives you alternatives, where satisfying either line is enough. depends does not work like that at all. It takes exactly one constraint, and a second one raises Chef::Exceptions::ObsoleteDependencySyntax, so you write ~> 12.0 rather than trying to say '>= 12.0', '< 13.0'. If you run CINC (short for CINC Is Not Chef, the trademark free rebuild of the same source), the client reports the same version numbers, so every constraint here behaves identically.

Metadata mistakes tend to be quiet ones. A license string nobody recognizes produces no error at upload. It only makes your cookbook harder for other tools to consume, months later, in somebody else's pipeline. cookstyle is the Chef specific linter, a program that reads your code and flags known mistakes without running any of it, built on top of RuboCop, the general purpose Ruby linter. It reads metadata.rb like any other Ruby file.

# break the license string the way a copy-paste does
sed -i "s/'Apache-2.0'/'Apache 2.0'/" metadata.rb
cookstyle metadata.rb

Inspecting 1 file
R
 
Offenses:
 
metadata.rb:4:18: R: [Correctable] Chef/Sharing/InvalidLicenseString: Cookbook metadata.rb does not use a SPDX compliant license string or "all rights reserved". See https://spdx.org/licenses/ for a complete list of licenses.
license          'Apache 2.0'
                 ^^^^^^^^^^^^
 
1 file inspected, 1 offense detected, 1 offense autocorrectable

cookstyle -a metadata.rb corrects it in place. SPDX (Software Package Data Exchange) is the standard list of license identifiers, and Supermarket, Chef's public cookbook site, reads that string literally. So does every license scanner in your build pipeline. To a machine, Apache 2.0 and Apache-2.0 are two unrelated pieces of text.

Reading a version constraint properly

The pessimistic operator ~> is the one you will type most. Think of it as a leash. It lets the version wander forward on its own, but only so far, and the number of digits you write decides how long the leash is. ~> 12.0 means anything from 12.0.0 up to but not including 13.0.0, so a new minor release is welcome and a new major one is refused at the gate. ~> 12.0.1 shortens the leash to the minor version: at least 12.0.1, less than 12.1.0. A plain >= 2.7 sets a floor and no ceiling whatsoever, which is how a cookbook that asked for firewall 2.7 quietly ends up on 6.0.6 three years later. A bare depends 'internal_base' with no constraint at all means >= 0.0.0. That is a standing promise to accept whatever exists on the day somebody runs the solver, the piece of code that picks one version of each cookbook that keeps every constraint happy at the same time.

That last one is a supply chain hole with a name on it. Supermarket is open publishing: anyone can register an account and push a cookbook, and cookbook code runs as root on every machine that loads it. If an upstream maintainer gets phished and a fresh release appears carrying one extra execute resource, an unconstrained depends is the pipe that carries it into your estate the next time your build server runs berks install. Pinning with ~> does not make you safe. It shrinks the blast radius, meaning how many machines a bad release can reach and how far into them it gets, and it turns a silent change into one a human can review. The lock file is what makes the change visible at all.

Berkshelf solves the graph and writes the answer down

metadata.rb says what you need. The Berksfile says which shop to buy it from. berks install reads both, walks the whole graph including the dependencies of your dependencies, and picks one version of each cookbook that satisfies every constraint at once. The metadata keyword in the Berksfile is what pulls your depends lines into that solve. Each cookbook line redirects where one specific cookbook is fetched from, without changing what the node believes it needs.

source 'https://supermarket.chef.io'    # public community cookbooks
 
metadata                                # pull in the depends lines from metadata.rb
 
# fetch this one from git instead of Supermarket
cookbook 'internal_base',
         git: '[email protected]:acme/internal_base.git',
         tag: 'v1.4.0'
 
# and this one from the working copy next door, while you edit both
cookbook 'sibling_app', path: '../sibling_app'

Notice tag: 'v1.4.0' rather than branch: 'main'. A branch is a moving pointer, so aiming at main means whatever somebody happened to push before you ran the solver, which is not a pin in any meaningful sense. A tag is better. ref: with a full commit hash (the fingerprint of one exact commit, computed from its contents) is better still, because a tag can be moved and a commit hash cannot. The path: entry is for a cookbook you are editing in the same sitting. It will never resolve on a build server, so keep it out of what you commit, or your pipeline breaks in a way that takes an afternoon to understand.

berks install

Resolving cookbook dependencies...
Fetching 'web_app' from source at .
Fetching 'internal_base' from [email protected]:acme/internal_base.git (at v1.4.0)
Fetching 'sibling_app' from source at ../sibling_app
Fetching cookbook index from https://supermarket.chef.io...
Using web_app (2.3.1) from source at .
Using internal_base (1.4.0) from [email protected]:acme/internal_base.git (at v1.4.0)
Using sibling_app (0.4.2) from source at ../sibling_app
Installing firewall (6.0.6)
Installing nginx (12.2.9)
Installing sudo (5.4.7)

Six cookbooks for three depends lines. sudo is in that list because internal_base depends on it, and nobody on your team ever typed its name. sibling_app is there because the Berksfile asked for it, even though no depends line mentions it, and that gap matters in a minute. firewall landed on 6.0.6, four major versions past the 2.7 you had in mind when you wrote >= 2.7. This is the moment to look, because it is the last moment where looking is cheap. Downloaded cookbooks land in ~/.berkshelf/cookbooks, and the resolved set is written into Berksfile.lock at the cookbook root.

{
  "DEPENDENCIES": {
    "internal_base": {
      "git": "[email protected]:acme/internal_base.git",
      "revision": "8f3c1c8a3f4b6e0e0f2a7f9a6b1d4c5e6a7b8c9d",
      "tag": "v1.4.0"
    },
    "sibling_app": {
      "path": "../sibling_app"
    },
    "web_app": {
      "path": ".",
      "metadata": true
    }
  },
  "GRAPH": {
    "firewall": {
      "version": "6.0.6"
    },
    "internal_base": {
      "version": "1.4.0",
      "dependencies": {
        "sudo": ">= 0.0.0"
      }
    },
    "nginx": {
      "version": "12.2.9"
    },
    "sibling_app": {
      "version": "0.4.2"
    },
    "sudo": {
      "version": "5.4.7"
    },
    "web_app": {
      "version": "2.3.1",
      "dependencies": {
        "firewall": ">= 2.7",
        "internal_base": ">= 0.0.0",
        "nginx": "~> 12.0"
      }
    }
  }
}

The lock records the exact versions the solver chose and, for git sources, the commit it actually fetched. Commit it to your repository. Then read it during code review, properly, because one line moving from 12.2.9 to 13.0.1 in that file is a larger event than most of the diffs you wave through in a week. berks update re-solves everything and rewrites the whole lock. berks update nginx unlocks that one cookbook and re-solves, which moves nginx and anything nginx drags along with it while leaving the rest of the graph where it was. That is what you want when you are chasing one security fix rather than pulling the entire graph forward on a Friday afternoon.

berks list
berks outdated
berks contingent sudo

Cookbooks installed by your Berksfile:
  * firewall (6.0.6)
  * internal_base (1.4.0)
  * nginx (12.2.9)
  * sibling_app (0.4.2)
  * sudo (5.4.7)
  * web_app (2.3.1)
 
Fetching cookbook index from https://supermarket.chef.io...
The following cookbooks have newer versions:
  * nginx (13.0.1)
  * sudo (6.0.2)
 
Cookbooks in this Berksfile contingent upon sudo:
  * internal_base (1.4.0)

berks contingent answers the question you ask in the middle of an incident: something is wrong with sudo 5.4.7, so who pulled it in? berks outdated is the one to run on a schedule, because a pinned dependency nobody ever revisits is its own category of risk, distinct from an unpinned one and often worse. Note that it checks every cookbook in the lock against your sources, transitive ones included, which is why sudo shows up even though you never asked for it. And berks package cookbooks.tar.gz writes the entire resolved set into a single archive file, which is how a known good set gets carried into an air gapped network, one with no route to the internet at all.

Upload, freeze, and who is allowed to overwrite you

berks upload pushes the resolved set to the Chef Infra Server: your cookbook plus every cookbook in the graph, at exactly the versions the lock recorded.

berks upload

Uploading firewall (6.0.6) to: 'https://chef.example.com/organizations/acme'
Uploading internal_base (1.4.0) to: 'https://chef.example.com/organizations/acme'
Uploading nginx (12.2.9) to: 'https://chef.example.com/organizations/acme'
Uploading sibling_app (0.4.2) to: 'https://chef.example.com/organizations/acme'
Uploading sudo (5.4.7) to: 'https://chef.example.com/organizations/acme'
Uploading web_app (2.3.1) to: 'https://chef.example.com/organizations/acme'

Berkshelf freezes what it uploads, and it does that by default. Frozen means that exact version number is now immutable on the server: those bytes, under that number, permanently. Which raises the obvious question. Edit a template, forget to bump version, upload again. What happens?

# edited templates/nginx.conf.erb, left version at 2.3.1
berks upload
echo "exit: $?"

Skipping firewall (6.0.6) (frozen)
Skipping internal_base (1.4.0) (frozen)
Skipping nginx (12.2.9) (frozen)
Skipping sibling_app (0.4.2) (frozen)
Skipping sudo (5.4.7) (frozen)
Skipping web_app (2.3.1) (frozen)
exit: 0

Read that twice, because it is the trap and almost nobody expects it. Berkshelf did not fail. It printed six flat identical lines, exited zero, and your edited template is still sitting on your laptop. Nothing on the server changed. If that command ran inside a build pipeline, the pipeline went green and told you the deploy succeeded. The line that matters, Skipping web_app (2.3.1) (frozen), is hiding in a wall of noise from five cookbooks you never touched. Two habits close this. Bump the version on every change, which is the actual fix. And pass --halt-on-frozen in CI (continuous integration, the build server that runs your pipeline for you), which converts that quiet skip into a hard failure and a non-zero exit with the message The cookbook web_app (2.3.1) already exists and is frozen on the Chef Server. Use the --force option to override. Compare all of this with knife cookbook upload, which does not freeze anything unless you pass --freeze. An estate that uploads with knife from a shell script is an estate where any version number can quietly start meaning different code.

Proving the dependency is real

Your own workstation is the worst possible place to test a dependency graph, because it already has everything. The honest test is a machine that starts with nothing and receives only what your metadata asked for, which is precisely what Test Kitchen builds: a fresh throwaway box, converged, then destroyed. Converge is Chef's word for the part of the run where it stops reading and starts changing the machine. Here is what a missing depends looks like the first time it meets a machine that was not sitting there while you were debugging.

kitchen converge default-ubuntu-2204

-----> Starting Test Kitchen (v3.5.0)
-----> Converging <default-ubuntu-2204>...
       Preparing files for transfer
       Resolving cookbook dependencies with Berkshelf 8.0.2...
       Removing non-cookbook files before transfer
       Preparing dna.json
       Preparing validation.pem
       Preparing client.rb
       Transferring files to <default-ubuntu-2204>
       Starting Chef Infra Client, version 18.4.12
       Patents: https://www.chef.io/patents
       Infra Phase starting
 
       ================================================================================
       Recipe Compile Error in /tmp/kitchen/cookbooks/web_app/recipes/default.rb
       ================================================================================
 
       Chef::Exceptions::CookbookNotFound
       ----------------------------------
       Cookbook sudo not found. If you're loading sudo from another cookbook, make sure you configure the dependency in your metadata
 
       Cookbook Trace: (most recent call first)
       ------------------------------------------
         /tmp/kitchen/cookbooks/web_app/recipes/default.rb:9:in `from_file'
 
       System Info:
       ------------
       chef_version=18.4.12
       platform=ubuntu
       platform_version=22.04
       ruby=ruby 3.1.4p223 (2023-03-30 revision 957bb7cb81) [x86_64-linux]
       program_name=/opt/chef/bin/chef-client
 
       Running handlers:
       [2026-07-21T09:12:44+00:00] ERROR: Running exception handlers
       Running handlers complete
       [2026-07-21T09:12:44+00:00] ERROR: Exception handlers complete
       Chef Infra Client failed. 0 resources updated in 03 seconds
       [2026-07-21T09:12:44+00:00] FATAL: Stacktrace dumped to /tmp/kitchen/cache/chef-stacktrace.out
       [2026-07-21T09:12:44+00:00] FATAL: Chef::Exceptions::CookbookNotFound: Cookbook sudo not found. If you're loading sudo from another cookbook, make sure you configure the dependency in your metadata
>>>>>> Converge failed on instance <default-ubuntu-2204>.

The error names the fix in plain words. Your recipe calls include_recipe 'sudo' and your metadata never mentions sudo. Here is the part that surprises people: Berkshelf fetched sudo anyway, because internal_base depends on it, so the files are sitting right there in the sandbox at /tmp/kitchen/cookbooks/sudo. The machine refuses regardless. The set of cookbooks a run is allowed to load is built from metadata, not from whatever happens to be on disk. Add depends 'sudo', bump the version to 2.3.2, run berks install, run berks upload, converge again. Then check against the server itself rather than trusting an upload summary you already read once.

knife cookbook list -a

firewall        6.0.6
internal_base   1.4.0
nginx           12.2.9  12.1.0
sibling_app     0.4.2
sudo            5.4.7
web_app         2.3.2  2.3.1  2.3.0

Two things in that listing repay attention. web_app now holds three versions on the server, each one frozen and permanent, which is what lets you answer 'what was actually running last Tuesday' with evidence instead of a shrug. And nginx holds two, which leads straight into the part of Berkshelf people misread.

Where Berkshelf runs out of road

Your lock file pins your workstation. It does not pin the node. When a machine checks in, the Chef Infra Server runs its own solve against that node's run-list (the ordered list of recipes and roles the node is meant to apply), its environment, and the metadata of the cookbooks the server happens to be holding. Then it sends back the newest versions that satisfy every constraint. With two nginx versions on the server and ~> 12.0 in your metadata, the node takes 12.2.9 today. If a colleague uploads 12.3.0 tomorrow, the node takes that instead on its very next run, with nothing changing on your side and nothing in your lock file to record that it happened. The Berkshelf answer to this was berks apply production, which copies exact = 12.2.9 pins out of the lock and into a Chef environment object. It works. The cost is that the pins now live in a completely separate object from the cookbook that needed them, and the two drift apart the moment somebody forgets.

Policyfiles close that gap by solving the graph once, on your side, and shipping the answer itself to the node rather than shipping a set of questions. A Policyfile.lock.json records the version of every cookbook plus an identifier derived from its content, so a node runs one specific pile of bytes instead of a set of constraints that a server re-solves for it three weeks later. That is why chef generate cookbook writes a Policyfile by default and why new work should start there. Berkshelf is still bundled with Chef Workstation, still what a great many existing estates run, and still reasonable for a shared cookbook that several teams consume at different versions. If that is where you are, the operating rules are short. Constrain every depends. Commit Berksfile.lock and read its diff in review. Run uploads with --halt-on-frozen so a skipped upload cannot pass as a successful one. Keep --force out of every script. And bump the version on every single change, because on a Chef Infra Server the version number is the only identity a cookbook has.

Try this

Run chef --version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the Berksfile is a shopping list, metadata.rb is the packing list. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related