Attributes & precedence

Where node values come from.

A laminated recipe card in a busy kitchen says the oven runs at 180C. The head chef tapes a note over that line reading 200C. The oven door has a stamped metal plate saying it tops out at 250C, and nobody argues with the oven. Chef Infra works the same way. A cookbook writes down a sensible starting number, a policy pastes something over the top of it, and the machine reports facts that no cookbook is allowed to contradict.

Your recipe asks for node['myapp']['port'] and gets back a single number. It never sees the layers. Chef already merged every source that had an opinion, in a fixed order, and handed you the winner. Attributes are the values attached to a node (one machine that Chef manages). Precedence is the tie-break rule that decides which source wins. Get it wrong and you ship a firewall allow-list wider than the one you typed, or you burn an afternoon editing a cookbook default that something above it has been quietly beating for months.

Six Kinds Of Attribute, Five Of Them Yours

Chef has six attribute types. Five are yours to set: default, force_default, normal, override and force_override. The sixth, automatic, belongs to Ohai (the inventory tool that runs at the start of every Chef Infra Client run and measures the machine it is standing on). Ohai fills in the hostname, the IP addresses (Internet Protocol, the numbers a machine answers on), total memory, platform, mounted filesystems and several hundred other facts. You read those. You do not write them.

default is the everyday baseline, and it is what you should be writing almost all of the time. Chef recomputes it from your cookbook code on every run, so deleting the line deletes the value. force_default sits above every other default source, which is how a cookbook takes the top of the default family back from a policy or a role. normal is the sticky one. Chef writes it into the node object stored on the Chef Infra Server and reads it back on the next run. override is the hammer. force_override is the bigger hammer. automatic beats all of them, always.

# cookbooks/myapp/attributes/default.rb
default['myapp']['port']         = 8080          # everyday baseline
default['myapp']['dir']          = '/opt/myapp'
default['myapp']['allowed_cidr'] = ['10.0.0.0/8']
 
force_default['myapp']['log_level'] = 'info'     # outranks every other default source
 
# read-only, supplied by Ohai:
#   node['fqdn']  node['ipaddress']  node['platform_version']  node['memory']['total']

Inside a recipe the same five types hang off the node object: node.default, node.force_default, node.normal, node.override, node.force_override. Each one has an _unless variant (node.default_unless, node.normal_unless) that writes only when the key is missing. node.set was removed in Chef 14. If an old cookbook still carries it, the replacement is node.normal, and you want to read the sticky-value warning further down before you make that swap.

# cookbooks/myapp/recipes/default.rb
node.default['myapp']['workers'] = node['cpu']['total'].to_i * 2  # Ohai fact -> derived default
node.override['myapp']['port']   = 9090                           # beats every default source
 
# nil-safe deep read: returns nil instead of blowing up mid-run on a missing key
ca_path = node.read('myapp', 'tls', 'ca_path') || '/etc/ssl/certs/ca-certificates.crt'
 
log "myapp will listen on #{node['myapp']['port']} using #{ca_path}"

The Ladder Has Fifteen Rungs

Lowest to highest: default in a cookbook attribute file, default set in a recipe, environment default, role default. Then force_default, attribute file first and recipe second. Then normal, attribute file then recipe. Then the override family in the order attribute file, recipe, role, environment. Then force_override, attribute file then recipe. Automatic sits alone at the top. Fifteen rungs.

The fifteen comes from counting "set in an attribute file" and "set in a recipe" as separate rows. Chef only keeps ten buckets internally, and that particular split is a statement about time rather than rank. Attribute files are evaluated while the run compiles, recipes execute after that, and both write into the same bucket, so the later write wins. That is why a node.default line in a recipe beats a default line in the attribute file of the same cookbook.

One inversion causes more confusion than everything else here. In the default family, a role beats an environment. In the override family, an environment beats a role. Same two sources, opposite winner, decided entirely by which family you set them in. When a value refuses to change, something higher on the ladder is winning, and it is usually an override buried in a policy or an Ohai fact you cannot beat at all.

Roles and environments are legacy. The current way to pin what a node runs is a Policyfile, and a Policyfile carries its own default and override blocks. Chef applies them at the role_default and role_override rungs. So a Policyfile default beats any ordinary cookbook default, and a Policyfile override beats any ordinary cookbook override, with force_default and force_override as the two exceptions that still sit above it inside their own family. Those attributes live in a file you review in Git and lock to a revision id, which is the difference between "somebody edited a role in the web console at 2am" and "somebody opened a merge request".

# Policyfile.rb
name 'web'
 
default_source :supermarket
run_list 'myapp::default'
 
cookbook 'myapp', path: 'cookbooks/myapp'
 
# applied at the role_default rung: beats any ordinary cookbook default,
# but a cookbook force_default still outranks it
default['myapp']['allowed_cidr'] = ['192.168.10.0/24']
 
# applied at the role_override rung
override['myapp']['log_level'] = 'warn'

chef install Policyfile.rb

Building policy web
Expanded run list: recipe[myapp::default]
Caching Cookbooks...
Using      myapp     0.1.0
 
Lockfile written to /home/eng/policies/web/Policyfile.lock.json
Policy revision id: 3f9a1c07b28d4e6f5a0b9c8d7e6f5a4b3c2d1e0f9a8b7c6d5e4f3a2b1c0d9e8f

Watch The Merge Happen

Reading the ladder is one thing. Watching it resolve is better. chef-shell is an interactive Chef session, a workbench version of a converge that hands you a live node object to poke at without changing anything on the machine. On CINC (the open source rebuild of Chef Infra with the trademarks stripped out) the command is cinc-shell, and every merge rule below behaves identically because it is built from the same source.

chef-shell

loading configuration: none (standalone session)
Session type: standalone
Loading......done.
 
This is the chef-shell.
 Chef Infra Client Version: 18.4.12
 https://www.chef.io/
 https://docs.chef.io/
 
run `help' for help, `exit' to quit.
 
Ohai2u eng@build01!
chef (18.4.12)>

Now build the collision by hand. node.attributes.role_default= is the writer a Policyfile uses when it applies its default block, so this is the same code path a real converge takes.

chef (18.4.12)> node.default['myapp']['port'] = 8080
chef (18.4.12)> node.attributes.role_default = { 'myapp' => { 'port' => 80 } }
chef (18.4.12)> node['myapp']['port']
chef (18.4.12)> node.override['myapp']['port'] = 9090
chef (18.4.12)> node['myapp']['port']
chef (18.4.12)> node.debug_value('myapp', 'port')

=> 8080
=> {"myapp"=>{"port"=>80}}
=> 80
=> 9090
=> 9090
=> [["default", 8080], ["env_default", :not_present], ["role_default", 80], ["force_default", :not_present], ["normal", :not_present], ["override", 9090], ["role_override", :not_present], ["env_override", :not_present], ["force_override", :not_present], ["automatic", :not_present]]

debug_value walks all ten buckets and reports what each one holds for that key, printing :not_present for the empty ones. For a plain value like a port number, read left to right and the last bucket with something in it is your answer. For arrays and hashes it is not that simple, which is the whole of the next section. Either way this is the fastest route to the stray override somebody left in a policy two quarters ago, and it works inside a recipe too if you wrap it in a log resource.

Arrays Combine Inside A Family And Replace Between Families

There are two merge rules, not one. Within a family (all four default sources, or all four override sources) Chef deep merges: hashes are combined key by key, and arrays are stuck end to end with duplicates dropped. Treat a family as one shopping list that four people are allowed to add items to. Between families (the whole default family, then normal, then the whole override family, then automatic) Chef merges hashes only. Arrays and plain values from the higher family replace what was underneath, the way handing someone a fresh list replaces the old one instead of being stapled to it.

chef (18.4.12)> node.default['myapp']['allowed_cidr'] = ['10.0.0.0/8']
chef (18.4.12)> node.attributes.role_default = { 'myapp' => { 'allowed_cidr' => ['192.168.10.0/24'] } }
chef (18.4.12)> node['myapp']['allowed_cidr']
chef (18.4.12)> node.override['myapp']['allowed_cidr'] = ['192.168.10.0/24']
chef (18.4.12)> node['myapp']['allowed_cidr']

=> ["10.0.0.0/8"]
=> {"myapp"=>{"allowed_cidr"=>["192.168.10.0/24"]}}
=> ["10.0.0.0/8", "192.168.10.0/24"]
=> ["192.168.10.0/24"]
=> ["192.168.10.0/24"]

So the fix for an array you meant to replace is to move it up a family, not up a rung. force_default still lives inside the default family, so it appends exactly like everything else there. Setting the array with node.override, or from the override block of a Policyfile, is what makes it replace. Confirm with debug_value, because this failure is silent. Nothing errors. The list is only longer than the file you edited.

Sticky Values And Where They Hide

default and override are rebuilt from your code on every run. normal is not. Chef saves it into the node object on the server at the end of each run and loads it back at the start of the next one, like a note written on the machine's own record card in permanent marker. That is genuinely useful for a value a node worked out once and has to keep, such as a generated instance id or a token seeded on first boot. It is a trap for ordinary configuration.

Here is what that looks like when it bites. The cookbook ships log_level info as a force_default. The node has been logging at debug for four months and nobody can find the line that sets it.

knife node show web01.example.com -a myapp.log_level

web01.example.com:
  myapp.log_level: debug

knife node show web01.example.com -l -F json | jq '.normal.myapp'

{
  "log_level": "debug",
  "token": "seeded-once"
}

The server keeps four buckets per node: default, normal, override and automatic. The client merges each family locally and uploads the result, which is why that query works. knife node edit shows you the name, environment, run-list, policy name, policy group and the normal bucket, and it hides the rest because everything else is regenerated on the next run anyway. To get rid of the value, take it out on purpose.

# cookbooks/myapp/recipes/cleanup.rb
# add to the run-list once, converge, verify, then delete this recipe
node.rm_normal('myapp', 'log_level')   # returns the removed value, or nil if it was absent
 
# the client saves the node object at the end of the run, which is what makes the removal stick

What A Fleet-Wide Attribute Change Really Is

Attributes are indexed on the Chef Infra Server and searchable, so one query answers a question about the whole estate.

knife search node 'policy_group:production' -a myapp.port

2 items found
 
web01.example.com:
  myapp.port: 9090
 
web02.example.com:
  myapp.port: 9090

Two things follow from that command working. First, it is the fastest inventory you own, and it reports what nodes actually converged to rather than what a repository claims they should be. Second, the same query is available to anything holding a valid client key. Out of the box a node can read other nodes, so one compromised web server can list every attribute on every host in the organisation. Whatever you put in an attribute is stored on the server in clear text and handed straight back by search. No passwords, no API tokens (an API, or Application Programming Interface, is the machine-to-machine door into a service), no private keys. Use an external secret store, or encrypted data bags, and remember that a data bag is encrypted with a shared key sitting on every node allowed to read it, so a compromised node gives those secrets up too. Then trim the node read permissions.

You can also cut down what leaves the machine in the first place. Chef Infra Client 18 takes allow and block lists per attribute family in client.rb. They control what gets saved to the server without changing what recipes can read during the run. Pick one or the other for a given family, not both.

# /etc/chef/client.rb
# keep the local user and group inventory out of the node object on the server
blocked_automatic_attributes [
  %w{etc passwd},
  %w{etc group},
]
 
# an allow list is stricter: anything not named here is dropped before upload,
# which will also break any knife search that relies on those keys
# allowed_normal_attributes [%w{myapp version}]

The other direction matters as much. Attributes feed templates, package versions, repository URLs and firewall rules. Whoever can edit a node's normal attributes or push a new policy revision to a policy group decides what every node in that group installs at its next converge. That is remote code execution wearing the costume of a config edit. Review policy pushes the way you review code, keep the push credentials narrow, and keep the Chef Automate event feed (or the Infra Server's own request logs) somewhere you can search, so a surprise attribute arrives with a name and a timestamp attached.

Rules That Keep The Ladder Boring

Write default in attribute files for almost everything. Put per-group differences in the default block of that group's Policyfile. Reach for override only when you cannot change the cookbook, which in practice means a community cookbook you do not own, and leave a comment saying why. force_default and force_override belong in wrapper cookbooks that have to beat the cookbook they wrap, and nowhere else. Use normal for values a node generates and must keep, knowing you now own their cleanup forever.

Then prove the change on a converged node instead of reading the cookbook and hoping. knife node show NODE -a myapp.port gives you the flattened answer a recipe would see, and node.debug_value('myapp', 'port') tells you which bucket produced it. When those two disagree with what you expected, the ladder has already told you which file to open.

Try this

Run chef install Policyfile.rb on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: your allow-list can end up wider than the file you edited. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related