Data bags & encrypted secrets

Shared data and secrets, encrypted.

A cookbook is a written set of instructions for building a machine, and some values have no business being written into instructions. The list of people who get shell accounts. The token your app posts metrics with. The database password. Bake any of those into a recipe and you have pressed them into an artifact that gets uploaded to the Chef Infra Server, copied down to every node that runs it, cached on disk under /var/chef/cache, and kept forever in that cookbook's version history. Changing the password now means bumping a cookbook version and shipping a release.

A data bag is the filing cabinet standing next to the kitchen instead of inside any one recipe. It lives on the Chef Infra Server, it belongs to no cookbook, and any node can pull a record out of it while it converges. (A converge is one run of chef-client, where the machine is brought into line with what the code says.) The drawer is the bag, which is a namespace and nothing else. The record is an item: a JSON document (JavaScript Object Notation, a plain-text format for structured data) with one mandatory field called id, which is the record's name. That is a different shape from attributes, which describe one node's own values and settle arguments between cookbooks through precedence rules. A data bag item is one record that many nodes read, identically.

The trouble starts with the first password. A plain data bag item is stored on the server as readable JSON and handed to any client that asks for it. So there are three levels to get straight, and the useful part is knowing exactly where each one stops: plain bags, encrypted bags, and chef-vault.

Plain Bags: One Record, Many Readers

Keep the JSON in Git under data_bags/<bag>/<item>.json and upload it with knife, the command-line tool on your workstation that talks to the Chef Infra Server. The id inside the file is what names the item on the server. Making the filename match it is a convention that keeps you sane, not a rule the server enforces. Ids accept letters, digits, dots, dashes and underscores and nothing else, so an email address will not work as an id, and neither will anything with a slash in it.

{
  "id": "charlie",
  "comment": "Charlie Ops",
  "shell": "/bin/bash",
  "ssh_key": "ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH0tK9v3q charlie@laptop"
}

# a bag is only a container: it holds items and nothing else
knife data bag create admins
 
# knife looks under data_bags/admins/ in the repo, so a bare filename works
knife data bag from file admins charlie.json
 
knife data bag list
knife data bag show admins charlie

Created data_bag[admins]
Updated data_bag_item[admins::charlie]
admins
apps
comment:  Charlie Ops
id:       charlie
shell:    /bin/bash
ssh_key:  ssh-ed25519 AAAAC3NzaC1lZDI1NTE5AAAAIH0tK9v3q charlie@laptop

Inside a recipe, data_bag('admins') hands you the list of item ids in that bag, and data_bag_item fetches one record. Each of those calls is its own HTTP request (the web protocol the agent and the server talk over) at converge time. A bag with four hundred items therefore costs four hundred round trips, on every run, on every node. Keep bags small. When you are sweeping up many records rather than naming one, search pulls the whole set back in a single request, because the server indexes data bag items the same way it indexes nodes.

# data_bag() returns the item ids; data_bag_item() fetches one record
# one request per item, so this is fine for ten admins and awful for four hundred
data_bag('admins').each do |name|
  admin = data_bag_item('admins', name)
 
  user admin['id'] do
    comment     admin['comment']
    shell       admin['shell']
    manage_home true
    action      :create
  end
 
  directory "/home/#{admin['id']}/.ssh" do
    owner admin['id']
    group admin['id']
    mode  '0700'
  end
 
  file "/home/#{admin['id']}/.ssh/authorized_keys" do
    owner   admin['id']
    group   admin['id']
    mode    '0600'
    content "#{admin['ssh_key']}\n"
  end
end
 
# the one-request version: search returns the item bodies, not only the ids
# search(:admins, '*:*').each do |admin|
#   ...
# end

Every Node Can Read Every Bag

Here is the fact that decides how you use everything below it. On a Chef Infra Server, the default permissions on data bags grant read to the clients group, and every node in the organization is a member of that group. A node's private key at /etc/chef/client.pem is therefore enough to read every data bag item you have ever uploaded, not only the ones its own run-list touches. (The run-list is the ordered set of recipes a node is told to run.) You can prove that in about thirty seconds, because knife will happily authenticate as any client whose key you hand it.

# a node's own key, copied off web01 the way anyone with root there would
knife data bag list                 -u web01 -k /tmp/web01-client.pem
knife data bag show payments stripe -u web01 -k /tmp/web01-client.pem

admins
apps
payments
api_key:  sk_live_51NxQ2ZLkd8fJ3mTbP
id:       stripe

web01 serves web pages. Its run-list has never mentioned payments. It read the live billing key anyway. You can take that access away one bag at a time with the knife-acl plugin (chef gem install knife-acl if your workstation does not have it yet), using knife acl remove group clients data payments read. Then you own the job of granting it back, bag by bag, node by node, forever, as the fleet changes underneath you. Almost nobody keeps that up. Which is why the encryption below is doing the real work here, and the permissions are not.

Encrypted Data Bags: One Key, Copied Everywhere

Encrypting an item is like sealing each value in its own envelope and then filing the envelopes in a folder whose tab is still printed in large letters. You generate one shared secret, knife encrypts against it on your workstation, and every node that needs to read the item has to hold a copy of that same file. By convention it sits at /etc/chef/encrypted_data_bag_secret.

# 512 random bytes, base64-encoded, with line feeds stripped so the file
# survives a trip through an editor or a copy-paste between platforms
openssl rand -base64 512 | tr -d '\r\n' > encrypted_data_bag_secret
chmod 0600 encrypted_data_bag_secret
 
# encrypt on upload: the JSON on disk is plaintext, the item on the server is not
knife data bag create passwords
knife data bag from file passwords mysql.json \
  --secret-file encrypted_data_bag_secret

Created data_bag[passwords]
Updated data_bag_item[passwords::mysql]

# what the server actually stores
knife data bag show passwords mysql -F json
 
# the same item, read with the key
knife data bag show passwords mysql --secret-file encrypted_data_bag_secret

WARNING: Encrypted data bag detected, but no secret provided for decoding. Displaying encrypted data.
{
  "id": "mysql",
  "password": {
    "encrypted_data": "6q1TfLcqUiN4rZ+RfSjEHOwZ8zLg0EpKcvTb0Xn5JgQ=\n",
    "iv": "Kx8mQeR0vYs2Tt1h\n",
    "auth_tag": "9dLNhFTfHTL0nFXECjMhcQ==\n",
    "version": 3,
    "cipher": "aes-256-gcm"
  }
}
Encrypted data bag detected, decrypting with provided secret.
id:        mysql
password:  7Hq!pR2vX9wKm

Read that JSON slowly, because four things in it matter. version: 3 and cipher: aes-256-gcm mean Chef Infra Client 18 used AES (Advanced Encryption Standard) at 256 bits in GCM (Galois/Counter Mode, which encrypts the value and at the same time produces a short tag proving nobody edited the ciphertext afterwards). That tag is the auth_tag field. Older items you inherit may say version 1, which is AES-256 in CBC (Cipher Block Chaining) with no tamper check at all, or version 2, which bolts an HMAC (Hash-based Message Authentication Code, a keyed fingerprint) on the side. You pin what you write with data_bag_encrypt_version and refuse to read the weak old formats with data_bag_decrypt_minimum_version.

The fourth thing is what is missing. id and password are sitting there in the clear. Encryption covers values, one value at a time, and never covers key names or the item id. A bag called payments holding an item called stripe with a field called api_key tells anyone who can list it exactly what they have found and exactly which key to go hunting for. Name your bags boringly.

Underneath, the contents of the secret file get their surrounding whitespace trimmed off and are then run through SHA-256 (Secure Hash Algorithm, 256-bit output) to produce the 32-byte AES key. The key is always 32 bytes no matter how big that file is. Generating 512 bytes is about having plenty of randomness to hash, not about a longer key. What actually matters is that a single file, copied to every node that needs any secret, opens every item encrypted with it. That file is the whole security boundary.

Chef Infra Client finds the key at /etc/chef/encrypted_data_bag_secret on its own, because that path is the built-in default whenever the file exists. Spell it out in client.rb anyway when the key lives somewhere else, and to leave a note for the next person.

chef_server_url  "https://chef.acme.internal/organizations/acme"
node_name        "web01"
policy_name      "acme_web"
policy_group     "production"
 
# where this node keeps its copy of the shared key (mode 0600, owned by root)
encrypted_data_bag_secret "/etc/chef/encrypted_data_bag_secret"
 
# refuse to decrypt anything written in the weaker legacy formats
data_bag_decrypt_minimum_version 3

# 1) the key is at the default path, so this decrypts with no extra arguments
mysql = data_bag_item('passwords', 'mysql')
mysql['password']
 
# 2) a node holding one key per team has to say which one to use.
#    load_secret trims trailing whitespace before the SHA-256 step; plain
#    IO.read does not, and one newline added by an editor derives a
#    different key and fails to decrypt with a confusing error.
secret = Chef::EncryptedDataBagItem.load_secret('/etc/chef/payments_secret')
stripe = data_bag_item('payments', 'stripe', secret)
 
# 3) the long-hand call, identical result to 2
stripe = Chef::EncryptedDataBagItem.load('payments', 'stripe', secret)
stripe['api_key']

Now the honest weakness. That secret file has to reach every node before the node can decrypt anything, which is a chicken-and-egg problem you solve outside of Chef. knife bootstrap will place the file for you, but only if you pass --secret-file FILE or --secret SECRET, and it says nothing at all if you forget. Rotating the key means re-encrypting every item and touching every node, in an order where nothing is broken in the middle. In practice, teams rotate it once, discover how painful that is, and never rotate it again.

chef-vault: One Envelope Per Name on the Signature Card

A bank does not hand every safe-deposit box holder the same key. It cuts one key per person and seals each copy in an envelope only that person can open. chef-vault works that way. It mints a fresh random shared secret for the item, encrypts the values with it once, then encrypts that shared secret separately for every client and every user you name, using RSA (a public-key scheme where each identity holds a matched pair of keys: a public half the server keeps and a private half that never leaves the machine it belongs to). Those wrapped copies live in a second data bag item, named after the first with _keys on the end. Nothing new has to be shipped to a node, because a node opens its own envelope with the /etc/chef/client.pem it was born with.

# encrypt to every node the search matches, plus two named humans.
# -J reads the values from a file so the password stays out of shell history.
knife vault create secrets mysql -J mysql-creds.json \
  --search 'policy_name:acme_mysql AND policy_group:production' \
  --admins 'alice,bob' \
  --mode client
 
knife vault show secrets mysql --mode client

id:        mysql
password:  7Hq!pR2vX9wKm

Pass --mode client every single time. The chef-vault knife plugin defaults to solo mode, which reads and writes a data_bags/ directory on your laptop instead of talking to the Chef Infra Server, and the failure is quiet: you get a vault item that no node will ever see. Put knife[:vault_mode] = 'client' in your workstation config so the flag stops being something you can forget. Put your own user name in --admins too, or you will write a vault you cannot read back. Add -p all to that show command and it prints the stored search and the member lists alongside the values.

# the mechanism lives in the second item. jq is a small command-line tool for
# picking JSON apart; 'keys' just lists the field names.
knife data bag show secrets mysql_keys -F json | jq 'keys'

[
  "admins",
  "alice",
  "bob",
  "clients",
  "db01",
  "db02",
  "id",
  "mode",
  "search_query"
]

Four of those nine field names are identities, and each one holds the item's shared secret encrypted with that identity's public key and base64-encoded. admins, clients and search_query are the bookkeeping: who was added, and the query that produced them. mode here means how the wrapped keys are stored, either default (all of them inside mysql_keys) or sparse (one small item per client, which is what you want once a vault covers a few hundred nodes and that single item gets slow and fat). Everything in there is public information as far as the server is concerned. Without a private key that never leaves the machine it belongs to, it is a list of names and a pile of base64.

The Failure You Will Meet First

Vault membership is a photograph, not a live feed. The search runs on your workstation at the moment you create or update the item, and the answer is frozen into mysql_keys. A node built tomorrow will match that search perfectly and still be locked out, because its public key did not exist when the photograph was taken. The first converge that needs the secret dies at compile time, before a single resource runs.

# first run on a freshly bootstrapped db03
sudo chef-client

Starting Chef Infra Client, version 18.4.12
Patents: https://www.chef.io/patents
Infra Phase starting
Using policy 'acme_mysql' at revision '4f2c8a1d9e7b3c05'
Resolving cookbooks for run list: ["acme_mysql::server"]
Synchronizing cookbooks:
  - acme_mysql (2.1.0)
Installing cookbook gem dependencies:
Compiling cookbooks...
 
================================================================================
Recipe Compile Error in /var/chef/cache/cookbooks/acme_mysql/recipes/server.rb
================================================================================
 
ChefVault::Exceptions::SecretDecryption
---------------------------------------
secrets/mysql is not encrypted with your public key.  Contact an administrator of the vault item to encrypt for you!
 
Cookbook Trace: (most recent call first)
----------------------------------------
  /var/chef/cache/cookbooks/acme_mysql/recipes/server.rb:4:in `from_file'
 
Relevant File Content:
----------------------
/var/chef/cache/cookbooks/acme_mysql/recipes/server.rb:
 
  1:  require 'chef-vault'
  2:
  3:  # membership was decided the last time an admin wrote this item
  4>> creds = ChefVault::Item.load('secrets', 'mysql')
  5:
 
System Info:
------------
chef_version=18.4.12
platform=ubuntu
platform_version=24.04
ruby=ruby 3.1.6p260 (2024-05-29 revision a777087be6) [x86_64-linux]
program_name=/opt/chef/bin/chef-client
executable=/opt/chef/bin/chef-client
 
Running handlers:
[2026-07-21T09:14:02+00:00] ERROR: Running exception handlers
Running handlers complete
[2026-07-21T09:14:02+00:00] ERROR: Exception handlers complete
Chef Infra Client failed. 0 resources updated in 03 seconds

The fix is to take the photograph again, and the way to know it worked is to look at who holds an envelope now rather than trusting that the command said nothing. knife vault refresh re-runs the search already stored in the item. knife vault update is what you use when the search itself or the values need to change. Better still, hand a node its envelope while it is being built: knife bootstrap takes --bootstrap-vault-item 'vault:item', repeatable, or --bootstrap-vault-json for several at once, and adds the new client to those vault items as part of the same command, so the first converge already has what it needs.

# db03 was built after the vault was written, so re-run the stored search
knife vault refresh secrets mysql --mode client
 
# verify: db03 should now hold a wrapped copy of its own
knife data bag show secrets mysql_keys -F json | jq 'keys'

[
  "admins",
  "alice",
  "bob",
  "clients",
  "db01",
  "db02",
  "db03",
  "id",
  "mode",
  "search_query"
]

# for the next node, do it at build time instead of after the first failed run
knife bootstrap db04.acme.internal \
  --node-name db04 \
  --policy-name acme_mysql --policy-group production \
  --bootstrap-vault-item 'secrets:mysql' \
  --connection-user ops --sudo

Keep the Secret Out of the Run Log

Decrypted is decrypted. Once a recipe holds that password it is an ordinary Ruby string, and the cipher is almost never what leaks it. Chef's own file diff is. The template and file providers print a line-by-line comparison of what changed, and a password is exactly the kind of line that changes.

# the recipe below, but without the sensitive property
sudo chef-client

Recipe: acme_mysql::server
  * template[/etc/mysql/conf.d/app.cnf] action create
    - update content in file /etc/mysql/conf.d/app.cnf from 4c9e1a to b7f320
    --- /etc/mysql/conf.d/app.cnf	2026-07-21 09:22:11.402931755 +0000
    +++ /etc/mysql/conf.d/.chef-app20260721-4412-1qk9xw.cnf	2026-07-21 09:22:11.398931755 +0000
    @@ -1,3 +1,3 @@
     [client]
     user=app
    -password=old-pw
    +password=7Hq!pR2vX9wKm
  * service[mysql] action enable (up to date)
  * service[mysql] action start (up to date)
  * service[mysql] action restart
    - restart service service[mysql]
 
Running handlers:
Running handlers complete
Chef Infra Client finished, 2/2 resources updated in 06 seconds

There is the production database password in the converge log, on the node, in the system journal, and in whatever collector ships that output somewhere central. Nothing was misconfigured. The diff did the one thing diffs are for. sensitive true is the fix, and it belongs on any resource whose content or arguments carry a credential.

require 'chef-vault'
 
# In Test Kitchen there is no Chef Infra Server and no client key, so fall
# back to a plain fixture bag under test/fixtures/data_bags/ with a fake value.
creds = if ChefVault::Item.vault?('secrets', 'mysql')
          ChefVault::Item.load('secrets', 'mysql')
        else
          data_bag_item('secrets', 'mysql')
        end
 
template '/etc/mysql/conf.d/app.cnf' do
  source    'app.cnf.erb'
  owner     'root'
  group     'mysql'
  mode      '0640'
  variables(password: creds['password'])
  sensitive true                 # no diff, no value, nothing in the run log
  notifies  :restart, 'service[mysql]', :delayed
end
 
service 'mysql' do
  action [:enable, :start]
end

# same run, one property added
sudo chef-client

Recipe: acme_mysql::server
  * template[/etc/mysql/conf.d/app.cnf] action create
    - update content in file /etc/mysql/conf.d/app.cnf from 4c9e1a to b7f320
    (suppressed sensitive resource)
  * service[mysql] action enable (up to date)
  * service[mysql] action start (up to date)
  * service[mysql] action restart
    - restart service service[mysql]
 
Running handlers:
Running handlers complete
Chef Infra Client finished, 2/2 resources updated in 06 seconds

Notice what survived: the resource name, the action, and both content checksums. That is enough to see that something changed and to bisect a bad run, without the value. sensitive covers the log and nothing else. The rendered file still holds the password as plain text at mode 0640, which is why that mode and that group are part of the security design and not decoration, and it will be copied into every backup and every disk image you take of the host.

Rotation, and What Removal Does Not Do

Nodes get decommissioned. Deleting the client on the server does not touch the vault, so the tidy-up is a second step. A plain refresh trips over a member that no longer exists on the server, and --clean-unknown-clients is what tells it to drop those members instead.

# db01 was decommissioned this morning
knife client delete db01 --yes
 
# re-run the stored search and drop members the server has never heard of
knife vault refresh secrets mysql --clean-unknown-clients --mode client
 
knife data bag show secrets mysql_keys -F json | jq 'keys'

Deleted client[db01]
[
  "admins",
  "alice",
  "bob",
  "clients",
  "db02",
  "db03",
  "id",
  "mode",
  "search_query"
]

db01 is off the list. That changes nothing about what db01 already knows. It has had the plaintext password in memory, and in /etc/mysql/conf.d/app.cnf, since the day it was built, and if that disk went to a recycler the password went with it. knife vault rotate keys secrets mysql --mode client mints a new shared secret for the item and re-wraps it for the current members, which kills any wrapped copy someone squirreled away, and knife vault rotate all keys does that across every vault after an admin leaves. Neither one changes the password. The only thing that revokes a leaked credential is changing it at the system that issued it, updating the vault item with the new value, then converging the nodes that use it. In that order.

Be clear about the ceiling you are working under. chef-vault is a good answer to key distribution and a poor answer to secret management. There is no record of which node read which secret and when, no expiry, no short-lived credentials, and anyone holding the Chef Infra Server's superuser key can add themselves as an admin to any vault and re-encrypt it to their own key. Compromise the server and you have the fleet's secrets plus the ability to run arbitrary code as root everywhere, which is a bad afternoon by any measure. (CINC, the community rebuild of Chef Infra from the same source with the trademarks stripped out, carries identical data bag and chef-vault code, so none of this changes on cinc-client.) If you keep exactly one thing in a vault item, make it the credential a node uses to fetch everything else from a system that expires secrets and logs the reads.

Try this

Run knife data bag create admins on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a data bag edit skips every control your cookbooks have. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related