Hiera & encrypted data

Separate data from code; eyaml secrets.

A recipe card in a restaurant chain says two spoons of the house blend. Every branch gets the same card. What sits in the jar labelled house blend is decided branch by branch, and the cook reaches for the nearest jar first: the one on their own bench, then the shared shelf, then the storeroom. Puppet is built the same way. Manifests are the recipe cards, and Hiera is the shelf system that answers one question for every value in them: what does this particular machine get?

Hiera is a lookup engine sitting on a ranked list of data files, usually YAML (a plain-text format for keys, values and lists). You ask for a key. Hiera walks the list from the most specific level (this one node) down to the most general (defaults for the whole estate), and the first level that holds the key wins. The manifest never changes. It asks for profile::nginx::port and a value comes back, whether that value was set for this node, for its role, for its operating system family, or for everyone.

The same machinery that hands a node its worker count hands it the production database password, and that is the part worth slowing down for. Hiera has no per-key permissions and no access-control list. What decides which machine may receive which secret is the shape of the hierarchy and the variables you paste into those file paths. Get one variable wrong and a lab box compiles a catalog containing production credentials. Nothing errors. Nothing alerts. The run succeeds, for the wrong machine.

The Shelf: Hierarchy and First Match

---
version: 5
 
defaults:
  datadir: data           # relative to the environment directory
  data_hash: yaml_data    # plain YAML files
 
hierarchy:
  - name: "Per-node data"
    path: "nodes/%{trusted.certname}.yaml"
 
  - name: "Per-role data"
    path: "roles/%{facts.role}.yaml"
 
  - name: "Per-OS defaults"
    path: "os/%{facts.os.family}.yaml"
 
  - name: "Common"
    path: "common.yaml"

Read it top to bottom, most specific first. The %{...} tokens are interpolation: Puppet substitutes the node's own data into the path before it goes looking for the file, so a single hierarchy entry becomes a different filename for every node. A missing file is not an error. Hiera skips it and moves down. If an interpolated variable is empty or undefined, Hiera drops that whole level rather than hunting for a file called roles/.yaml.

Look hard at which variables appear in those paths, because they are not equally trustworthy. One is a name badge the visitor fills in themselves at the door. The other is a photo ID the guard checked before letting them through. %{facts.os.family} and %{facts.role} come from Facter (Puppet's fact-gathering tool) running on the node, and the node itself decides what to report. %{trusted.certname} is read out of the TLS certificate (the signed identity file the agent proves itself with over an encrypted connection) that the agent authenticated with, and no node can rewrite that. Hold on to the difference. The last section takes that per-role level apart.

---
profile::nginx::workers: 4
profile::nginx::port: 80
profile::ssh::allowed_groups:
  - sysadmins
 
lookup_options:
  profile::ssh::allowed_groups:
    merge: unique     # lower levels ADD to this list instead of replacing it

---
profile::nginx::port: 8080     # replaces 80 outright (first match wins)
profile::ssh::allowed_groups:
  - contractors                # unique merge: appended, not substituted

The default merge behaviour is first: the most specific level holding the key wins, and everything below it is ignored. lookup_options changes that per key. A door list can work two ways. Either tonight's list replaces last night's, or every list ever written gets added together. unique flattens arrays and drops duplicates, hash merges the top level of a hash, and deep merges nested hashes too. Picking between them is a security decision as much as a formatting one. Under first, a per-node file can only replace a list of allowed groups, and a reviewer sees the replacement sitting there. Under unique or deep, a per-node file can append to that list, so anyone who can edit node data adds an entry to a list another team owns without ever touching the line that team reviewed.

Class Parameters That Fill Themselves In

Automatic parameter lookup is what makes Hiera feel invisible. It works like a form at a clinic desk that arrives with your details already printed in: you only write in the boxes you want to change. When a class is declared, every parameter you did not pass by hand is looked up under the key <class name>::<parameter name>. A parameter called $port inside class profile::nginx is fed by the Hiera key profile::nginx::port. There is no lookup call anywhere in the manifest. (The profile:: prefix is the roles-and-profiles convention: a profile is a class that sets up one technology the way your shop wants it, and a role is the short list of profiles that makes a machine what it is.)

class profile::nginx (
  Integer[1, 65535] $port    = 80,   # default, used only if Hiera has nothing
  Integer[1]        $workers = 2,
) {
  file { '/etc/nginx/conf.d/tuning.conf':
    ensure  => file,
    # epp() renders an Embedded Puppet template with the values you hand it
    content => epp('profile/nginx-tuning.epp',
                   { 'port' => $port, 'workers' => $workers }),
  }
}

Resolution runs in a fixed order: a value passed explicitly in a resource-like declaration, then Hiera, then the default written into the class, then a compile failure if there is nothing anywhere. Two details bite people. Automatic lookup applies to classes only, so a defined type has to call lookup() itself. And a resource-like declaration such as class { 'profile::nginx': port => 80 } switches Hiera off for that parameter, which is how a hard-coded value buried in a profile beats the data file a colleague edited an hour ago and swears is correct.

Prove Which Level Actually Won

Guessing where a value came from is how afternoons disappear. puppet lookup runs the real lookup from the command line, for any node you name, and --explain prints the whole search: every path it tried, every path it skipped, and the level that finally answered.

$ sudo puppet lookup profile::nginx::port \
    --node web1.acme.internal \
    --environment production --explain

[ the preceding search for "lookup_options" is trimmed ]
Searching for "profile::nginx::port"
  Global Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/puppet/hiera.yaml"
    Hierarchy entry "Common"
      Path "/etc/puppetlabs/puppet/data/common.yaml"
        Original path: "common.yaml"
        Path not found
  Environment Data Provider (hiera configuration version 5)
    Using configuration "/etc/puppetlabs/code/environments/production/hiera.yaml"
    Hierarchy entry "Per-node data"
      Path "/etc/puppetlabs/code/environments/production/data/nodes/web1.acme.internal.yaml"
        Original path: "nodes/%{trusted.certname}.yaml"
        Found key: "profile::nginx::port" value: 8080

--node decides whose certname and facts are used, which is the entire point of a node-shaped hierarchy. On a primary server it fetches that node's facts from PuppetDB (Puppet's store of facts, catalogs and reports). If PuppetDB is not wired up, or you want to test a machine that does not exist yet, --facts <file> supplies them from a JSON or YAML file instead. Drop --explain and you get the value on its own, which is the fastest way to watch a merge happen.

$ sudo puppet lookup profile::ssh::allowed_groups --node web1.acme.internal

---
- contractors
- sysadmins

Notice the Global Data Provider at the top of that explain output, reading /etc/puppetlabs/puppet/hiera.yaml. That layer is searched before your environment's hierarchy, on every lookup. A stray key left there silently overrides every environment on that server, including the one you are testing in. Leave the global layer empty unless you have a specific reason not to, and check it first when a value refuses to change no matter what you edit.

eyaml: Ciphertext You Can Commit

A locked drop-box in a post-office lobby takes envelopes from anyone through the slot, and only the key holder at the back can open it. Public-key cryptography works like that, and hiera-eyaml (encrypted YAML) applies it one value at a time. Encrypting needs only the public key, so any engineer's laptop can add a secret. Decrypting needs the private key, and that lives on exactly one class of machine: whatever compiles catalogs.

The gem (a packaged Ruby library) has to be installed twice, into two different Rubies, which is the first thing nearly everyone gets wrong. The eyaml command line tool goes into Puppet's own Ruby so humans can encrypt values. The same library also has to go into Puppet Server's JRuby (Ruby running inside the Java virtual machine, with its own separate shelf of gems), because that process is the one decrypting during compilation. New gems are picked up when the JVM starts, so restart the service rather than reloading it. The same two-step applies on OpenVox, the community fork that appeared after the Puppet 8 licence change.

# 1. the CLI tool, in Puppet's Ruby (gives you the `eyaml` command)
$ sudo /opt/puppetlabs/puppet/bin/gem install hiera-eyaml
 
# 2. the library, inside Puppet Server's JRuby (this is what decrypts at compile time)
$ sudo puppetserver gem install hiera-eyaml
$ sudo systemctl restart puppetserver

Fetching highline-2.1.0.gem
Fetching optimist-3.1.0.gem
Fetching hiera-eyaml-4.2.0.gem
Successfully installed highline-2.1.0
Successfully installed optimist-3.1.0
Successfully installed hiera-eyaml-4.2.0
[ documentation lines trimmed ]
3 gems installed

Now make the key pair. eyaml createkeys writes a 2048-bit RSA private key (RSA is the classic public-key algorithm) and a matching public certificate into ./keys, relative to wherever you are standing. Run it outside your code repository and move the result into place.

$ cd /etc/puppetlabs/puppet
$ sudo /opt/puppetlabs/puppet/bin/eyaml createkeys
$ sudo mv keys eyaml
$ sudo ls -l /etc/puppetlabs/puppet/eyaml

[hiera-eyaml-core] Created key directory: ./keys
[hiera-eyaml-core] Keys created OK
total 8
-rw------- 1 root root 1704 Jul 21 10:14 private_key.pkcs7.pem
-rw-r--r-- 1 root root 1050 Jul 21 10:14 public_key.pkcs7.pem

Here is a trap that costs people an hour. The eyaml command never reads hiera.yaml. It looks for its own config file, and with none it falls back to ./keys/private_key.pkcs7.pem relative to your current directory, then fails with a missing-key error the moment you run it from anywhere else. Write the config once and stop passing key paths on every command.

---
# read by the eyaml CLI only; Puppet Server gets its paths from hiera.yaml
pkcs7_public_key:  /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem

Engineers who only need to add secrets keep the same file at ~/.eyaml/config.yaml on their own laptop, pointing at a copy of the public key and nothing else. That is the whole reason to use a key pair: adding a secret needs no access to the machine that can read them.

Then point Hiera at the keys. lookup_key: eyaml_lookup_key swaps the backend for those paths only. The eyaml_lookup_key function ships inside Puppet 8 itself, and the gem supplies the cryptography behind it. Because it is a lookup_key function rather than a data_hash one, Hiera asks it for a single key at a time, so only the values a node genuinely needs ever get decrypted.

hierarchy:
  - name: "Per-node secrets"
    lookup_key: eyaml_lookup_key
    path: "nodes/%{trusted.certname}.eyaml"
    options: &eyaml_keys
      pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
      pkcs7_public_key:  /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
 
  - name: "Per-role secrets"
    lookup_key: eyaml_lookup_key
    path: "roles/%{facts.role}.eyaml"
    options: *eyaml_keys        # YAML anchor: same two keys, written once
 
  # ... the plain-YAML levels from earlier follow here ...

Encrypting a value takes the label it will carry as a Hiera key. Use -p to be prompted for the secret rather than -s 'secret', which writes the plaintext straight into your shell history where it will sit for months. The default output prints the same ciphertext twice, once as a single line and once folded into a block; -o block prints only the second form, which is the one that belongs in a data file.

$ /opt/puppetlabs/puppet/bin/eyaml encrypt -l 'profile::db::password' -p

Enter password: **************
string: ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAACAQEwDQYJKoZIhvcNAQEBBQAEggEAvJ3nQ1Fyq9mE1u0oR6Yy0Fh0K9r3sQ0Xh8mJ7d2bV1oU3xS6cQeGm4pZ0tR9lN8wA1dK5jP2vX7yB4hC6sT0nM3fW8gE1aL9kO2iD5uR7pQ4xJ6bN0cV3zY8mH1sG5tK2wF4eA7rP9dX0jB6vC3nL8qS1yT5uI2oE4aM7fZ0=]
 
OR
 
block:
    profile::db::password: >
        ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAAC
        AQEwDQYJKoZIhvcNAQEBBQAEggEAvJ3nQ1Fyq9mE1u0oR6Yy0Fh0K9r3sQ0Xh8mJ7d
        7pQ4xJ6bN0cV3zY8mH1sG5tK2wF4eA7rP9dX0jB6vC3nL8qS1yT5uI2oE4aM7fZ0=]

---
profile::db::host: db1.acme.internal    # plaintext and ciphertext share the file
profile::db::password: >
    ENC[PKCS7,MIIBiQYJKoZIhvcNAQcDoIIBejCCAXYCAQAxggEhMIIBHQIBADAFMAAC
    AQEwDQYJKoZIhvcNAQEBBQAEggEAvJ3nQ1Fyq9mE1u0oR6Yy0Fh0K9r3sQ0Xh8mJ7d
    7pQ4xJ6bN0cV3zY8mH1sG5tK2wF4eA7rP9dX0jB6vC3nL8qS1yT5uI2oE4aM7fZ0=]

Only the value is ciphertext. Key names, comments and every non-secret value beside it stay readable, so a reviewer can see that profile::db::password changed without being able to read what it changed to. That is the practical gap between eyaml and encrypting a whole file. Editing later is eyaml edit, which decrypts into your editor, shows each secret inside DEC(n)::PKCS7[...]! markers, and re-encrypts on save. The decrypted copy exists as a temporary file for as long as your editor is open, so do that on a machine you would trust with the secret anyway.

$ export EDITOR=vim
$ sudo -E /opt/puppetlabs/puppet/bin/eyaml edit \
    data/nodes/web1.acme.internal.eyaml

#| This is eyaml edit mode. This text (lines starting with #| at the top of the
#| file) will be removed when you save and exit.
#|  - To edit encrypted values, change the contents of the DEC(<num>)::PKCS7[]!
#|    block.
#|    WARNING: DO NOT change the number in the parentheses.
[ banner trimmed ]
---
profile::db::host: db1.acme.internal
profile::db::password: DEC(1)::PKCS7[pr0d-Rot4te-Me]!

Where the Plaintext Actually Goes

eyaml protects data at rest in the repository. It does not make the secret vanish on the way to the machine that needs it. Follow one value: decrypted in memory on the compiling server, written into the catalog (the compiled, node-specific list of resources), sent over TLS to the agent, applied, and cached on the node's own disk as JSON (a plain-text data format).

Here is the class that consumes it. A shop receipt stars out most of your card number while the card itself is still in your pocket, unchanged. Sensitive() does the receipt half: it tells Puppet to redact that value in run output, in file diffs and in the report it sends to PuppetDB.

class profile::db (
  String[1]    $password,                      # no default: Hiera must supply it
  Stdlib::Host $host = 'db1.acme.internal',    # Stdlib::Host comes from puppetlabs-stdlib
) {
  file { '/etc/app/db.conf':
    ensure  => file,
    owner   => 'root',
    mode    => '0600',
    content => Sensitive(epp('profile/db.conf.epp',
                             { 'host' => $host, 'password' => $password })),
  }
}

The first agent run after all of this usually fails, and the error is worth recognising on sight. createkeys left the private key owned by root, but Puppet Server runs as the puppet user, so the decryption call cannot open the file.

$ sudo puppet agent -t

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error:
  Evaluation Error: Error while evaluating a Function Call, Permission denied @ rb_sysopen -
  /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem (file: /etc/puppetlabs/code/environments/
  production/site-modules/role/manifests/database.pp, line: 4, column: 3) on node web1.acme.internal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

$ sudo chown -R puppet:puppet /etc/puppetlabs/puppet/eyaml
$ sudo chmod 0500 /etc/puppetlabs/puppet/eyaml
$ sudo chmod 0400 /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
$ sudo chmod 0444 /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
$ sudo puppet agent -t

Info: Using environment 'production'
Info: Caching catalog for web1.acme.internal
Info: Applying configuration version '1784628960'
Notice: /Stage[main]/Profile::Db/File[/etc/app/db.conf]/content: content changed [redacted] to [redacted]
Notice: Applied catalog in 4.13 seconds

That [redacted] is real protection against the most common secret leak in configuration management, which is a password ending up in a build log, a report, or a screenshot of a terminal. It is a logging control. It says nothing about what is on the disk of the node itself, and the catalog cache proves it. Read that cache with jq (a command-line JSON reader).

$ sudo jq '.resources[] | select(.title == "/etc/app/db.conf") | .parameters.content' \
    /opt/puppetlabs/puppet/cache/client_data/catalog/web1.acme.internal.json

{
  "__ptype": "Sensitive",
  "__pvalue": "host=db1.acme.internal\npassword=pr0d-Rot4te-Me\n"
}

Blast radius, meaning how much one break gets you, is the thing you control here. A node only receives values that its own hierarchy levels resolved to, so a compromised web server should give up the web server's secrets and nothing more. Put everything in one common.eyaml that every node reads, and a single compromised host reads the entire estate's credentials out of a JSON file. Keep secrets on the narrowest level that works, usually per-role or per-node.

A stricter variant types the parameter itself as Sensitive[String[1]] $password and adds convert_to: "Sensitive" for that key under lookup_options, so the value is wrapped the moment Hiera hands it back and no later code can accidentally paste it into a notify message. Declare the type without the convert_to and compilation fails with parameter 'password' expects a Sensitive value, got String, which is baffling for about ten minutes the first time you meet it.

The Hierarchy Is the Access Control

Every ordinary fact a node reports is written by that node. Facter collects most of them from the operating system, and any file dropped into /opt/puppetlabs/facter/facts.d/ becomes a fact too. That directory is writable by root on the node, and root on the node is precisely who you are defending against once a machine is compromised. So a hierarchy level keyed on %{facts.role} is a level where the node fills in its own name badge and the guard waves it through.

# on a compromised lab box: invent a role
$ echo 'role=db-prod' | sudo tee /opt/puppetlabs/facter/facts.d/role.txt
$ puppet facts show role

role=db-prod
{
  "role": "db-prod"
}

# on the primary server: replay that node's lookup with the fact it now claims
$ cat /tmp/spoofed-facts.json
$ sudo puppet lookup profile::db::password \
    --node dev01.acme.internal \
    --facts /tmp/spoofed-facts.json \
    --environment production

{ "role": "db-prod", "os": { "family": "RedHat" } }
--- pr0d-DB-Rot4te-2026

The server decrypted a production secret for a lab machine, because the lab machine claimed a role and the hierarchy believed it. No certificate was forged and no key was stolen. The fix is to key privileged levels on data the node cannot write: %{trusted.certname}, which comes from the signed certificate, or %{trusted.extensions.pp_role}, a certificate extension baked in at signing time.

  # BEFORE: the node decides which secrets it can read
  - name: "Per-role secrets"
    lookup_key: eyaml_lookup_key
    path: "roles/%{facts.role}.eyaml"
 
  # AFTER: the certificate decides, and the node cannot edit its certificate
  - name: "Per-role secrets"
    lookup_key: eyaml_lookup_key
    path: "roles/%{trusted.extensions.pp_role}.eyaml"

$ sudo puppet lookup profile::db::password \
    --node dev01.acme.internal \
    --facts /tmp/spoofed-facts.json \
    --environment production

Error: Function lookup() did not find a value for the name 'profile::db::password'

pp_role gets into a certificate through /etc/puppetlabs/puppet/csr_attributes.yaml, the file that shapes the CSR (certificate signing request, the application a new node sends before it gets a certificate). Your provisioning process writes it before the node's first run, and it is fixed for the life of that certificate. Changing it means revoking and re-issuing. That moves a real security decision to signing time: whoever approves a request carrying pp_role: db-prod is handing over the production database password, which is a very good argument against signing every request that shows up automatically.

---
extension_requests:
  pp_role: db-prod          # baked into the signed cert, unforgeable afterwards
  pp_environment: production

Rotating the Key Does Not Rotate the Secrets

Every value in your repository is encrypted to the same public key, so replacing that key pair means re-encrypting every value. eyaml recrypt does it file by file. It decrypts with the private key you point it at and re-encrypts with the public key you point it at, which lets you run one pass with the old private key and the new public key. The recrypted data and the server's new private key have to land together, so stage the whole swap in a test environment first.

$ sudo /opt/puppetlabs/puppet/bin/eyaml recrypt \
    --pkcs7-private-key /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem \
    --pkcs7-public-key  /etc/puppetlabs/puppet/eyaml-new/public_key.pkcs7.pem \
    data/nodes/web1.acme.internal.eyaml
$ git diff --stat data/

 data/nodes/web1.acme.internal.eyaml | 6 +++---
 1 file changed, 3 insertions(+), 3 deletions(-)

PKCS7 (the standard envelope format eyaml uses) picks a fresh random session key each time, so the ciphertext changes even when the secret does not, and every recrypt produces a diff like that one. Reviewers learn to ignore those diffs, which is exactly when a swapped value slips through. Keep secret changes in their own small commits.

The last point is the one that catches teams during an incident. Rotating the eyaml key pair does not rotate a single password. Old ciphertext is still in Git history, every clone taken months ago still holds it, and anyone who ever had the private key can still read all of it. If that key leaks, the only real remediation is changing the actual passwords, tokens and keys it protected, in the systems that accept them. Re-encrypting the repository is the housekeeping you do afterwards.

Try this

Run sudo puppet lookup profile::ssh::allowed_groups --node web1.acme.internal on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: sensitive() hides values from logs, not from the node. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related