The catalog: compile & apply

How a run actually works.

A tailor keeps one paper pattern for a jacket and cuts a different set of pieces for every customer who walks in. The pattern never leaves the shop. What reaches the cutting table is a plan for one body: these panels, this length, in this order. Puppet works the same way. Your manifests (the .pp files that hold your Puppet code) are the pattern. The measurements are facts, gathered on each machine by a program called Facter: operating system, memory, network addresses, disks. What the machine actually receives is neither of those. It is a catalog: one flat, node-specific document naming every resource this one node should have, with the final value of every setting already filled in.

The agent never runs your .pp files. It has never seen them. It runs the catalog and nothing else. That one split, compile on the server and apply on the node, explains most of what you will ever debug in Puppet: where an error shows up, why the machine running your code is not the machine your code describes, and why two nodes handed identical manifests can end up with wildly different catalogs. It also draws a security line straight through the system. Everything that decides anything happens on the Puppet Server, as the unprivileged puppet user. Everything that changes anything happens on the node, as root.

What one run actually does

Left alone, the agent wakes every 30 minutes. That interval is the runinterval setting. Splay, a random delay so a thousand nodes do not all knock on the door in the same second, is switched off by default; turn it on with splay = true and bound it with splaylimit. Then a short sequence runs. The agent takes a run lock, which is one file at /opt/puppetlabs/puppet/cache/state/agent_catalog_run.lock, so two runs can never overlap. It asks the server which environment it belongs to. It downloads code from every module in that environment over HTTPS: custom facts, custom functions, custom resource types and providers. It runs Facter and gathers facts. It posts those facts, under its certificate name, to the Puppet Server and asks for a catalog. Then it waits, because the next part is not its job.

The server does all the thinking. It classifies the node (a node block in site.pp, or an external node classifier, which is a script or service the server asks "what should this machine be?" and which hands back a role), then evaluates the matching manifests with the node's facts bound as variables. Every conditional is decided here. Every lookup() into Hiera 5 (Puppet's data layer: a stack of YAML files in your control repository, searched from most specific to most general) resolves here, so every eyaml-encrypted secret is decrypted here, on the server. Eyaml is the Hiera backend that keeps values as ciphertext in those same YAML files, safe to commit to git. Every EPP or ERB template (Embedded Puppet and Embedded Ruby, Puppet's two template languages) is rendered here. What comes out is a JSON document: a list of resources, each carrying its final parameter values, plus a containment tree recording which class holds which resource. No if, no function call, no lookup survives into the catalog. Only their answers do. The single exception is a Deferred value, which is written down on purpose as an instruction the agent resolves later.

Two of those inputs deserve very different levels of belief, and mixing them up is a vulnerability rather than a style problem. $facts come from the node, which means from a machine somebody else may already own. Root on that box can drop a custom fact into /opt/puppetlabs/facter/facts.d/ reporting anything at all: a different operating system family, a hostname belonging to a server two racks over. $trusted comes from the certificate your certificate authority signed, and the server reads it out of the authenticated TLS connection (the encrypted session where both ends already proved who they are with certificates) rather than out of the request body. So if a profile says if $facts['networking']['fqdn'] =~ /^bastion/ { include profile::admin_access }, root on any node in the fleet can write that fact, ask for its own catalog through the front door as usual, and get the admin class compiled into it. Branch on $trusted['certname'] instead, or on classification data the node never gets to supply.

That code download deserves a second look, because it happens before any classification. Pluginsync copies the lib directory of every module in the environment to every agent, whether or not that node is classified with the module, and Facter then executes each custom fact it finds, as root, on every run. A module you add to your control repository for one node is Ruby that runs on all of them.

A run you can read line by line

Here is a profile. In the roles-and-profiles pattern, a role says what a machine is and a profile wraps exactly one piece of technology and makes it configurable, so this is the profile layer. Both class parameters are typed and neither has a default, so Puppet fills them by automatic parameter lookup: while compiling, it asks Hiera for profile::app::port and profile::app::db_pass, keys built from the class name. If Hiera has no answer, compilation fails. That is the behaviour you want. A node that cannot be described completely should get no catalog at all rather than half a configuration.

class profile::app (
  Integer[1,65535]  $port,     # Hiera: profile::app::port
  Sensitive[String] $db_pass,  # Hiera + eyaml: profile::app::db_pass
) {
  package { 'nginx':
    ensure => installed,
  }
 
  file { '/etc/nginx/conf.d/app.conf':
    ensure  => file,
    owner   => 'root',
    group   => 'root',
    mode    => '0644',
    content => epp('profile/app.conf.epp', { 'port' => $port }),
    require => Package['nginx'],   # install before configure
    notify  => Service['nginx'],   # reload only if this file changed
  }
 
  file { '/etc/app':
    ensure => directory,
    owner  => 'root',
    group  => 'root',
    mode   => '0750',
  }
 
  file { '/etc/app/db.conf':
    ensure    => file,
    owner     => 'root',
    group     => 'root',
    mode      => '0600',
    show_diff => false,          # belt and braces; Sensitive already hides it
    content   => Sensitive("password=${$db_pass.unwrap}\n"),
    require   => File['/etc/app'],
  }
 
  service { 'nginx':
    ensure => running,
    enable => true,
  }
}

Two details in that file are easy to get wrong. First, typing a parameter Sensitive[String] converts nothing by itself. Eyaml decrypts to a plain string, so give that key lookup_options with convert_to: 'Sensitive' in your Hiera data and the value arrives already wrapped. Second, require => Package['nginx'] and notify => Service['nginx'] do not become edges in the catalog's edge list. They travel down as ordinary parameters on the file resource, written as the strings Package[nginx] and Service[nginx], and the agent turns them into a real dependency graph in memory when the apply starts. Hold on to that, because it decides where a dependency cycle blows up. Now say you changed the port in Hiera from 8080 to 8443 and merged it. Here is the next run on web01.

# -t is short for --test: one run, right now, in the foreground, verbose
sudo puppet agent -t

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for web01.example.com
Info: Applying configuration version '1784626439'
Notice: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]/content:
--- /etc/nginx/conf.d/app.conf   2026-07-14 11:02:55.412000000 +0000
+++ /tmp/puppet-file20260721-3412-1u9k2q   2026-07-21 09:34:01.884000000 +0000
@@ -1,4 +1,4 @@
 server {
-  listen 8080;
+  listen 8443;
   server_name web01.example.com;
 }
 
Notice: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]/content: content changed '{sha256}5b1f0c...' to '{sha256}9c4ae2...'
Info: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]: Scheduling refresh of Service[nginx]
Notice: /Stage[main]/Profile::App/Service[nginx]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 4.62 seconds

Read that from the top. Using environment 'production' is the environment the server picked, which is not always the one the node asked for. Caching catalog for web01.example.com is the agent writing the catalog to disk before applying it, and that cache is about to matter twice. Applying configuration version '1784626439' is the catalog's version, which by default is the epoch second (a plain count of seconds since 1970) at which the compile finished. Point config_version in environment.conf at a small script that echoes the git commit hash of the deployed code, and every log line and every report then names the exact commit behind it. That turns "when did this node get this config" from an argument into a lookup.

The middle of the run is the interesting part. Puppet printed a unified diff because --test switches show_diff on, and show_diff is off by default. Then it restated the change as a pair of checksums, which are short fingerprints of the file contents. Scheduling refresh of Service[nginx] is the notify relationship firing. Triggered 'refresh' from 1 event is nginx actually reloading. If a change line ever ends in (corrective), read it twice. That suffix means the value on disk differed from the value Puppet itself last set, so something outside Puppet moved it. A normal deploy, where the catalog changed and the machine did not, produces no such suffix. Corrective changes are drift, and drift on a node nobody deployed to is the cheapest intrusion signal you will ever get. If you want to eyeball the current state by hand, puppet resource file /etc/nginx/conf.d/app.conf prints what is on disk right now in Puppet's own syntax.

--test is one flag standing in for seven settings: onetime, verbose, no-daemonize, no-splay, show_diff, no-usecacheonfailure and detailed-exitcodes. Two of those do more than change what you read, and both come back to bite people before the end of this lesson. (If your fleet runs OpenVox, the community fork that appeared after the Puppet 8 licence change, every command and log line here is identical. The packages are named openvox-agent and openvox-server, and the paths under /opt/puppetlabs are unchanged.)

Open the catalog and read it

The agent keeps the last catalog it received on disk, as JSON, which is a plain-text data format anything can read. Opening it is the fastest way to settle an argument about what a node was actually told to do, as opposed to what you believe your code says. The tool below is jq, a command-line JSON reader; install it if it is not already there.

CAT="$(sudo puppet config print client_datadir)/catalog/$(sudo puppet config print certname).json"
sudo ls -l "$CAT"
 
# what the node was told, in one summary
sudo jq '{version, environment, code_id, catalog_uuid,
          resources: (.resources|length), edges: (.edges|length)}' "$CAT"
 
# the first few resources, by type and title
sudo jq -r '.resources[] | "\(.type)[\(.title)]"' "$CAT" | head -9

-rw-rw---- 1 root root 41273 Jul 21 09:34 /opt/puppetlabs/puppet/cache/client_data/catalog/web01.example.com.json
{
  "version": 1784626439,
  "environment": "production",
  "code_id": null,
  "catalog_uuid": "6a1b0d61-8e97-4e5c-9a5e-2b6e5f9a1c30",
  "resources": 47,
  "edges": 46
}
Stage[main]
Class[Settings]
Class[main]
Class[Profile::App]
Package[nginx]
File[/etc/nginx/conf.d/app.conf]
File[/etc/app]
File[/etc/app/db.conf]
Service[nginx]

Forty-seven resources, forty-six edges, and not one line of logic anywhere in the file. catalog_uuid is unique to this one compile, which makes it the join key when you line an agent log up against a compile on the server. code_id stays null until you run versioned code deployment, at which point the catalog names the exact code version that built it. version is the epoch timestamp you saw in the log.

The edges array is not what most people assume, and the misunderstanding is expensive. Those forty-six edges are containment only: Stage[main] holds Class[Profile::App], which holds Package[nginx], and so on down the tree, one edge per resource. Your require and notify relationships are nowhere in that list. They sit in .parameters on the resources that declared them. The consequence is that a dependency cycle is not a compile error. The server hands out a looping catalog quite happily; the agent builds the graph, finds the loop, and kills the entire run with Error: Failed to apply catalog: Found 1 dependency cycle. Nothing is applied at all. A cycle can therefore sail through review, reach production, and take down every node in the fleet on the same schedule.

# the edge list is containment, nothing else
sudo jq -r '.edges[] | "\(.source) -> \(.target)"' "$CAT" | head -5
 
# so where did require and notify actually go?
sudo jq -c '.resources[] | select(.title == "/etc/nginx/conf.d/app.conf")
            | {require: .parameters.require, notify: .parameters.notify}' "$CAT"
 
# and what did the secret file get written down as?
sudo jq '.resources[] | select(.title == "/etc/app/db.conf")
         | {mode: .parameters.mode, content: .parameters.content}' "$CAT"

Stage[main] -> Class[Settings]
Stage[main] -> Class[main]
Stage[main] -> Class[Profile::App]
Class[Profile::App] -> Package[nginx]
Class[Profile::App] -> File[/etc/nginx/conf.d/app.conf]
{"require":"Package[nginx]","notify":"Service[nginx]"}
{
  "mode": "0600",
  "content": {
    "__ptype": "Sensitive",
    "__pvalue": "password=Tr0ub4dor-prod-9f3c\n"
  }
}

What noop does not stop

--noop is short for "no operation". Treat it as a dress rehearsal in full costume with the doors locked. The agent does the whole run except the last step: for each resource it reads the current state, compares it to the desired state, reports the gap, then declines to close it. Hold on to the word "whole", because the server still compiled that catalog for real to make the report possible.

# somebody proposes moving the port again; check before you merge
sudo puppet agent -t --noop

Info: Using environment 'production'
Info: Loading facts
Info: Caching catalog for web01.example.com
Info: Applying configuration version '1784626876'
Notice: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]/content:
--- /etc/nginx/conf.d/app.conf   2026-07-21 09:34:01.884000000 +0000
+++ /tmp/puppet-file20260721-3980-1x4d7c   2026-07-21 09:41:16.220000000 +0000
@@ -1,4 +1,4 @@
 server {
-  listen 8443;
+  listen 9443;
   server_name web01.example.com;
 }
 
Notice: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]/content: current_value '{sha256}9c4ae2...', should be '{sha256}1d77b8...' (noop)
Info: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]: Scheduling refresh of Service[nginx]
Notice: /Stage[main]/Profile::App/Service[nginx]: Would have triggered 'refresh' from 1 event
Notice: Class[Profile::App]: Would have triggered 'refresh' from 1 event
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 3.18 seconds

Same diff, same checksums, but the change line has changed shape. Instead of announcing a change it now reports current_value ..., should be ... (noop), and the refresh reads Would have triggered rather than Triggered, because a simulated change sends no real refresh event. The two extra Would have triggered lines on the class and the stage are that non-event bubbling up the containment tree. This output is what belongs in a change ticket.

Three kinds of real work still happen during a dry run, and all three are documented behaviour rather than bugs. First, the compile is real, so any custom function in any module you are about to deploy executes Ruby on the Puppet Server, as the puppet user, before you see a single line of preview. Second, the noop metaparameter beats the global flag in both directions, so a resource declared with noop => false gets applied for real in the middle of your dry run. Grep for it before you trust a preview of somebody else's code. Third, exec resources run their onlyif and unless commands during a noop run, because running those commands is how Puppet decides whether the exec is already in sync. (creates is different: it only tests whether a path exists, so it executes nothing.) An exec carrying unless => 'curl -s https://example.com/check.sh | bash' runs that pipeline as root while you believe you are only looking. A preview previews the apply. It does not sandbox the code.

A failed resource stops nothing else

Puppet works through a catalog the way a nurse works through a ward round, not the way a bank moves money. There is no transaction and there is no rollback, ever. When a resource fails, Puppet marks everything downstream of it as skipped and carries on with everything unrelated. Here is the same catalog on a host whose apt sources are broken.

sudo puppet agent -t ; echo "exit=$?"

Info: Caching catalog for web03.example.com
Info: Applying configuration version '1784627105'
Error: /Stage[main]/Profile::App/Package[nginx]/ensure: change from 'purged' to 'present' failed: Could not update: Execution of '/usr/bin/apt-get -q -y -o DPkg::Options::=--force-confold install nginx' returned 100: Reading package lists...
Building dependency tree...
Reading state information...
E: Unable to locate package nginx
Notice: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]: Dependency Package[nginx] has failures: true
Warning: /Stage[main]/Profile::App/File[/etc/nginx/conf.d/app.conf]: Skipping because of failed dependencies
Notice: /Stage[main]/Profile::App/File[/etc/app]/ensure: created
Notice: /Stage[main]/Profile::App/File[/etc/app/db.conf]/ensure: defined content as [redacted]
Notice: /Stage[main]/Profile::App/Service[nginx]: Dependency Package[nginx] has failures: true
Warning: /Stage[main]/Profile::App/Service[nginx]: Skipping because of failed dependencies
Notice: Applied catalog in 6.04 seconds
exit=6

The package failed, so the two resources that depend on it were skipped rather than attempted, while the two that depend on nothing were applied normally. The secret file logged as [redacted] because its content is Sensitive. You finish a run like that with a machine in a genuinely mixed state. That is correct behaviour, and worth knowing before you write an alert on it. The exit code is where the trap sits. --test switched on detailed-exitcodes, so 0 means a clean run with nothing to do, 1 means the run failed outright or never started because another run held the lock, 2 means resources changed, 4 means resources failed, and 6 means both. A CI job (continuous integration, the robot that checks every change before it merges) that treats every non-zero exit as failure will call a perfectly good deploy broken every single time Puppet changes anything. A job that leaves detailed exit codes off sees 0 while four resources burn. Check for 4 and 6. Treat 2 as success.

The quiet failure: yesterday's catalog

Now the setting that hides broken deploys in plain sight. usecacheonfailure defaults to true, and it means what it says: when the agent cannot get a fresh catalog, it applies the one it cached on the last successful run. Do not confuse it with use_cached_catalog, a separate setting that tells the agent never to request a new catalog at all.

Compile it before it ever reaches a node

All of which argues for compiling the catalog yourself, early, where a failure costs nothing. There are two different checks here and people confuse them constantly. puppet parser validate is a spell-checker: it parses one file and catches a missing brace or a stray comma. That is the whole job. It will happily pass code with an undefined variable, a class name that does not exist, a Hiera key you forgot to add, or a dependency cycle. Only a compile catches those, because only a compile has facts, Hiera and the module path in front of it.

# 1. syntax only: catches a missing brace, catches nothing else
puppet parser validate site-modules/profile/manifests/app.pp
 
# 2. a real compile for a real node, on the Puppet Server, as the puppet user
sudo -u puppet puppet catalog compile web01.example.com --render-as json > /tmp/web01.json

(puppet parser validate prints nothing and exits 0)
 
Error: Could not compile catalog for web01.example.com: Evaluation Error: Error while evaluating a Resource Statement, Class[Profile::App]: expects a value for parameter 'db_pass' (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 9, column: 3)

The syntax was flawless. The code was still undeployable, because nobody added profile::app::db_pass to the eyaml file for this node. puppet catalog compile runs the compiler locally in server mode, so it belongs on the Puppet Server, and run it as the puppet user rather than root or you will leave root-owned files in caches that puppetserver needs to write to later. One caveat catches people out: unless PuppetDB is wired in as the facts source in routes.yaml, the compile uses the Puppet Server's own facts rather than the node's, and any fact-driven conditional goes the wrong way. With PuppetDB in place it is the closest thing you have to a rehearsal for one named machine. Catalogs built from two git branches can also be compared resource by resource with octocatalog-diff, which is how you answer "what does this refactor actually change on 400 nodes" instead of guessing.

The version of this that belongs in CI is a unit test. rspec-puppet compiles a catalog in memory against a set of fake facts and then makes assertions about what came out, and the Puppet Development Kit (PDK) wires the whole harness up for you.

require 'spec_helper'
 
describe 'profile::app' do
  on_supported_os.each do |os, os_facts|
    context "on #{os}" do
      let(:facts) { os_facts }
      let(:params) do
        { 'port' => 8443, 'db_pass' => sensitive('not-the-real-one') }
      end
 
      it { is_expected.to compile.with_all_deps }
      it { is_expected.to contain_file('/etc/app/db.conf').with_mode('0600') }
    end
  end
end

pdk test unit

pdk (INFO): Using Ruby 3.2.5
pdk (INFO): Using Puppet 8.10.0
[✔] Preparing to run the unit tests.
 
profile::app
  on ubuntu-22.04-x86_64
    is expected to compile into a catalogue without dependency cycles
    is expected to contain File[/etc/app/db.conf] with mode => "0600"
  on redhat-9-x86_64
    is expected to compile into a catalogue without dependency cycles
    is expected to contain File[/etc/app/db.conf] with mode => "0600"
 
Finished in 3.42 seconds (files took 1.68 seconds to load)
4 examples, 0 failures

compile.with_all_deps is the line that earns its keep. It compiles the catalog, builds the relationship graph the agent would build, checks it for cycles, and checks that every require, notify, before and subscribe points at a resource that genuinely exists. Those are precisely the two failures the Puppet Server will never catch on your behalf, because it never builds that graph. One of them stops a run dead on every node at once. The other waits until a machine with a slightly different fact set asks for its catalog at 3am. Put that expectation on every class, run pdk validate and pdk test unit on every branch, and the compile error that would have quietly parked 400 nodes on last week's configuration becomes a red cross on a pull request instead.

Try this

Run sudo puppet agent -t on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: your catalog cache is a credential store. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related