Manifests & the DSL

Declarative code and ordering.

Hand a builder a numbered to-do list and you have already made every scheduling decision for them: pour the foundation, frame the walls, hang the door, in that order, because that is the order you wrote them in. Hand them a blueprint instead and you have described the finished house. Working out a sane order is their job. Puppet manifests are blueprints. A manifest describes what a machine should look like once Puppet has finished with it, and the order those descriptions sit in the file is not a promise about when they happen.

That last sentence is where people get hurt. You write the package, then the config file, then the service, in a tidy top-to-bottom block that reads like a shell script, and it works. Six months later somebody splits those three resources across three classes. The tidy order quietly stops meaning what you thought it meant. The service starts before the hardened config reaches the disk, and nothing in the run output complains. This lesson covers the language you write, and the graph that decides what actually happens.

What a manifest is made of

Manifests are plain text files ending in .pp, written in Puppet's DSL (domain-specific language: a small language built for one job, rather than a general-purpose one like Ruby or Python). The menu is short on purpose. You declare resources, set variables, branch with if and case, call functions, and define classes. There is no for loop and no while loop; when you need to repeat yourself you hand a list to each or map. There is a return function for leaving a class body early, and that is about the only jump the language gives you. You also cannot reassign a variable: write $port = 8080 twice in the same scope and compilation stops with "Cannot reassign variable". That rule feels harsh for about a week, until you notice there is no meaningful "later" in a document that describes an end state.

Every resource declaration has the same three parts: the type (what kind of thing this is), the title in quotes (a label that has to be unique for that type across the whole machine's configuration), and attributes describing the state you want. Read the block below as four statements in the present tense. /etc/motd is a file with this content and these permissions. nginx is installed. nginx is running, and comes back after a reboot.

file { '/etc/motd':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0644',
  content => "Managed by Puppet. Local edits get reverted.\n",
}
 
package { 'nginx':
  ensure => installed,
}
 
service { 'nginx':
  ensure => running,
  enable => true,
}

That file sits inside an environment, a directory of code handed to a group of nodes. Every .pp file under production/manifests together forms the site manifest, and it is evaluated for every node in that environment. Real codebases keep it nearly empty and push the content into modules, classes and Hiera 5 data files (Hiera is Puppet's data lookup layer: the code says "apply a password policy", the data says which one), arranged with the roles-and-profiles pattern. For learning the language, one flat file shows cause and effect fastest. Keep your scratch manifests somewhere like /root/lab rather than in manifests/, because anything sitting in that directory is live code for every node in the environment.

Check the grammar before you check the logic

puppet parser validate reads a manifest, parses it, and stops. No catalog is compiled, no server is contacted, nothing on the machine is touched. On success it prints nothing at all and exits 0, which feels anticlimactic and is exactly what you want from a pipeline check.

$ cd /etc/puppetlabs/code/environments/production
$ puppet parser validate manifests/site.pp && echo "parsed OK"

parsed OK

Now delete one character: the comma at the end of line 5, after mode => '0644'. The parser runs on to the next line, finds a bare word where it wanted a comma or a closing brace, and tells you where it gave up.

$ puppet parser validate manifests/site.pp

Error: Could not parse for environment production: Syntax error at 'content' (file: /etc/puppetlabs/code/environments/production/manifests/site.pp, line: 6, column: 3)

What that check cannot tell you is whether the manifest means anything. A misspelled attribute, a module that is not installed, a class parameter you forgot to pass: those surface later, when a catalog gets compiled. The gap matters more than it sounds, because of what puppet agent does when compilation fails. The usecacheonfailure setting defaults to true, so an agent that cannot fetch a fresh catalog falls back to the last one it cached and applies that instead. A broken manifest does not announce itself as an outage. It shows up as a fleet that keeps enforcing yesterday's configuration, looks healthy while doing it, and silently stops receiving your fixes, the urgent security ones included.

In a pipeline, put the whole thing through PDK (the Puppet Development Kit, the toolchain for building and testing Puppet code). pdk validate runs the parser, puppet-lint style checks and metadata checks in one command, and fails the merge request before anything reaches a compiler. If your shop has moved to OpenVox, the community fork that appeared after the Puppet 8 licence change, the language and every command in this lesson are unchanged, and the binary is still called puppet.

Make it real, then run it twice

puppet apply compiles and enforces a manifest on the machine you are sitting on, with no server involved. It is how you rehearse. Start with --noop (no operation), which is the walkthrough with a clipboard: Puppet reads the current state of every resource, works out the difference, prints what it would have done, and touches nothing.

$ sudo puppet apply --noop manifests/site.pp

Notice: Compiled catalog for web01.acme.internal in environment production in 0.34 seconds
Notice: /Stage[main]/Main/File[/etc/motd]/content: current_value '{sha256}0f9c1a...', should be '{sha256}4b2ae9...' (noop)
Notice: /Stage[main]/Main/Package[nginx]/ensure: current_value 'absent', should be 'present' (noop)
Notice: /Stage[main]/Main/Service[nginx]/ensure: current_value 'stopped', should be 'running' (noop)
Notice: /Stage[main]/Main/Service[nginx]/enable: current_value 'false', should be 'true' (noop)
Notice: Class[Main]: Would have triggered 'refresh' from 4 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 0.42 seconds

Nothing moved. Drop the flag and the same run does the work. This is a RHEL-family host, where installing a package does not start its service for you, so all four changes land in one go. On Debian-family hosts the package's post-install script starts and enables nginx itself, so by the time Puppet looks, the service is already in the state you asked for and those two service lines never appear.

$ sudo puppet apply manifests/site.pp

Notice: Compiled catalog for web01.acme.internal in environment production in 0.31 seconds
Notice: /Stage[main]/Main/File[/etc/motd]/content: content changed '{sha256}0f9c1a...' to '{sha256}4b2ae9...'
Notice: /Stage[main]/Main/Package[nginx]/ensure: created
Notice: /Stage[main]/Main/Service[nginx]/ensure: ensure changed 'stopped' to 'running'
Notice: /Stage[main]/Main/Service[nginx]/enable: enable changed 'false' to 'true'
Notice: Applied catalog in 8.06 seconds

Now run it a third time, with nothing edited.

$ sudo puppet apply manifests/site.pp

Notice: Compiled catalog for web01.acme.internal in environment production in 0.29 seconds
Notice: Applied catalog in 0.44 seconds

That silence is the test. Puppet is idempotent (running it twice changes nothing the second time) because every resource is read before it is written. If a second run still reports changes, something cannot converge: an exec with no guard, two resources fighting over one file, or a process outside Puppet rewriting the file behind your back. A manifest that never goes quiet is worse than useless, because once every run is noisy you lose the ability to spot the run where something genuinely drifted. Ship those reports to PuppetDB and the same changed-or-not data is what you query later to answer "which nodes changed last night".

State that Puppet wrote can be read back in the same language. puppet resource asks the Resource Abstraction Layer (the translation layer between Puppet's vocabulary and the package manager, init system or user database underneath) what something looks like right now, and prints the answer as a resource declaration you could paste into a manifest.

$ puppet resource service nginx

service { 'nginx':
  ensure   => 'running',
  enable   => 'true',
  provider => 'systemd',
}

That is the machine talking, not your code. Point it at something you have never managed (puppet resource user root, puppet resource package openssh-server) and you get a working snippet with the real attribute names in it, which beats guessing them.

File position is a hint, the graph is the contract

A project plan models this better than a checklist does. Some tasks have arrows between them because one genuinely cannot start until another finishes. The rest can happen whenever there is a free pair of hands. Puppet builds exactly that. At the end of compilation every resource becomes a point in a relationship graph, a directed acyclic graph (points joined by one-way arrows, with no route that leads back to where it started), and the run walks it.

Edges get into that graph four ways: relationship metaparameters, chaining arrows, autorequire rules baked into resource types, and class containment. Where two resources have no edge between them, Puppet falls back to manifest order, meaning the order the compiler evaluated them in. In one flat file that is top to bottom. Across classes it is the order the compiler reached those classes, which shifts the day somebody moves an include statement. You can ask Puppet which tiebreak it is using.

$ puppet config print ordering

manifest

manifest has been the default since Puppet 4 and it is the right setting for production, but the other two values are worth knowing. title-hash orders unrelated resources by a hash of their titles: stable between runs, and unrelated to how the file reads. random reshuffles them every run. That sounds like sabotage and makes an excellent test-node setting, because a missing edge that top-to-bottom luck has been covering up then fails on a throwaway box in CI instead of during a rebuild at 3am.

You never have to guess what Puppet decided. Turn on graph output and it writes the graph to disk in .dot format, the plain text format graphviz reads.

$ sudo puppet apply --graph manifests/site.pp
$ sudo ls "$(sudo puppet config print graphdir)"

Notice: Compiled catalog for web01.acme.internal in environment production in 0.29 seconds
Notice: Applied catalog in 0.41 seconds
expanded_relationships.dot  relationships.dot  resources.dot

resources.dot is the containment graph: which class or stage holds which resource. relationships.dot is the dependency graph with those containers still sitting in it. expanded_relationships.dot is the same graph after the containers have been spliced out, so every edge starts and ends on a real resource, which makes it the one that matches what the run actually walks. Render any of them with graphviz: dot -Tpng /opt/puppetlabs/puppet/cache/state/graphs/expanded_relationships.dot -o graph.png. When a colleague insists two resources are ordered, the picture settles it in about four seconds.

Saying out loud what depends on what

Greasing the tin and heating the oven can happen in either order, but the cake goes in after both. You do not order everything. You order the pairs that matter. Puppet gives you four metaparameters for that (a metaparameter is an attribute every resource type accepts, describing how Puppet should handle the resource rather than what the resource is). require and before are pure ordering, written from either end of the same edge. notify and subscribe add a refresh signal on top of the ordering, which is how a changed config file gets its service to reload.

package { 'nginx':
  ensure => installed,
}
 
file { '/etc/nginx/conf.d/hardening.conf':
  ensure  => file,
  owner   => 'root',
  group   => 'root',
  mode    => '0644',
  content => "server_tokens off;\nadd_header X-Frame-Options SAMEORIGIN;\n",
  require => Package['nginx'],   # the package owns /etc/nginx, so it lands first
  notify  => Service['nginx'],   # if this file changed, poke the service
}
 
service { 'nginx':
  ensure => running,
  enable => true,
  # subscribe => File['/etc/nginx/conf.d/hardening.conf'],  <- the mirror image
  #    of the notify above. Use one of them, never both.
}
 
# The same two edges written as a chain, INSTEAD OF the metaparameters above:
#   Package['nginx'] -> File['/etc/nginx/conf.d/hardening.conf'] ~> Service['nginx']

A refresh is a doorbell, not a rerun. Puppet does not re-apply the target resource; it calls that resource's refresh method, and what happens next depends on the type. A service restarts, but only if it is running at that moment: a stopped service logs "Skipping restart; service is not running" at debug level and stays stopped. An exec runs its command, or the separate command in its refresh attribute if you set one, and an exec with refreshonly => true runs at no other time. Types with no refresh method, package and user among them, ignore the event completely, so notifying them buys you ordering and nothing else.

Change the header line in that content, apply again, and the run tells you the service acted.

$ sudo puppet apply /root/lab/nginx.pp

Notice: Compiled catalog for web01.acme.internal in environment production in 0.44 seconds
Notice: /Stage[main]/Main/File[/etc/nginx/conf.d/hardening.conf]/content: content changed '{sha256}a1d472...' to '{sha256}7e08bb...'
Notice: /Stage[main]/Main/Service[nginx]: Triggered 'refresh' from 1 event
Notice: Applied catalog in 1.86 seconds

The Triggered line is your evidence: the service acted, and it acted on exactly one upstream change. See a content changed line with no Triggered line under it and the edge is missing. Add -v (verbose) and Puppet prints the other half of the story at info level, "Scheduling refresh of Service[nginx]", which names the resource that sent the event. Agent runs show that line without being asked, because puppet agent -t turns verbose on for you.

Here is why this matters beyond tidiness. Puppet guarantees the bytes in /etc/ssh/sshd_config. It does not guarantee what the running sshd is enforcing, because sshd reads its config at startup and hands the parsed copy to each connection it forks. Ship a hardened config with no notify and PermitRootLogin no sits on disk while the live process keeps whatever it started with, sometimes for months, while every audit script that greps the file reports you compliant. The restart is cheap: the stock sshd unit sets KillMode=process, so restarting the daemon leaves anyone already logged in exactly where they are.

Those edges can also be written as arrows between resource references, which are the capitalised form of the type with the title in brackets: File['/etc/motd']. -> means applied before. ~> means applied before, and send a refresh if I changed. Reversed forms exist and read right to left, which is precisely how people misread them, so leave them alone. Pick one style per codebase, because writing both the metaparameter and the arrow for the same pair is harmless and doubles the number of places the next person has to look.

The edges Puppet writes for you

Some dependencies are so obvious that resource types declare them on your behalf. Puppet calls these autorequire rules, and they behave like a sous-chef who puts the pan on the heat before you ask. A file autorequires the nearest ancestor directory that is also in the catalog, walking up the path until it finds one, and a symlink autorequires the target it points at when you manage that too. A user autorequires the group named in its gid, plus any groups it belongs to. An exec autorequires its cwd, any absolute path that appears in its command, and the user it runs as. Write two file resources in the wrong order and watch it work anyway.

# deliberately written in the "wrong" order
file { '/opt/app/config.yaml':
  ensure  => file,
  mode    => '0640',
  content => "listen: 127.0.0.1\n",
}
 
file { '/opt/app':
  ensure => directory,
  mode   => '0750',
}

$ sudo puppet apply /root/lab/app.pp

Notice: Compiled catalog for web01.acme.internal in environment production in 0.27 seconds
Notice: /Stage[main]/Main/File[/opt/app]/ensure: created
Notice: /Stage[main]/Main/File[/opt/app/config.yaml]/ensure: defined content as '{sha256}3f1b9c...'
Notice: Applied catalog in 0.05 seconds

The directory was created first even though it is declared second, because an autorequire edge is a real edge in the graph, and real edges beat the manifest-order tiebreak.

Three honest limits. Autorequire is catalog-wide, so it does reach across class and module boundaries; the folklore that it stops at a class boundary is wrong. It only fires when the other resource is actually in the catalog, and when it is not there is no edge, no warning and no clue. And it only covers the pairs the type's author thought of. Nothing links a package to the service it installs, so Package['nginx'] -> Service['nginx'] is yours to declare. Treat autorequire as a convenience, and write the edge yourself whenever a security property hangs on it.

Classes contain their resources, mostly

A class works like a shipping container. Put an edge on the container and everything inside inherits it. Class['profile::app::firewall'] -> Class['profile::app::service'] orders every resource in the first class ahead of every resource in the second, which is how ordering gets expressed once a codebase grows past one file.

There is a trap in that, and it has bitten most people who write profiles. include declares a class but does not put it inside the class that included it, so an edge on the outer class reaches nothing the inner class declared. Your profile says firewall rules before the service, the classes were included rather than contained, and the ordering covers nothing at all. The service comes up before the rules do, on every node, and the code reads as if it were safe. contain declares the class and puts it inside.

class profile::app {
  contain profile::app::firewall   # contain, not include
  contain profile::app::service
 
  # This edge now covers every resource inside both classes.
  Class['profile::app::firewall'] -> Class['profile::app::service']
}

Before contain existed, people wired the same guarantee by hand with the anchor pattern from the stdlib module (Anchor['x::begin'] -> Class['inner'] -> Anchor['x::end']). You still meet it inside older Forge modules. New code should use contain.

Every arrow points somewhere, so drawing a circle is easy. Add require => Service['nginx'] to the config file that already notifies that same service, and the graph has no valid order left in it.

$ sudo puppet apply /root/lab/nginx.pp

Notice: Compiled catalog for web01.acme.internal in environment production in 0.38 seconds
Error: Found 1 dependency cycle:
(File[/etc/nginx/conf.d/hardening.conf] => Service[nginx] => File[/etc/nginx/conf.d/hardening.conf])\nTry the '--graph' option and opening the resulting '.dot' file in OmniGraffle or GraphViz
Error: Failed to apply catalog: One or more resource dependency cycles detected in graph

The path in brackets is the loop, printed in order: start anywhere on it, follow the arrows, arrive back where you began. That stray \n sitting in the middle of the message is Puppet's own formatting quirk, not something you typed. Add --graph and Puppet writes the loop to cycles.dot in graphdir instead of only suggesting it, which pays off when the printed path runs to a dozen resources.

Try the failure once, on a machine you are allowed to break. Manage /etc/ssh/sshd_config with no notify, change a setting, apply, then compare two timestamps. The unit is sshd on RHEL-family systems and ssh on Debian-family ones.

$ stat -c '%y  %n' /etc/ssh/sshd_config
$ systemctl show sshd --property=ActiveEnterTimestamp

2026-07-21 09:41:07.882314216 +0000  /etc/ssh/sshd_config
ActiveEnterTimestamp=Mon 2026-07-20 14:12:55 UTC

The file is nineteen hours newer than the process reading it. The Puppet run reported success and said nothing about that gap, because closing it was never Puppet's decision to make. It was yours, and it is one line: notify => Service['sshd'].

Try this

Run puppet parser validate manifests/site.pp && echo "parsed OK" on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a dry run is a preview, not a promise. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related