PuppetDB & exported resources

Cross-node orchestration.

A village noticeboard works because nobody maintains it. Each shop pins up its own card: name, address, opening hours. The delivery driver walks past every morning, reads whatever is on the board that day, and plans a route from that. Shops open, shops close, cards go up and come down. The board stays current on its own, because the people who know the facts are the people posting them.

Puppet on its own has no board. Every catalog (the finished, node-specific list of resources Puppet will enforce) is compiled in isolation, from one machine's facts and one machine's classes, and it knows nothing about any other machine you own. That is fine for "install nginx and keep it running". It falls apart the moment a machine's correct configuration depends on which other machines exist. A load balancer needs to know its backends. A monitoring server needs its targets. A bastion host, the single gateway box everyone connects through, needs every other host's SSH (secure shell) key. PuppetDB is the noticeboard, bolted to a filing cabinet holding everything your fleet has ever reported.

What PuppetDB Actually Holds

PuppetDB is a separate service that runs beside your Puppet server with a PostgreSQL database behind it. The old embedded database was dropped years ago, so Postgres is the only supported option now. It keeps four kinds of thing. Facts: every value Facter (Puppet's fact-gathering program, which inventories a machine before every run) collected on a node, as of that node's last check-in. Catalogs: the complete compiled plan for each node, resource by resource, every parameter filled in with its final value. Reports: what each run actually did, which resources changed, what failed, how long it took. And exported resources, which get lifted out of those stored catalogs and offered to whoever asks.

It listens at two doors, and the difference matters. Port 8080 is plain HTTP, unencrypted, with no authentication at all, bound to localhost by default so only the PuppetDB machine itself can knock. Port 8081 speaks TLS (Transport Layer Security, the encryption behind HTTPS) and demands a client certificate signed by your Puppet CA (certificate authority, the service that signs every agent's certificate). Be honest about what that second door checks, though. Out of the box it checks that the certificate was signed by your CA, and nothing else. Every agent you manage holds one of those. Unless you add a certificate allowlist, any node in your fleet can read every catalog you have stored, which is a far wider door than most people picture.

The tool you will live in is puppet query, from the puppetdb-cli package. It speaks PQL (Puppet Query Language, PuppetDB's own query syntax) and reads its connection details from ~/.puppetlabs/client-tools/puppetdb.conf. Despite the .conf name that file is JSON (JavaScript Object Notation, a plain-text data format), and it points at the TLS port with a certificate and key to present.

{
  "puppetdb": {
    "server_urls": ["https://puppetdb.acme.internal:8081"],
    "cacert": "/etc/puppetlabs/puppet/ssl/certs/ca.pem",
    "cert": "/etc/puppetlabs/puppet/ssl/certs/admin.acme.internal.pem",
    "key": "/etc/puppetlabs/puppet/ssl/private_keys/admin.acme.internal.pem"
  }
}

# on the PuppetDB host itself, the loopback port answers with no credentials at all
$ curl -s http://localhost:8080/pdb/meta/v1/version
 
# from a workstation holding a signed cert, ask which nodes are still live
$ puppet query 'nodes[certname] { deactivated is null and expired is null }'

{"version":"8.8.1"}
 
[
  {
    "certname": "bastion1.acme.internal"
  },
  {
    "certname": "web1.acme.internal"
  },
  {
    "certname": "web2.acme.internal"
  }
]

Wiring It Up

Puppet does not talk to PuppetDB until you tell it to, and the first step is a package rather than a setting. Install puppetdb-termini on the Puppet server. That package ships the terminus (Puppet's word for a pluggable storage backend), the report processor, and the puppet node subcommands you will want later. After that, two settings switch on catalog storage, one more sends reports, and a small file says where PuppetDB lives. storeconfigs_backend does the real work: it swaps the catalog terminus for one that hands every compiled catalog to PuppetDB on its way out the door.

[main]
certname = puppet.acme.internal
 
# [server] is the Puppet 8 name for this section. Older docs and modules
# still write [master], which is the deprecated alias for the same thing.
[server]
storeconfigs         = true
storeconfigs_backend = puppetdb   # already the default; spelled out so nobody guesses
reports              = store,puppetdb

[main]
server_urls = https://puppetdb.acme.internal:8081

Running masterless with puppet apply works too, but it needs a different route table, because there is no server in the picture to do the storing. You put the same storeconfigs pair in the [main] section instead, and add a routes.yaml telling apply to cache its catalog, its resources and its facts into PuppetDB. The path is whatever puppet config print route_file prints. The applying machine still needs its own signed certificate, because it is knocking on port 8081 like everybody else.

---
apply:
  catalog:
    terminus: compiler
    cache: puppetdb
  resource:
    terminus: ral
    cache: puppetdb
  facts:
    terminus: facter
    cache: puppetdb_apply

Almost nobody writes these by hand. The puppetlabs-puppetdb module installs and configures the service (include puppetdb on the database host) and wires the server to it (include puppetdb::master::config on the Puppet server), writing every file above for you. Knowing what it writes still pays off, because when a collector mysteriously returns nothing, a missing storeconfigs = true is the first thing to check. If you moved to OpenVox, the community fork that appeared after the Puppet 8 licence change, the equivalent service is OpenVoxDB and it keeps the same paths, ports and command names.

One At-Sign Or Two

A single @ in front of a resource is a sticky note on your own fridge. It exists, it is written down, and nothing happens until somebody in that house picks it up. That is a virtual resource: declared in this node's catalog, inert, waiting for a collector (<| |>) or a realize() call somewhere else in the same catalog to switch it on. Two at-signs, @@, is a card pinned to the village board. Same inertness at home, plus one extra step: the resource is written into PuppetDB and tagged as exported, where any other node's collector can pull it in.

That collector is the double-angle form, Sshkey <<| |>>. Empty pipes mean "every export of this type, from anywhere". Put a search expression between them and you narrow the pull. Notice what does not happen: the node that exports a resource does not apply it. web1 publishing its own SSH host key writes nothing on web1. The export is a message for other machines.

# Every managed node publishes its own SSH host key. Applied fleet-wide.
class profile::ssh::hostkey {
 
  # a host that never generated an ed25519 key has no such fact, so guard it
  if $facts['ssh'] and $facts['ssh']['ed25519'] {
 
    @@sshkey { $trusted['certname']:
      ensure       => present,
      type         => $facts['ssh']['ed25519']['type'],  # 'ssh-ed25519'
      key          => $facts['ssh']['ed25519']['key'],   # base64 blob, no algorithm prefix
      host_aliases => [
        $facts['networking']['hostname'],
        $facts['networking']['ip'],
      ],
    }
  }
}

# The bastion collects every published key into /etc/ssh/ssh_known_hosts.
class profile::ssh::known_hosts {
 
  Sshkey <<| |>>            # every exported host key, fleet-wide
 
  # drop entries nobody is publishing any more (read the caveat further down)
  resources { 'sshkey':
    purge => true,
  }
}

Read the title of that export closely. $trusted['certname'] is not decoration. It buys two things at once. It is globally unique, because your CA will not hand out two live certificates with the same certname, and exported titles have to be unique across every node whose exports land in one catalog, which in practice means the whole fleet. It is also unforgeable. $trusted is built on the server from the certificate the agent presented during the TLS handshake, while every single value in $facts is supplied by the agent itself. A compromised machine can claim any hostname, any IP address, any operating system it fancies. It cannot claim a certname it holds no signed key for.

One footnote on that manifest. The sshkey type moved out of Puppet core in version 6 and now lives in the puppetlabs-sshkeys_core module. The puppet-agent package ships it bundled, so it works out of the box, but a gem-only install needs the module added by hand.

Watch It Happen

Here is the whole cycle on a three-node fleet: web1, web2, bastion1. Start on web1, which has profile::ssh::hostkey in its role and nothing else new. Read the output for what is missing.

root@web1:~# puppet agent -t

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for web1.acme.internal
Info: Applying configuration version '1721558402'
Notice: Applied catalog in 4.13 seconds

Not one line about an sshkey. web1 declared the resource and applied nothing, exactly as designed. To find out whether the card reached the board, ask PuppetDB directly. The exported = true filter is the part that matters.

$ puppet query 'resources[certname, title, parameters] { exported = true and type = "Sshkey" }'

[
  {
    "certname": "web1.acme.internal",
    "title": "web1.acme.internal",
    "parameters": {
      "ensure": "present",
      "host_aliases": ["web1", "10.20.0.11"],
      "key": "AAAAC3NzaC1lZDI1NTE5AAAAIB9xk1p6QOL8yTfKz0hXWn2u4dCq5vRmJ3sT7aE0lPqZ",
      "type": "ssh-ed25519"
    }
  },
  {
    "certname": "web2.acme.internal",
    "title": "web2.acme.internal",
    "parameters": {
      "ensure": "present",
      "host_aliases": ["web2", "10.20.0.12"],
      "key": "AAAAC3NzaC1lZDI1NTE5AAAAIH4vQ2mNbX8sT1kR7pLdW0zYcF6gJ9uA3eKiO5rMnBtV",
      "type": "ssh-ed25519"
    }
  }
]

Two cards on the board. Now go to the bastion and run with --noop first ("no operation": compile, compare against the live system, report what would change, touch nothing). On a collector this is your review gate. It is the first moment you find out how many resources you are about to inherit from other people's machines.

root@bastion1:~# puppet agent -t --noop

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for bastion1.acme.internal
Info: Applying configuration version '1721558511'
Notice: /Stage[main]/Profile::Ssh::Known_hosts/Sshkey[web1.acme.internal]/ensure: current_value 'absent', should be 'present' (noop)
Notice: /Stage[main]/Profile::Ssh::Known_hosts/Sshkey[web2.acme.internal]/ensure: current_value 'absent', should be 'present' (noop)
Notice: Class[Profile::Ssh::Known_hosts]: Would have triggered 'refresh' from 2 events
Notice: Stage[main]: Would have triggered 'refresh' from 1 event
Notice: Applied catalog in 3.87 seconds

root@bastion1:~# puppet agent -t
root@bastion1:~# puppet resource sshkey web1.acme.internal

Notice: /Stage[main]/Profile::Ssh::Known_hosts/Sshkey[web1.acme.internal]/ensure: created
Notice: /Stage[main]/Profile::Ssh::Known_hosts/Sshkey[web2.acme.internal]/ensure: created
Notice: Applied catalog in 4.02 seconds
 
sshkey { 'web1.acme.internal':
  ensure       => 'present',
  host_aliases => ['web1', '10.20.0.11'],
  key          => 'AAAAC3NzaC1lZDI1NTE5AAAAIB9xk1p6QOL8yTfKz0hXWn2u4dCq5vRmJ3sT7aE0lPqZ',
  target       => '/etc/ssh/ssh_known_hosts',
  type         => 'ssh-ed25519',
}

That is the verification chain worth memorising, and it has three links rather than one. puppet query proves the export reached PuppetDB. --noop on the collector proves the collector can see it, before anything gets written. puppet resource reads the live system back and proves the entry landed on disk with the values you expected. Skip the middle link and the first failure you notice will be a config file that changed on a host nobody was watching.

The security payoff is concrete. With a fleet-wide ssh_known_hosts maintained this way, nobody on your team ever has to answer the "authenticity of host cannot be established" prompt by typing yes and hoping. The fingerprint check starts working the way it was designed to, which means somebody sitting in the middle of an SSH connection produces a hard failure instead of a shrug.

The Backend Pool That Maintains Itself

The other classic use is the one the puppetlabs-haproxy module documents. Each web node registers itself with the load balancer instead of somebody editing a list by hand. Build a new web node, it exports a backend entry, the load balancer picks it up on its next run. Decommission one properly and the entry goes away. The pool tracks reality with no human in the loop.

# on each web node
class profile::app_backend {
 
  @@haproxy::balancermember { $trusted['certname']:
    listening_service => 'app00',
    server_names      => $facts['networking']['hostname'],
    ipaddresses       => $facts['networking']['ip'],
    ports             => '8080',
    options           => 'check',
  }
}
 
# on the load balancer, alongside the haproxy::listen { 'app00': } that defines the pool
class profile::lb {
 
  Haproxy::Balancermember <<| listening_service == 'app00' |>>
}

That search expression is doing security work, not tidiness work. listening_service == 'app00' matches a parameter whose value was written in your manifest and compiled on your server, so a node cannot rewrite it on the way out. Draw the line carefully, though, because the guarantee only holds while your classification is code-controlled too. If web2 receives profile::app_backend because of a role fact it reported about itself, a compromised web2 can classify itself into any pool it likes and the filter never catches a lie, because there isn't one. Classify on $trusted['extensions'] from the certificate, or on data your server owns. And notice what the filter does not cover at all: the values inside the collected resource. ipaddresses is read from a fact, and facts are whatever the agent chose to send.

Asking The Fleet A Question

Exported resources are one thing PuppetDB gives you. The other is that you can interrogate the whole fleet from a single shell. At three in the morning with a security advisory open, "which of my machines has this" is the only question that matters, and PuppetDB answers it in under a second from data it already holds, without waking a single node.

# which machines run a Debian-family OS?
$ puppet query 'inventory[certname] { facts.os.family = "Debian" }'
 
# which machines have not reported in a week? dead agent, or the box is gone
$ puppet query 'nodes[certname, report_timestamp] { report_timestamp < "2026-07-14T00:00:00Z" }'
 
# which machines actually received the known_hosts class?
$ puppet query 'resources[certname] { type = "Class" and title = "Profile::Ssh::Known_hosts" }'

[
  { "certname": "web1.acme.internal" },
  { "certname": "web2.acme.internal" }
]
 
[
  {
    "certname": "old-jump.acme.internal",
    "report_timestamp": "2026-07-02T04:11:39.882Z"
  }
]
 
[
  { "certname": "bastion1.acme.internal" }
]

Sit with that third answer for a second. One node. You wrote the class, you tested it, you shipped it, and it is applied on exactly one machine. The gap between "the code exists" and "the code runs here" is where most real compliance findings live, and a PQL query surfaces it in a way a code review never will. The second answer is the same idea pointed at your agents: a node that stopped reporting a fortnight ago is a node nobody is patching.

Be honest about the limits of those answers. PuppetDB tells you what each node last said about itself. A machine powered off for a week reports week-old facts. A machine an attacker owns reports whatever facts the attacker wants it to report, including a package version it does not have. PuppetDB is an excellent first pass and a poor witness. For anything that has to hold up, go and look at the host.

Lag, Ghosts And Collisions

Three things go wrong with exported resources, and all three are ordinary rather than exotic. The first is lag. An export becomes visible once PuppetDB has stored the catalog containing it, and that storage is triggered when the catalog compiles on the server, not when the agent finishes. Strictly, the server posts a command onto PuppetDB's queue and PuppetDB works through the queue, which is usually sub-second and occasionally minutes on a busy server. Then the collecting node has to compile, on its own schedule. Default check-in is every 30 minutes (runinterval = 30m), so counting from the moment a new web node is built you are waiting on one run there and one run on the load balancer. Half an hour on average. An hour at the far end. If your deploy pipeline assumes a node joins the pool the second it boots, the pipeline is wrong, and the fix is to trigger a run on the load balancer once the new backend has reported.

The second is collisions. Exported titles have to be unique across every node feeding the same collector, because they all end up in one catalog on the collecting machine. Title an export with a short hostname, then build a second web in another datacentre, and the load balancer stops compiling. Not degrading. Stopping.

root@lb1:~# puppet agent -t

Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Error: Could not retrieve catalog from remote server: Error 500 on SERVER: Server Error:
  Evaluation Error: Error while evaluating a Collect Expression, Duplicate declaration:
  Haproxy::Balancermember[web] is already declared; cannot redeclare
  (file: /etc/puppetlabs/code/environments/production/modules/profile/manifests/lb.pp,
  line: 4, column: 3) on node lb1.acme.internal
Warning: Not using cache on failed catalog
Error: Could not retrieve catalog; skipping run

The third is ghosts. A decommissioned node keeps its card on the board until somebody takes it down, so the load balancer keeps sending traffic to an address that stopped answering, and the bastion keeps trusting a host key for a machine that was wiped and rebuilt. Deactivating is one command. There is a trap in it: a deactivated node comes straight back to life the moment it submits facts again, so run this after the box is actually off, or you will wonder why the entry reappeared half an hour later. PuppetDB will also expire quiet nodes on a timer, driven by TTL (time to live) settings.

root@puppet:~# puppet node deactivate web2.acme.internal

Submitted 'deactivate node' for web2.acme.internal with UUID 3f2a0b58-6c41-4f9e-9d2a-1b77c5e08a34

[database]
# these are the shipped defaults in PuppetDB 8, written out so they are visible
 
# a node silent this long is marked expired and stops exporting
node-ttl       = 7d
# expired or deactivated nodes are deleted entirely this long afterwards
node-purge-ttl = 14d
# reports older than this are dropped
report-ttl     = 14d

Now the part that catches people. Deactivating a node stops its exports being collected. It does not undo work Puppet already did on the collector. The Sshkey[web2.acme.internal] line sits in the bastion's ssh_known_hosts forever, because Puppet has stopped managing that resource, and an unmanaged resource is a resource Puppet leaves alone. That is what the resources { 'sshkey': purge => true } block in the earlier manifest is for. It tells Puppet to delete every resource of that type on the node that its catalog does not mention. Powerful and blunt in equal measure, since it will happily delete entries a colleague added by hand.

What An Attacker Does With This

Take the load balancer example and drop one compromised web node into it. The attacker has root on web2, so they write a file into /opt/puppetlabs/facter/facts.d/, the external facts directory Facter reads on every run. External facts outrank built-in ones, so whatever sits there is what your server believes. Override the networking fact and web2's exported haproxy::balancermember now carries an address the attacker controls. Next compile, the load balancer collects it and adds their box to the production pool. No ticket, no alert, and the change is attributed to Puppet, so your audit trail says the configuration management system did it. Which is true, and useless.

The same trick against the SSH example is worse if you title your exports with facts. A compromised node claiming the fully qualified name of a recently decommissioned host can export that host's name carrying its own key. Every collector then writes a line saying "this attacker-held key is the legitimate host key for that name", and the fingerprint check that was supposed to catch interception now blesses it instead. Titling with $trusted['certname'] closes that door, because certname comes from the signed certificate rather than from anything the node typed. It is the same rule that makes open certificate autosigning a bad idea: never make a trust decision on a value the other side controls.

One habit worth adopting today. Before you hand a PuppetDB archive to a vendor, a support case, or a colleague's laptop for debugging, run puppet db export pdb-archive.tgz --anonymization full rather than a plain export. Same shape of data, hostnames and parameter values scrubbed. Get that flag wrong once and you have emailed your fleet's blueprint to somebody outside your company, and there is no version of that you can take back.

Try this

Run curl -s http://localhost:8080/pdb/meta/v1/version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a --noop Run Still Publishes Your Exports. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related