CoursesPuppetSecuring Puppet infrastructure

Securing Puppet infrastructure

Certs, the master, and node trust.

Advanced14 min · lesson 12 of 12

Puppet's security model fits in one sentence: a machine may only receive the configuration that belongs to the name on its certificate. Break that sentence anywhere and a compromised web server can talk your Puppet server into handing over the database's private key, as root, every thirty minutes, with nobody watching.

The moving parts work like the key desk in an office lobby. Your Puppet server runs a certificate authority (CA, the thing that issues identity documents and vouches for them later). A new machine walks up and asks for a key. Somebody decides whether it gets one. After that, every visit starts with the desk checking the key, and the key is stamped with the holder's name plus a few facts that nobody inside the building can restamp. Four questions follow from that: who gets a key, what the key is allowed to say, what a key holder may ask for, and how you take a key back. Encrypting the data Puppet ships is the Hiera lesson's job. This one is identity, and how far one compromised machine can reach.

Enrollment Is The Only Moment You Get To Say No

On its first run, a fresh node generates a private key on its own disk and sends a certificate signing request to the Puppet server on port 8140. The private key never leaves the machine. A certificate signing request (CSR) is a short document that says "here is the name I claim, here is my public key, please vouch for me". Puppet then stops and waits. That pause is the only point in the whole system where a human gets to refuse a machine an identity.

terminal
# on the brand new node
sudo /opt/puppetlabs/bin/puppet agent --test
output
Info: Creating a new RSA SSL key for web3.acme.internal
Info: csr_attributes file loading from /etc/puppetlabs/puppet/csr_attributes.yaml
Info: Creating a new SSL certificate request for web3.acme.internal
Info: Certificate Request fingerprint (SHA256): 5A:1E:9F:03:C4:88:7B:2D:6E:11:A0:44:9C:52:3F:B8:D7:60:1C:E5:2A:93:F1:07:48:BB:C6:15:8E:32:D9:44
Info: Certificate for web3.acme.internal has not been signed yet
Couldn't fetch certificate from CA server; you might still need to sign this agent's certificate (web3.acme.internal).
Exiting; no certificate found and waitforcert is disabled

Read the fingerprint line. SHA-256 (Secure Hash Algorithm, 256-bit) squeezes the whole request down to a fixed-length string that changes completely if a single byte of the request changes. Your CA prints a fingerprint too. Comparing the two by eye is how you know the request sitting in your queue is the one this machine actually sent, and not a substitute posted by somebody who guessed your naming scheme first.

terminal
# on the Puppet server
sudo puppetserver ca list --all
output
Requested Certificates:
web3.acme.internal (SHA256) 5A:1E:9F:03:C4:88:7B:2D:6E:11:A0:44:9C:52:3F:B8:D7:60:1C:E5:2A:93:F1:07:48:BB:C6:15:8E:32:D9:44
Signed Certificates:
puppet.acme.internal (SHA256) 0A:3D:71:22:EF:9B:15:C8:44:0D:A9:63:2E:17:8F:B0:5C:D4:36:E2:71:98:AA:0F:3B:12:6D:C9:04:57:E8:FF alt names: ["DNS:puppet", "DNS:puppet.acme.internal"]
web1.acme.internal (SHA256) 6B:C0:14:9A:3E:D2:77:58:81:F6:2B:40:CE:19:A3:07:65:BD:8C:31:F0:4E:92:16:DA:5F:73:08:B4:2C:E1:9D
db1.acme.internal (SHA256) 9C:47:E0:B3:25:1A:6F:D8:70:39:C2:8E:14:5B:A6:F3:02:D1:7C:49:E5:88:30:BF:61:0A:97:2E:D3:56:1B:C8
Revoked Certificates:
old-node.acme.internal (SHA256) 2F:88:D4:60:19:73:AC:0E:5D:B1:36:F2:47:9A:C5:20:E8:71:3B:04:9F:6C:12:8D:A3:55:E7:1F:B9:48:C0:2A

Fingerprints match, so sign it. Notice what signing means. You are making a statement, good for five years by default, that this public key belongs to this name. The ca_ttl setting controls that lifetime (ttl is time to live, how long the statement stays valid). Five years is a long time to be wrong about a machine.

terminal
sudo puppetserver ca sign --certname web3.acme.internal
output
Successfully signed certificate request for web3.acme.internal

One historical note that saves you an hour of confused searching. The old puppet cert sign command was removed in Puppet 6. On Puppet 8 the CA lives behind puppetserver ca, which talks to the running server over its own web interface, so the service has to be up for the command to work at all. If you are running OpenVox, the community fork that appeared after the 2025 change to how Puppet is developed and packaged, the commands and paths in this lesson are the same.

Autosigning Without Handing Out Master Keys

Signing by hand stops working the moment an autoscaling group replaces twelve machines while you are asleep. The tempting fix is autosign = true, a front door that swings open for anyone who knocks and says a name. Work out what that buys somebody who can reach port 8140 from a coffee shop. They pick a name nobody has claimed yet, get it signed, and now hold an identity your fleet accepts. If your code classifies by name pattern, calling themselves db99.acme.internal is enough to be handed the database role's catalog, secrets and all. They can also write facts and exported resources into PuppetDB under that identity, and exported resources get collected by other nodes and applied there as root. That is the path from one open port to code running on machines the attacker never touched.

/etc/puppetlabs/puppet/puppet.conf
# On the Puppet server. Pick exactly one autosign line.
[server]
# autosign = true # signs anything that asks. Never do this.
# autosign = /etc/puppetlabs/puppet/autosign.conf # a list of names and globs. The requester picks the name.
autosign = /usr/local/bin/puppet-autosign # policy executable: exit 0 signs, anything else refuses
ca_ttl = 5y # lifetime of every certificate this CA signs
[agent]
server = puppet.acme.internal
certificate_revocation = chain # check the server's full chain against the revocation list (default)

The middle option looks safer than it is. A line like *.acme.internal in autosign.conf reads as "only our machines", but the name in a signing request is chosen by whoever sends the request. A glob is a wildcard pattern, the same * you use to list files, and a laptop on hotel wifi can call itself db99.acme.internal and match it perfectly. The name proves nothing until something other than the name has convinced you to trust it.

The third option is the one that holds up. Point autosign at an executable and Puppet Server runs it once per request, handing it the certname as the first argument and the raw signing request on standard input. Exit 0 and the CA signs. Any other exit code leaves the request sitting in the queue. That gives you somewhere to check something the requester cannot make up on its own.

/usr/local/bin/puppet-autosign
#!/opt/puppetlabs/puppet/bin/ruby
# Autosign policy. chmod 0755, owned by root. Puppet Server runs it as the 'puppet' user.
# Anything this script writes to stderr lands in /var/log/puppetlabs/puppetserver/puppetserver.log,
# so say out loud why you refused.
require 'openssl'
require 'fileutils'
TOKEN_DIR = '/etc/puppetlabs/puppet/enrollment-tokens'.freeze # mode 0700, owned by puppet
certname = ARGV[0].to_s
# 1. The name is attacker-controlled text. Constrain it before it touches a file path.
unless certname =~ /\A[a-z0-9]([a-z0-9-]*[a-z0-9])?\.acme\.internal\z/
warn "autosign: refusing #{certname.inspect}: not a well-formed acme.internal name"
exit 1
end
# 2. Pull the challengePassword the provisioner baked into csr_attributes.yaml.
csr = OpenSSL::X509::Request.new($stdin.read)
attribute = csr.attributes.find { |a| a.oid == 'challengePassword' }
presented = attribute && attribute.value.value.first.value.to_s
# 3. Compare it against the one-time token issued for this exact name.
token_file = File.join(TOKEN_DIR, certname)
unless presented && File.file?(token_file)
warn "autosign: refusing #{certname}: no token presented, or none was ever issued"
exit 1
end
unless OpenSSL.secure_compare(File.read(token_file).strip, presented)
warn "autosign: refusing #{certname}: token mismatch"
exit 1
end
# 4. One shot only. Burn the token so a replayed request cannot enroll a second machine.
FileUtils.rm_f(token_file)
warn "autosign: signing #{certname}"
exit 0

OpenSSL.secure_compare takes the same amount of time whether the first byte is wrong or the last one is, which stops an attacker from learning the token one character at a time by timing how fast you say no. The token directory lives on the CA server, mode 0700, owned by puppet, because the script both reads and deletes files in it. Your provisioner does two things at launch: it drops one token file on the CA server named after the machine, and it writes the matching value into that machine's csr_attributes.yaml before Puppet ever starts.

/etc/puppetlabs/puppet/csr_attributes.yaml
# Written by the provisioner on the new node, before Puppet ever starts.
# Read exactly once, at enrollment, then never again.
---
custom_attributes:
1.2.840.113549.1.9.7: "6f1c9d40e2a74b3f8c05d1ab97e6f2c3" # challengePassword: the one-time token
extension_requests:
pp_role: db-primary # OID 1.3.6.1.4.1.34380.1.1.13
pp_environment: production # OID 1.3.6.1.4.1.34380.1.1.12
pp_datacenter: fra1 # OID 1.3.6.1.4.1.34380.1.1.19

Those two sections behave completely differently, and the difference is the whole trick. Everything under custom_attributes exists only inside the signing request. The CA reads it while deciding, then throws it away, which is exactly what you want for a shared secret. Everything under extension_requests gets copied into the signed certificate and stays there for the life of that certificate. That is how a machine ends up carrying a role stamp it cannot edit. Keep the token in quotes, by the way: an unquoted run of digits is read as a number by the YAML parser (YAML is the indented plain-text data format Puppet uses here) and the comparison then fails for reasons that take a while to spot.

Two ways autosign quietly betrays you
First, autosign = true hands a working identity to anybody who can open a connection to port 8140 and pick a name nobody has claimed, which is enough to be served whatever catalog your classification gives that name, secrets included. Second, Puppet decides how to read the autosign path by looking at the executable bit: executable means policy script, not executable means a plain list of names to match against. Forget the chmod and your carefully written Ruby is read as a list of certnames that matches nothing. It fails closed, which is the good news, and it looks exactly like your policy script being ignored, which is the bad news. Check with test -x /usr/local/bin/puppet-autosign && echo executable.

What The Node Claims Versus What You Signed

A fact is a name badge the visitor wrote themselves. A trusted fact is the photo identification the desk already checked. Both turn up as variables in your manifests, they look almost the same, and that is why people mix them up and why mixing them up gets expensive.

Facter (Puppet's inventory tool, the thing that reports what a machine is and has) runs on the node, and the agent uploads the results to the server at the start of every run. Alongside the built-in facts, Facter reads any file dropped into /etc/puppetlabs/facter/facts.d/. Those are external facts. Writing one takes a single shell redirect. So does inventing one.

terminal
# on web1, from a shell that has already got root there
echo 'role=db-primary' | sudo tee /etc/puppetlabs/facter/facts.d/role.txt
sudo /opt/puppetlabs/bin/facter role
sudo /opt/puppetlabs/bin/puppet agent --test --noop
output
role=db-primary
db-primary
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for web1.acme.internal
Info: Applying configuration version '1784625120'
Notice: /Stage[main]/Profile::Db_primary/Package[postgresql-16]/ensure: current_value 'purged', should be 'present' (noop)
Notice: /Stage[main]/Profile::Db_primary/File[/etc/postgresql/16/main/server.key]/ensure: current_value 'absent', should be 'file' (noop)
Notice: Class[Profile::Db_primary]: Would have triggered 'refresh' from 2 events
Notice: Applied catalog in 3.91 seconds

One line in a text file, and the Puppet server compiled a catalog containing the database's TLS private key for a web server. --noop (no operation, a dry run that reports what it would change and changes nothing) means nothing was written this time, so that output is a rehearsal of the theft rather than the theft. Drop the flag and the key file lands on disk, owned by root, exactly as the server instructed. The web box broke into nothing. It filled in a form and the server believed it.

$trusted closes that door. Puppet builds it on the server from the certificate the agent presented during the TLS handshake (TLS, Transport Layer Security, is the encryption and identity layer underneath HTTPS), not from anything the agent typed. It holds authenticated, certname, domain, hostname, an extensions hash carrying the stamps from extension_requests, and external. An agent cannot move any of it without persuading your CA to sign a new certificate.

site-modules/profile/manifests/node_role.pp
# Decide privilege from the certificate, never from a fact.
class profile::node_role {
# WRONG. $facts arrives from the node. Anyone with root there picks the value.
# if $facts['role'] == 'db-primary' { include profile::db_primary }
# RIGHT. $trusted['extensions'] comes out of the certificate your CA signed.
case $trusted['extensions']['pp_role'] {
'db-primary': { include profile::db_primary }
'webserver': { include profile::webserver }
default: {
# No signed role means baseline only. Fail closed, and say so in the server log.
notice("no pp_role in certificate for ${trusted['certname']}, applying baseline only")
include profile::baseline
}
}
# certname also comes from the certificate, so this is sound as well. Remember that
# the requester chose that name in its signing request, so it is only ever as strong
# as the signing policy that let the name through.
if $trusted['certname'] =~ /^prod-/ {
include profile::prod_hardening
}
}

Hiera 5 can key off the same signed data, so the lookup that returns a secret depends on the certificate rather than on the node's word. Two details about the keys inside $trusted['extensions']. An OID (object identifier) is the dotted number that formally names an extension, and Puppet's registered ones sit under 1.3.6.1.4.1.34380.1.1, which is why they get friendly short names like pp_role in both csr_attributes.yaml and your manifests. Anything you invent yourself belongs under 1.3.6.1.4.1.34380.1.2 and turns up keyed by its full dotted number instead.

/etc/puppetlabs/code/environments/production/hiera.yaml
---
version: 5
defaults:
datadir: data
data_hash: yaml_data
hierarchy:
- name: "Per-node overrides"
path: "nodes/%{trusted.certname}.yaml"
- name: "Per-role data, keyed off the signed certificate"
path: "roles/%{trusted.extensions.pp_role}.yaml"
- name: "Secrets, same key, decrypted on the server only"
lookup_key: eyaml_lookup_key
paths:
- "secrets/roles/%{trusted.extensions.pp_role}.eyaml"
- "secrets/common.eyaml"
options:
pkcs7_private_key: /etc/puppetlabs/puppet/eyaml/private_key.pkcs7.pem
pkcs7_public_key: /etc/puppetlabs/puppet/eyaml/public_key.pkcs7.pem
- name: "Common defaults"
path: "common.yaml"

A node with no pp_role in its certificate interpolates to nothing, so the path becomes roles/.yaml, no such file exists, and Hiera drops through to the common defaults. Missing identity gets you the least privilege on offer, which is the right way round.

Now prove the fix instead of assuming it. Leave the forged fact file exactly where it is and run the agent again.

terminal
# the bogus fact is still sitting in /etc/puppetlabs/facter/facts.d/role.txt
sudo /opt/puppetlabs/bin/facter role
sudo /opt/puppetlabs/bin/puppet agent --test --noop
output
db-primary
Info: Using environment 'production'
Info: Retrieving pluginfacts
Info: Retrieving plugin
Info: Retrieving locales
Info: Loading facts
Info: Caching catalog for web1.acme.internal
Info: Applying configuration version '1784627251'
Notice: Applied catalog in 2.44 seconds

Not one database resource in the catalog. The other half of the story is on the server rather than on the node, because notice() runs during compilation and its output goes to the Puppet Server log instead of back to the agent's terminal. People lose afternoons to that, so go and look.

terminal
# on the Puppet server
sudo grep 'no pp_role' /var/log/puppetlabs/puppetserver/puppetserver.log | tail -n 1
output
2026-07-21T09:47:31.402Z INFO [qtp1594887619-68] [puppetserver] Puppet Scope(Class[Profile::Node_role]): no pp_role in certificate for web1.acme.internal, applying baseline only

The node still shouts that it is the database. The catalog no longer cares. Keep that pair of runs as a test, because it fails loudly the day somebody reintroduces a fact-based conditional.

There is a third source of truth worth knowing about. Set trusted_external_command in puppet.conf and the server runs your script, passes it the certname, and drops whatever JSON comes back (JSON, JavaScript Object Notation, is the curly-brace data format) into $trusted['external']. The node contributes nothing but the name on its certificate, so you can look up roles in a configuration management database (CMDB, the inventory system that already knows what each machine is for) without trusting the machine at all. It also fixes a real annoyance with certificate extensions. A stamp baked into a certificate cannot be edited, so repurposing a node means cleaning it and enrolling it again. Certificate extensions for the things that never change, external data for the things that do. The cost is one more service that has to be up before any catalog will compile.

THREE SOURCES OF NODE DATA, THREE LEVELS OF TRUST
$facts (the node's word)
Facter + /etc/puppetlabs/facter/facts.d
any root user on the node can write a fact
Good for
OS family, kernel, memory, disks, IP addresses
Never for
roles, environments, entitlement to a secret
$trusted (the certificate's word)
certname, domain, hostname
read from the common name on the presented certificate
extensions.pp_role and friends
from extension_requests, fixed at signing time
To change it
clean the cert and get your CA to sign a new one
$trusted['external'] (your CMDB's word)
trusted_external_command
runs on the server, receives only the certname
Good for
role and owner data that changes without a rebuild
Cost
one more service that must be up for catalogs to compile
Same syntax in a manifest, completely different guarantees. Facts describe a machine; certificates and external data are the only ones that may authorize it.

One Name, One Catalog

Holding a valid certificate is not the same as being allowed to ask for anything, in the way a building pass gets you through the front door but not into the safe. Puppet Server's authorization rules live in auth.conf, written in HOCON (Human-Optimized Config Object Notation, a friendlier spelling of JSON that allows real comments and drops most of the quotes). Rules are sorted by sort-order, the first one whose pattern matches the request wins, and the last rule denies everything that fell through.

/etc/puppetlabs/puppetserver/conf.d/auth.conf
authorization: {
version: 1
rules: [
{
# The rule that keeps web1 out of db1's catalog. The regex captures the
# certname from the URL, and allow: "$1" demands that the capture equals
# the common name on the certificate presented in the TLS handshake.
match-request: {
path: "^/puppet/v3/catalog/([^/]+)$"
type: regex
method: [get, post]
}
allow: "$1"
sort-order: 500
name: "puppetlabs catalog"
},
# ... report, file_metadata, file_content and node rules follow ...
{
match-request: {
path: "/"
type: path
}
deny: "*"
sort-order: 999
name: "puppetlabs deny all"
}
]
}

That is the shipped default and it earns its keep. Watch what happens when a node asks for somebody else's catalog.

terminal
# on the Puppet server, after web1 tried to fetch db1's catalog
sudo grep 'Forbidden request' /var/log/puppetlabs/puppetserver/puppetserver.log | tail -n 1
output
2026-07-21T09:41:07.882Z ERROR [qtp1594887619-73] [p.t.a.rules] Forbidden request: web1.acme.internal(10.0.1.21) access to /puppet/v3/catalog/db1.acme.internal (method :post) (authenticated: true) denied by rule 'puppetlabs catalog'.

authenticated: true is the detail to notice. The certificate was perfectly valid. Authentication answered "who are you", authorization answered "and what may you have", and the second answer was no. Any rule you add on top of the defaults should keep that shape: match the narrowest path you can, allow the smallest set of names, and let the deny-all rule catch everything you forgot.

Two switches on the CA back this up. Both live in ca.conf, and both are already correct out of the box. Leave them alone, and understand why.

/etc/puppetlabs/puppetserver/conf.d/ca.conf
certificate-authority: {
# A subject alternative name is an extra identity carried inside one certificate.
# Signing a request that asks for other machines' names hands out an impersonation ticket.
allow-subject-alt-names: false
# Extensions under 1.3.6.1.4.1.34380.1.3.* grant access in auth.conf.
# Letting a node request its own authorization is letting it write its own pass.
allow-authorization-extensions: false
}
terminal
sudo puppetserver ca sign --certname build02.acme.internal
output
Error:
Could not sign request for build02.acme.internal.
CSR 'build02.acme.internal' contains subject alternative names (DNS:build02, DNS:puppet, DNS:puppet.acme.internal), which are disallowed. Use `puppetserver ca sign --allow-alt-names build02.acme.internal` to sign this request.

That request asked to also be called puppet, which is the name every agent uses for the server itself. Signing it would mint a certificate that can pose as your Puppet server, and a fake Puppet server hands root-level instructions to the whole fleet. The error helpfully offers --allow-alt-names, and there is exactly one situation that deserves it: certificates for the server itself, or for extra compilers behind a load balancer, where several names genuinely belong to one host. Per request, deliberately, never as a standing setting in ca.conf.

You can read what a certificate actually asserts at any time, which is the quickest way to confirm a role stamp landed where you meant it to.

terminal
CERT=/etc/puppetlabs/puppetserver/ca/signed/db1.acme.internal.pem
sudo openssl x509 -in "$CERT" -noout -subject -dates
sudo openssl x509 -in "$CERT" -noout -text | grep -A1 '1.3.6.1.4.1.34380.1.1.13'
output
subject=CN = db1.acme.internal
notBefore=Jun 2 10:14:03 2026 GMT
notAfter=Jun 1 10:14:03 2031 GMT
1.3.6.1.4.1.34380.1.1.13:
..db-primary

OpenSSL does not know Puppet's short names, so it prints the raw OID and dumps the value with its header bytes showing as dots. Certificates store values in ASN.1 (Abstract Syntax Notation One), where every value carries a type byte and a length byte in front of it, and those are the two dots. notAfter is five years out, straight from ca_ttl. An expired agent certificate fails as a connection error rather than as anything that says the word expired, so put that date somewhere you will see it coming.

Taking A Key Back

Revocation is the stolen-key notice pinned up behind the desk. The CA writes the certificate's serial number onto a certificate revocation list (CRL), and everything that reads the list stops accepting that certificate. Puppet gives you two commands here and people mix them up constantly. revoke kills the certificate and leaves its files on the CA. clean revokes it and deletes the request and the certificate as well, which frees the name for reuse.

terminal
# compromised, and the name must never work again
sudo puppetserver ca revoke --certname old-node.acme.internal
# decommissioned, and the name will come back on a rebuilt machine
sudo puppetserver ca clean --certname decom-node.acme.internal
output
Certificate for old-node.acme.internal has been revoked
Cleaned files related to decom-node.acme.internal
terminal
# the CRL on disk is already updated. Reload the server so it starts enforcing the new one.
sudo systemctl reload puppetserver
sudo openssl crl -in /etc/puppetlabs/puppetserver/ca/ca_crl.pem -noout -text | grep -A1 'Serial Number'
output
Serial Number: 0F
Revocation Date: Jul 21 09:52:14 2026 GMT
--
Serial Number: 12
Revocation Date: Jul 21 09:52:15 2026 GMT
Revocation is not finished until you reload
Puppet Server reads the revocation list when it starts and then holds that copy in memory. Revoke a certificate and walk away, and the loaded copy keeps serving the machine you thought you had cut off, so run systemctl reload puppetserver and treat the job as unfinished until you have. Once it takes hold, the revoked node's runs fail with a TLS alert number 44, certificate revoked, rather than with anything that mentions Puppet by name. Two more traps sit at the other end. puppet ssl clean on the agent removes its local key and certificate, and without that a rebuilt machine keeps presenting the dead one. And if you used revoke rather than clean, the old certificate is still sitting on the CA, so the rebuilt node's fresh request is refused because a signed certificate for that name already exists.

The Server Is The Whole Fleet

Everything above assumes the Puppet server is honest. It compiles the instructions that run as root on every machine you own, it holds the CA key that decides who is who, and it holds the eyaml key (eyaml is encrypted YAML, the usual way Hiera stores secrets) that turns your encrypted data back into plaintext. Root on that one host is root on all of them, plus a copy of every secret, plus the ability to issue itself fresh identities afterwards.

terminal
sudo ls -l /etc/puppetlabs/puppetserver/ca/
sudo ls -l /etc/puppetlabs/puppet/eyaml/
output
total 60
-rw-r--r-- 1 puppet puppet 1029 Jul 21 09:52 ca_crl.pem
-rw-r--r-- 1 puppet puppet 2065 Jun 2 10:11 ca_crt.pem
-rw-r----- 1 puppet puppet 3243 Jun 2 10:11 ca_key.pem
-rw-r--r-- 1 puppet puppet 800 Jun 2 10:11 ca_pub.pem
-rw-r--r-- 1 puppet puppet 932 Jul 21 09:52 infra_crl.pem
-rw-r--r-- 1 puppet puppet 0 Jun 2 10:11 infra_inventory.txt
-rw-r--r-- 1 puppet puppet 0 Jun 2 10:11 infra_serials
-rw-r--r-- 1 puppet puppet 1782 Jul 21 09:41 inventory.txt
drwxr-x--- 2 puppet puppet 4096 Jul 21 09:41 requests
-rw-r----- 1 puppet puppet 5 Jul 21 09:41 serial
drwxr-x--- 2 puppet puppet 4096 Jul 21 09:41 signed
total 8
-rw------- 1 puppet puppet 1704 May 14 08:20 private_key.pkcs7.pem
-rw-r--r-- 1 puppet puppet 1050 May 14 08:20 public_key.pkcs7.pem

Puppet Server runs as the unprivileged puppet user, which is why those files belong to puppet rather than to root. The practical consequence catches people out. Membership of the puppet group, or the ability to run sudo -u puppet, is equivalent to owning your certificate authority. Audit that group the way you audit sudoers. Keep interactive logins on this host down to a handful of named people, and let nothing else share the box.

The code is as privileged as the keys, because the code turns into root commands on two thousand machines. Deploy it with r10k, the tool that pulls each Git branch into a matching Puppet environment, from a repository that requires review. Never by hand-editing files under /etc/puppetlabs/code on the server. (Puppet Enterprise ships Code Manager, which wraps r10k behind a service; on open source Puppet 8 you run r10k yourself.) Run pdk validate and pdk test unit in your pipeline so syntax errors and obvious mistakes never reach a real node, PDK being the Puppet Development Kit, the toolchain for linting and unit-testing modules. Rehearse changes in a separate environment with --noop first, and know the limit of that trick: the agent asks for the environment it wants, so environments are a workflow boundary rather than a security one, unless a node classifier on the server overrides what the agent requested.

PuppetDB deserves the same care and rarely gets it. It stores every node's facts, catalogs and reports, which adds up to a complete map of your estate. By default any client holding a certificate your CA signed can query it, so one compromised web server can read the inventory of everything else. Fix that with an allowlist of the certnames that genuinely need to ask.

/etc/puppetlabs/puppetdb/conf.d/jetty.ini
[jetty]
ssl-host = 0.0.0.0
ssl-port = 8081
ssl-key = /etc/puppetlabs/puppetdb/ssl/private.pem
ssl-cert = /etc/puppetlabs/puppetdb/ssl/public.pem
ssl-ca-cert = /etc/puppetlabs/puppetdb/ssl/ca.pem
# One certname per line. Without this, every node with a signed cert can read
# the facts of every other node. Older releases spell it certificate-whitelist.
certificate-allowlist = /etc/puppetlabs/puppetdb/certificate-allowlist

Three commands before you close this page. One asks whether the front door is propped open, one finds every place your code is still taking a node's word about what it is allowed to be, and one lists everybody who can already read your CA key.

terminal
grep -n '^ *autosign' /etc/puppetlabs/puppet/puppet.conf
sudo grep -rn "facts\['role'\]" /etc/puppetlabs/code --include='*.pp'
getent group puppet
output
14:autosign = /usr/local/bin/puppet-autosign
/etc/puppetlabs/code/environments/production/site-modules/profile/manifests/legacy_classify.pp:7: if $facts['role'] == 'db-primary' {
puppet:x:52:deploy,ci-runner

The first line is fine. The second is a live copy of the bug from earlier in this lesson, still sitting in production. The third is usually the shortest output and the biggest surprise: two accounts that can read ca_key.pem, and anything that can read ca_key.pem can mint a certificate for any name in your fleet, including the Puppet server's own. Check primary groups as well, because getent group puppet lists secondary members only.

Quick check
01Why is $trusted['extensions']['pp_role'] safe to branch on for a security decision while $facts['role'] is not?
Incorrect — Both travel inside the same TLS connection, so encryption is not the difference between them.
Incorrect — Facter reads any file in /etc/puppetlabs/facter/facts.d on any node, production included.
Correct — one is a statement your CA made, the other is a statement the node made about itself.
Incorrect — Both are available during compilation, so the difference is the source of the data, not the timing.
02In csr_attributes.yaml, what happens to values under custom_attributes compared with values under extension_requests?
Correct — which is why a one-time enrollment token belongs in custom_attributes and a role stamp belongs in extension_requests.
Incorrect — Custom attributes are never written into the certificate at all, so there is nothing for a manifest to read.
Incorrect — Neither section is sent to PuppetDB; both travel inside the certificate signing request.
Incorrect — The agent signs the whole request once, so the sections differ in what survives signing, not in who signs them.
03You ran puppetserver ca revoke --certname web7.acme.internal an hour ago after web7 was compromised. puppetserver ca list --all now shows it under Revoked Certificates, but web7 is still fetching catalogs successfully every thirty minutes. What is the most likely cause, and the fix?
Incorrect — ca_ttl sets how long a new certificate is issued for and has nothing to do with when a revocation is enforced.
Correct — the list on disk is updated immediately, but the running service keeps its in-memory copy until you reload or restart it.
Incorrect — revoke does write the serial number to the CRL; clean additionally deletes the files so the name can be reused.
Incorrect — That setting controls how an agent checks the server's chain, not how the server checks agents, so a node cannot opt itself out.

Try this

Run sudo /opt/puppetlabs/bin/puppet agent --test on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: two ways autosign quietly betrays you. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related