Pillar: secure per-minion data

Targeted, private configuration data.

Two kinds of paper come out of a company's back office. The staff handbook is printed once and handed to everybody. Nothing in it is secret, and a copy left on a train costs nobody anything. Payslips are the opposite: one per person, sealed, given only to the employee named on the envelope. Salt splits its data along exactly that line.

States are the handbook. Shared code, identical on every machine, safe in a repository your whole platform team can read. Pillar is the payslip run. It is structured data, written by default in YAML (a plain-text format for nested lists and key/value pairs), that lives on the master (the central Salt server holding your code and issuing the orders) and is compiled separately for every minion (the Salt agent running on each managed machine). Put a database password in pillar for db* and it never lands on a web server, because the master never sends it there.

The plumbing is a tree of SLS files (SaLt State files: plain YAML that gets pushed through a template engine before anything parses it) sitting on the master under pillar_roots, wired together by their own top file. That top file is a different thing from the state top file you met last lesson, and it does a different job. Read it as the guest list on the door, not the running order of the party.

When a minion asks for pillar, the master works through the pillar top file for that one minion, renders every matched file through Jinja (the templating language Salt runs over your files before it parses the YAML) with that minion's grains in scope, merges the results into a single dictionary, and returns it over the request channel on port 4506. Grains are facts a minion reports about itself: operating system, processor count, hostname. The reply is sealed properly. The master generates a throwaway symmetric key, encrypts the pillar with it, then locks that key with the minion's own public key, so no other minion can read the payload even if it captures the packets off the wire. On the minion, pillar lives in memory. By default it is never written to the minion's disk.

The Trust Anchor Is the Key You Accepted

Targeting only means something if identities cannot be faked. A door policy of "let in anyone wearing a badge that says STAFF" is worthless when the badge printer sits outside the door. In Salt, a minion's identity is its ID plus the public key the master accepted for that ID, and salt-key is where you make that binding by hand. Every pillar match you write later leans on it.

# on the master (Ubuntu 24.04). The bootstrap script ships from GitHub releases now;
# the bootstrap.saltproject.io URL in older tutorials no longer serves it.
curl -fsSL -o bootstrap-salt.sh \
  https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh
 
# -M also installs salt-master. No -P: that flag allows pip-based installs,
# which the packaged build does not need and a master should not have.
sudo sh bootstrap-salt.sh -M stable 3007
 
# on db1, web1, web2: same script, no -M, -A points the minion at the master
sudo sh bootstrap-salt.sh -A salt.acme.internal stable 3007

 *  INFO: sh bootstrap-salt.sh -- Version 2026.07.10
 *  INFO: System Information:
 *  INFO:   CPU:          AuthenticAMD
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   6.8.0-51-generic
 *  INFO:   Distribution: Ubuntu 24.04
 *  INFO: Installing minion
 *  INFO: Running install_ubuntu_stable_deps()
 *  INFO: Running config_salt()
 *  INFO: Running install_ubuntu_stable()
 *  INFO: Running install_ubuntu_stable_post()
 *  INFO: Salt installed!

Nothing is trusted yet. Each minion has sent its public key and is sitting in the waiting room. Before you let one in, compare the fingerprint the master received against the one the box prints about itself, the same way you would read a code back over the phone instead of trusting the caller ID.

# on the master: see who is waiting
sudo salt-key -L
 
# what fingerprint did the master receive for db1?
sudo salt-key -f db1
 
# run this ON db1: what fingerprint does the box itself report?
sudo salt-call --local key.finger
 
# back on the master: accept one key at a time, on purpose
sudo salt-key -a db1
sudo salt '*' test.ping

Accepted Keys:
Denied Keys:
Unaccepted Keys:
db1
web1
web2
Rejected Keys:
 
Unaccepted Keys:
db1:  9d:07:c0:9b:96:67:7c:97:be:f6:cb:95:9c:0e:81:ea:16:86:39:ed:d4:84:3d:e9:9d:0d:29:67:29:a3:8b:1a
 
local:
    9d:07:c0:9b:96:67:7c:97:be:f6:cb:95:9c:0e:81:ea:16:86:39:ed:d4:84:3d:e9:9d:0d:29:67:29:a3:8b:1a
 
The following keys are going to be accepted:
Unaccepted Keys:
db1
Proceed? [n/Y] y
Key for minion db1 accepted.
 
db1:
    True
web1:
    True
web2:
    True

Those fingerprints are SHA-256 (a hashing algorithm that boils a key down to a fixed-length fingerprint), the default hash_type on 3007. Now the shortcuts. salt-key -A -y accepts every waiting key without asking, auto_accept: True in the master config skips the question forever, and the bootstrap script's -Q quickstart flag installs master and minion together and then runs salt-key -yA on your behalf. Every one of those is how a pillar leak begins. Anyone who can reach ports 4505 and 4506 picks their own minion ID, so a rogue box calling itself db-rogue matches the glob db* on the very next refresh. The master ships with auto_accept: False for that reason.

pillar_roots, and a Top File That Is Really a Guest List

Point the master at a directory, restart it once, then write down who gets what.

pillar_roots:
  base:
    - /srv/pillar
 
# Master CONFIG changes need a restart: sudo systemctl restart salt-master
# Files under /srv/pillar do not. They are read fresh on every compile.

base:
  '*':
    - common          # harmless defaults, fine for everyone
  'db*':
    - db_secrets      # minion IDs starting with db, and nothing else

# an NTP (Network Time Protocol) server address nobody needs to hide
ntp_server: time.acme.internal

# Rendered ON THE MASTER, once per matching minion,
# with that minion's grains in scope.
db:
  host: db1.acme.internal
  port: 5432
  password: 'S3cr3t-rotate-me'
{% if grains['os_family'] == 'Debian' %}
  service: postgresql
{% else %}
  service: postgresql-server
{% endif %}

Read every line of that top file as an entitlement rather than as behaviour. A bad match in the state top file installs the wrong package on the wrong box, and you find out because something breaks. A bad match in the pillar top file hands over data, quietly, and nothing in the output complains. Changing 'db*' to '*' is one keystroke and it is a breach.

Because each matched file is rendered per minion, the Jinja above sees whichever minion is being compiled for right now. One file emits a Debian service name for one machine and a Red Hat one for the next, with nothing duplicated. All matched files then merge into a single dictionary, the way you would stack recipe cards and read whichever line ended up on top. The default pillar_source_merging_strategy: smart merges nested dictionaries recursively, and where two files set the same key, the file matched later in the top file wins. Lists replace instead of joining end to end, unless you flip pillar_merge_lists: True, which is off by default and worth leaving off. A list of permitted SSH (Secure Shell) groups that silently grows by appending is a nasty way to lose an afternoon. Namespace your keys (db:, app:, monitoring:) so a collision shows up as an obvious duplicate in code review instead of a value that quietly disappears in production.

What Actually Goes Stale, and What Does Not

Saving a file under /srv/pillar changes nothing on any running minion by itself. Picture a shop with the price list taped inside the till. Head office can rewrite the master copy all morning; the till keeps showing the taped-up version until somebody walks over and replaces the paper.

So which commands read the taped-up copy, and which phone head office? pillar.get, pillar.item and pillar.raw read the minion's in-memory copy. That copy is built when the minion daemon starts and changes only when something runs saltutil.refresh_pillar. pillar.items with no arguments goes the other way: it calls the master and compiles fresh every single time, and it leaves the in-memory copy exactly as it found it.

Here is the part almost everyone has backwards. A state run compiles pillar fresh. Publish salt 'db1' state.apply app.db from the master and it fetches newly compiled pillar for that run, so your new password lands in the config file while salt 'db1' pillar.get db:password still reports the old one. Salt's own documentation spells it out: change pillar, run states, and the states see the change even though pillar.item does not. The stale in-memory copy still matters plenty, because beacons and reactors (Salt's event watchers and the actions they trigger) and every execution module that reads a setting through config.get are all served from it.

sudo salt '*' saltutil.refresh_pillar wait=True timeout=60
sudo salt 'db1' pillar.items

db1:
    True
web1:
    True
web2:
    True
 
db1:
    ----------
    db:
        ----------
        host:
            db1.acme.internal
        password:
            S3cr3t-rotate-me
        port:
            5432
        service:
            postgresql
    ntp_server:
        time.acme.internal

Without wait=True, refresh_pillar fires an event on the minion and returns straight away. The True you get back means the event went out, not that the new data arrived. A script that refreshes and then immediately reads pillar.get, or calls a module that reads pillar internally, can win that race and act on the old value. wait=True timeout=60 blocks until the minion reports the refresh finished, and that is the form you want in a pipeline.

Prove It in Both Directions

Confirming that db1 got the password proves nothing about who else did. Auditing the other direction is the half people skip, and it costs one command.

# the check that actually matters: who else can read it?
sudo salt '*' pillar.get db:password
 
# what did the Jinja branch actually see?
sudo salt 'db1' grains.item os_family
 
# render a minion's pillar entirely on the master, without touching the minion
sudo salt-run pillar.show_pillar db1

db1:
    S3cr3t-rotate-me
web1:
web2:
 
db1:
    ----------
    os_family:
        Debian
 
db:
    ----------
    host:
        db1.acme.internal
    password:
        S3cr3t-rotate-me
    port:
        5432
    service:
        postgresql
ntp_server:
    time.acme.internal

Those two blank lines under web1 and web2 are the result you are looking for. pillar.get hands back an empty string when the key is missing, because pillar_raise_on_missing is off unless you switch it on, so a blank means the master never sent the value. Run that one-liner after every top-file change. It is the cheapest leak test you own. The last command is a runner, meaning it executes on the master itself rather than on any minion: salt-run pillar.show_pillar db1 compiles the whole thing master-side from the minion's cached grains, without contacting the box at all, which makes it a sane check to wire into CI (continuous integration, the automated run that inspects a change before it merges). From the box itself, salt-call pillar.items does the same job locally, and salt-ssh 'db1' pillar.items works agentless as long as the host has a roster entry.

The Grain Trap

Sooner or later somebody proposes targeting by role instead of by name, because db* does not survive contact with a fleet whose hostnames were chosen by four different teams over six years.

# The tempting version. Do not ship this one.
base:
  '*':
    - common
  'role:db':
    - match: grain
    - db_secrets

Grains are facts the minion reports about itself, and that phrase is doing all the work. A grain is a name badge the guest fills in out in the car park. Built-in grains like os_family are read off the machine honestly, but any grain can be written by whoever has root on that machine, in /etc/salt/grains or with one command. grains.setval even refreshes pillar afterwards, since refresh_pillar=True is its default, so the whole attack is a single line typed on a box the attacker already owns.

# on a compromised web1, as root:
salt-call grains.setval role db     # writes /etc/salt/grains AND refreshes pillar
salt-call pillar.get db:password    # a fresh salt-call compiles pillar from scratch
 
# prove the same thing from the master without compromising anything:
# extra key=value args to this runner are merged into the minion's grains
sudo salt-run pillar.show_pillar web1 role=db

local:
    ----------
    role:
        db
 
local:
    S3cr3t-rotate-me
 
db:
    ----------
    host:
        db1.acme.internal
    password:
        S3cr3t-rotate-me
    port:
        5432
    service:
        postgresql
ntp_server:
    time.acme.internal

Consume It in a State Without Leaking It Back

Inside a state or a template, {{ pillar['db']['password'] }} works right up to the moment the key is missing, at which point rendering dies with a KeyError and takes the whole run down with it. Write salt['pillar.get']('db:password') instead. Colons walk down the nesting, and you can pass a fallback as a second argument.

The subtler leak is on the way back. State results travel from the minion to the master, land in the master's job cache under /var/cache/salt/master/jobs, and are forwarded to whatever returners you have wired up (returners are plugins that copy job results somewhere else: Slack, Elasticsearch, a ticketing webhook). A file.managed state that rotates a password puts the old and the new value into its diff, and that diff rides along to every one of those places like a carbon copy you forgot was in the pad. On first creation the diff reads only New file, so it looks harmless. The day you rotate the value is the day it bites. show_changes: False swaps the diff for a placeholder in real runs and in test=True runs alike. For a file whose entire content is the secret, contents_pillar pulls the value straight out of pillar so it never passes through a template. Lock the mode too, quoted so YAML does not read it as a decimal number. Pillar controls delivery; once the value is on disk it is an ordinary file with ordinary permissions.

# State code stays generic. The data comes from pillar.
db_config:
  file.managed:
    - name: /etc/app/db.conf
    - user: root
    - group: root
    - mode: '0600'
    - show_changes: False      # keeps the diff out of returns and the job cache
    - contents: |
        host={{ salt['pillar.get']('db:host', 'localhost') }}
        port={{ salt['pillar.get']('db:port', 5432) }}
        password={{ salt['pillar.get']('db:password') }}
 
# for a file that is nothing but the secret, skip the template entirely
deployer_key:
  file.managed:
    - name: /root/.ssh/id_ed25519
    - user: root
    - mode: '0600'
    - show_changes: False
    - contents_pillar: deploy:ssh_private_key

sudo salt 'db*' state.apply app.db

db1:
----------
          ID: db_config
    Function: file.managed
        Name: /etc/app/db.conf
      Result: True
     Comment: File /etc/app/db.conf updated
     Started: 09:14:32.118632
    Duration: 41.219 ms
     Changes:
              ----------
              diff:
                  <show_changes=False>
----------
          ID: deployer_key
    Function: file.managed
        Name: /root/.ssh/id_ed25519
      Result: True
     Comment: File /root/.ssh/id_ed25519 updated
     Started: 09:14:32.160102
    Duration: 12.884 ms
     Changes:
              ----------
              diff:
                  <show_changes=False>
 
Summary for db1
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:  54.103 ms

When a pillar file fails to render, the master deliberately keeps its mouth shut. pillar_safe_render_error defaults to True, so the minion gets a stub message while the real traceback, which might quote the very secret that broke the template, stays in /var/log/salt/master. States then refuse to run at all rather than write a half-empty config, which is the behaviour you want.

sudo salt 'db1' pillar.items
sudo salt 'db1' state.apply app.db

db1:
    ----------
    _errors:
        - Rendering SLS 'db_secrets' failed. Please see master log for details.
    ntp_server:
        time.acme.internal
 
db1:
    Data failed to compile:
----------
    Pillar failed to render with the following messages:
----------
    Rendering SLS 'db_secrets' failed. Please see master log for details.
ERROR: Minions returned with non-zero exit code

Where Pillar Stops Being a Secret Store

Pillar decides who receives a value. It does not encrypt anything at rest. /srv/pillar is plain text to anyone with root on the master and to anyone holding a clone of the repository behind it, which is usually a longer list of people than you would guess.

Every compile is real work: walk the top file, render Jinja for this one minion, merge dictionaries, once per minion per refresh. Fire salt '*' saltutil.refresh_pillar at a few thousand minions and you get to watch the master's processors pin. pillar_cache: True remembers the result for pillar_cache_ttl seconds (3600 by default), which solves that and buys you a different problem. It is the difference between sealing each payslip on demand and photocopying the whole run into a drawer anyone can open. With the default pillar_cache_backend: disk, rendered pillars are serialised into the master cache, and Salt's own documentation states in capitals that pillars are stored UNENCRYPTED. Anything a renderer decrypted during the compile now sits in plain text under /var/cache/salt/master. Price that in before you switch it on, and remember salt-run pillar.clear_pillar_cache exists for the day a rotated secret has to leave that cache before its TTL (time to live) runs out.

A handful of master settings decide how much of this holds up. pillar_opts defaults to False, and leaving it there keeps the master's own configuration file out of every minion's pillar. publisher_acl (ACL is short for access control list) lets non-root users on the master run particular modules, and a user allowed pillar.items against '*' can read every secret belonging to every minion they may target, so scope those entries as tightly as you scope the top file. The trap is sudo_acl, which defaults to False: a user with sudo rights on the salt command walks straight past publisher_acl unless you set it, and even then the check only bites while publisher_acl has entries in it.

auto_accept: False               # default; never flip it on a master holding secrets
open_mode: False                 # default; True accepts all authentication
pillar_opts: False               # default; True ships the master config to every minion
pillar_safe_render_error: True   # default; keeps render tracebacks off the minions
sudo_acl: True                   # NOT the default; enforce publisher_acl under sudo too
 
publisher_acl:
  deploy:
    - test.ping
    - state.apply
 
# CAREFUL: the module blacklist is checked before the master authenticates
# anyone, so it applies to root too. Blacklist pillar.* here and your own
# leak test stops working. Keep it narrow.
publisher_acl_blacklist:
  modules:
    - cmd.run
    - cmd.script

For data too large or too dynamic to sit in files, ext_pillar plugs outside sources into the same per-minion compile: a git repository, a database, an HTTP API (a web service the master queries over the network), HashiCorp Vault. The Vault pillar templates the minion name into the lookup path (- vault: path=secret/minions/{minion}/db), so each box only ever pulls its own branch. The at-rest gap is what the GPG renderer closes (GPG is GNU Privacy Guard, the standard command-line encryption tool), with values encrypted inside git and decrypted only during the master's compile, and that workflow gets its own lesson shortly under *Encrypted pillars & secrets*.

Before you close this tab, add one check to your pipeline. For every secret key you keep in pillar, run sudo salt '*' pillar.get db:password and fail the build if a single minion outside the intended set answers with anything other than a blank line.

Try this

Run sudo sh bootstrap-salt.sh -M stable 3007 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a pillar match is an authorization decision, so never gate one on a grain. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related