Encrypted pillars & secrets

GPG-encrypted data and backends.

A secret sitting in a plain pillar file travels like a postcard. The address names one minion, but every hand it passes through can read the message: the Git repository, the CI (continuous integration) runner that clones it, last night's backup tarball, the colleague with checkout access, anyone who can read the master's disk. GPG-encrypted pillar swaps the postcard for a sealed envelope. Copy it, archive it, push it to a public repository if you feel brave. Only the Salt master holds the key that opens it, and it opens the envelope at the last possible moment, while compiling pillar for the one minion that asked.

Pillar already handles targeting. Each minion receives only its own slice of data, over its own encrypted connection. What pillar does not handle is storage. Pillar SLS files (SaLt State files, the YAML-plus-Jinja documents Salt reads) sit in plaintext on the master's disk and, worse, in version control forever. Salt's answer is the GPG renderer. A renderer is the part of Salt that turns a file on disk into the data structure Salt actually uses, and GPG (GNU Privacy Guard, the standard open-source implementation of the OpenPGP encryption format) hands you a matched pair: a public half that can only lock things, and a private half that opens them. Padlocks and keys. You can hand padlocks to strangers. So you lock a value with the master's public half, commit the ciphertext, and the master unlocks it with the private half while it renders pillar. Same idea as Ansible Vault, SOPS or Puppet's eyaml, wired into Salt's renderer chain.

# the problem: a plain pillar secret is readable at rest, and forever in history
sudo cat /srv/pillar/secrets.sls
 
# git's "pickaxe" search finds every commit that added or removed the string
sudo git -C /srv/pillar log --oneline -S 'S3cr3t!pgAdmin' --all

db:
  host: db1.internal
  user: myapp
  password: S3cr3t!pgAdmin
 
4f9c1ad rotate prod db password
a02e7b1 add prod db password

Deleting that line today changes nothing. Both commits still hand the password to anyone who clones the repository. That is the gap encryption closes.

Give the master a key it never hands out

If you want a sandbox first, the official bootstrap script installs a current Salt release in one shot. -M adds a master alongside the minion, and -A 127.0.0.1 writes that master's address into the minion config, so a single virtual machine can play both roles.

curl -o bootstrap-salt.sh -L https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh
sudo sh bootstrap-salt.sh -M -A 127.0.0.1 stable 3007
 
sudo salt-key -A -y        # lab shortcut; in production compare fingerprints first
sudo salt 'web1' test.ping

 *  INFO: Running install_ubuntu_onedir_deps()
 *  INFO: Running install_ubuntu_onedir()
 *  INFO: Salt installed!
 
The following keys are going to be accepted:
Unaccepted Keys:
web1
Key for minion web1 accepted.
 
web1:
    True

Now the master needs its own GPG keyring, kept well away from any human's ~/.gnupg. A keyring is a directory holding key files, the same way a physical keyring holds keys. Salt looks for one called gpgkeys sitting next to the master config, so /etc/salt/gpgkeys on a stock install, and you can move it with gpg_keydir. Two rules are non-negotiable. The directory must be mode 0700 and owned by the user the master process runs as (root, unless you set user in the master config), because GPG refuses to touch a keyring with loose permissions, and because Salt's renderer shells out to the real gpg binary rather than using a Python library. Whatever gpg can read as that user is exactly what Salt can read. The second rule: the key must have no passphrase, because the master decrypts unattended on every pillar compile, with nobody at a keyboard to type one. There is a third rule nobody warns you about. If you name an explicit algorithm to --quick-generate-key, GPG creates a signing primary key only and skips the encryption subkey, and every later encrypt attempt fails with a baffling error. Pass default default and you get both.

sudo mkdir -p /etc/salt/gpgkeys
sudo chmod 0700 /etc/salt/gpgkeys
 
# 'default default' = default algorithm, default usage -> primary key AND
# an encryption subkey. Naming e.g. rsa4096 here gives you no subkey at all.
sudo gpg --homedir /etc/salt/gpgkeys --batch --passphrase '' \
     --quick-generate-key 'salt-master <[email protected]>' default default never
 
sudo gpg --homedir /etc/salt/gpgkeys --list-keys

gpg: keybox '/etc/salt/gpgkeys/pubring.kbx' created
gpg: /etc/salt/gpgkeys/trustdb.gpg: trustdb created
gpg: directory '/etc/salt/gpgkeys/openpgp-revocs.d' created
gpg: revocation certificate stored as '/etc/salt/gpgkeys/openpgp-revocs.d/3A9C0F1E7B2D4E58A1C43F0D6AFDE92B1F5C7A34.rev'
 
/etc/salt/gpgkeys/pubring.kbx
-----------------------------
pub   ed25519 2026-07-21 [SC]
      3A9C0F1E7B2D4E58A1C43F0D6AFDE92B1F5C7A34
uid           [ultimate] salt-master <[email protected]>
sub   cv25519 2026-07-21 [E]

The line that proves the key is usable is the last one. sub with [E] is the encryption subkey. No [E], no encrypted pillar, and you find out three commits later. If your organisation insists on RSA for compatibility with older tooling, run sudo gpg --homedir /etc/salt/gpgkeys --full-generate-key interactively instead and pick RSA and RSA, 4096 bits, no expiry, empty passphrase.

Encrypt on your laptop, commit the ciphertext

Day-to-day encryption happens wherever you write pillar, never on the master. Export the public half once and pass it around like a padlock anyone may snap shut. The private half stays where it was born and never moves. Two flags on the encrypt command earn their place. --trust-model always stops GPG pausing to ask whether you really believe this key belongs to your master, which it has no way to know because nobody signed it. And echo -n suppresses the trailing newline, so that newline does not quietly become the last character of your password. That one produces the classic ghost bug: the credential works from your shell and fails everywhere else.

# on the master: hand out the public half only
sudo gpg --homedir /etc/salt/gpgkeys --armor --export salt-master > salt-master.pub.asc
 
# on your workstation: import it once
gpg --import salt-master.pub.asc
 
# encrypt a value for the master. -n = no trailing newline in the plaintext.
echo -n 'S3cr3t!pgAdmin' | gpg --armor --batch --trust-model always \
     --encrypt -r salt-master

gpg: key 6AFDE92B1F5C7A34: public key "salt-master <[email protected]>" imported
gpg: Total number processed: 1
gpg:               imported: 1
 
-----BEGIN PGP MESSAGE-----
 
hF4DhoFAT9uL0hASAQdA4mQz8xYb2Nn7RrLpXd4aE9uGm2Vw8sPhY6Tc0oJbN1cw
7yQnR2m5aVXcFj0pKtLbzHwe3MgUiO4vBt9pSnCyKW0rDlQxZa8hVuT2jF6cE1mY
0j0BqTz3b1J1vC5m0kQz8xYb2Nn7RrLpXd4aE9uGm2Vw8sPhY6Tc0oJbN1fRk7Xq
A5dLzHwe3MgUiO4vBt8x
=Xq7v
-----END PGP MESSAGE-----

Paste that block into a pillar file whose first line is the shebang #!yaml|gpg. That line declares the renderer pipeline, read left to right, the way an assembly line runs: parse the file as YAML first, then hand the resulting data structure to the GPG renderer, which walks every value and decrypts anything shaped like a PGP (Pretty Good Privacy) message. Everything else passes through untouched, so plain keys and encrypted keys live happily in the same file. If the file also needs templating, the order becomes #!jinja|yaml|gpg, meaning Jinja runs first and decryption runs last. You can also hand an encrypted value to a one-off job with pillar_enc=gpg on the command line, once you flatten the newlines into literal \n sequences.

#!yaml|gpg
 
db:
  host: db1.internal          # plain and encrypted keys coexist in one file
  user: myapp
  password: |
    -----BEGIN PGP MESSAGE-----
 
    hF4DhoFAT9uL0hASAQdA4mQz8xYb2Nn7RrLpXd4aE9uGm2Vw8sPhY6Tc0oJbN1cw
    7yQnR2m5aVXcFj0pKtLbzHwe3MgUiO4vBt9pSnCyKW0rDlQxZa8hVuT2jF6cE1mY
    0j0BqTz3b1J1vC5m0kQz8xYb2Nn7RrLpXd4aE9uGm2Vw8sPhY6Tc0oJbN1fRk7Xq
    A5dLzHwe3MgUiO4vBt8x
    =Xq7v
    -----END PGP MESSAGE-----

base:
  # exact minion ID, matched against a key the master has already accepted.
  # never gate a secret on a grain: the minion reports its own grains.
  'web1':
    - secrets

Prove the master can actually open it

Do not find out at highstate time, when a hundred minions are already applying states. salt-run pillar.show_pillar compiles pillar on the master for a named minion and prints the result locally, without contacting that minion or putting anything on the wire. It reads the grains the master already has cached for that host, so a minion that has never checked in gets a stub instead. It is still the fastest way to confirm the renderer opened what you committed. One caution: the output is plaintext on your screen, so do not run it in a shared session and do not paste the result into a ticket. After that, saltutil.refresh_pillar tells the minion to fetch a fresh copy, and pillar.get reads a single value back.

# master-side dry compile: renders web1's pillar without sending it anywhere
sudo salt-run pillar.show_pillar 'web1'
 
# then from the minion's point of view
sudo salt 'web1' saltutil.refresh_pillar
sudo salt 'web1' pillar.get db:password

db:
    ----------
    host:
        db1.internal
    password:
        S3cr3t!pgAdmin
    user:
        myapp
 
web1:
    True
 
web1:
    S3cr3t!pgAdmin

Now the failure case, which behaves differently on the version you are running. When a PGP block will not open, older masters logged a warning and passed the raw ciphertext through as the value, so an application quietly received a 200-character string starting -----BEGIN PGP MESSAGE----- as its database password. Salt 3007.0 flipped the default of gpg_decrypt_must_succeed from False to True, so a block the keyring cannot open is now a hard rendering error instead of a silently wrong secret. You see it as an _errors entry in pillar, with the real reason sitting in the master log. To dig out that reason by hand, run the exact command the renderer runs, gpg --homedir /etc/salt/gpgkeys --status-fd 2 --no-tty -d, against the offending block. The --status-fd 2 part is what makes GPG print machine-readable [GNUPG:] lines alongside the human ones.

# a block encrypted for a key this master does not hold
sudo salt 'web1' pillar.items
 
# reproduce the renderer's own call against the offending block
sudo gpg --homedir /etc/salt/gpgkeys --status-fd 2 --no-tty -d /tmp/block.asc

web1:
    ----------
    _errors:
        - Rendering SLS 'secrets' failed. Please see master log for details.
 
[GNUPG:] ENC_TO 9DE5701F2447698D 18 0
gpg: encrypted with cv25519 key, ID 9DE5701F2447698D, created 2025-11-04
      "salt-master-old <[email protected]>"
[GNUPG:] NO_SECKEY 9DE5701F2447698D
[GNUPG:] DECRYPTION_FAILED
gpg: decryption failed: No secret key

What this costs you

Decryption runs on every pillar compile, for every encrypted value, for every minion that matches. Each one forks a gpg process. A handful of secrets across fifty minions is free. Thousands of secrets across thousands of minions becomes measurable master CPU, and pillar compiles are already the slowest thing a busy master does. The opt-in gpg_cache setting skips the repeat work, but read what it does before you switch it on. With gpg_cache: True the master stores the decrypted values in its own cache, using the disk backend by default (gpg_cache_backend: disk) with a 24-hour TTL (time to live, how long an entry stays valid, gpg_cache_ttl: 86400). You have traded CPU for plaintext on the master's filesystem.

Three more gaps deserve saying out loud, because no configuration flag closes them. There is no per-secret audit trail. Salt can tell you pillar was rendered, never who read which value. Rotation is entirely manual: you re-encrypt each value and commit again. And revocation does not exist. If the master's private key ever leaves the building, every ciphertext you have ever committed becomes readable, including the ones you thought you deleted three years ago, because they are still sitting in Git history. Those three gaps are precisely what an external secrets backend buys you.

Do not leak it on the way out

Decrypting safely is half the job. The other half is getting that value into a config file without spraying it somewhere else on the way, the way a careful courier still has to watch what falls out of the envelope. The usual culprit is the state diff. file.managed prints a unified diff of what changed, and that diff lands in your terminal, in the master's job cache under /var/cache/salt/master/jobs/, and in the log of whatever CI job ran the highstate. Set show_changes: False on any state that touches a secret. Cleaner still, contents_pillar pulls the value straight from pillar into the file, so the secret never appears in the SLS at all and there is nothing for a debug log to capture.

#!jinja|yaml
 
deploy-db-config:
  file.managed:
    - name: /etc/myapp/db.conf
    - user: myapp
    - group: myapp
    - mode: '0600'
    - show_changes: False          # keep the value out of diffs, job cache and CI logs
    - contents: |
        DB_USER=myapp
        DB_PASSWORD={{ salt['pillar.get']('db:password') }}
 
# for a file that holds nothing but the secret, keep it out of the template too:
deploy-api-token:
  file.managed:
    - name: /etc/myapp/api.token
    - mode: '0600'
    - show_changes: False
    - contents_pillar: api:token

sudo salt 'web1' state.apply myapp.dbconf test=True

web1:
----------
          ID: deploy-db-config
    Function: file.managed
        Name: /etc/myapp/db.conf
      Result: None
     Comment: The file /etc/myapp/db.conf is set to be changed
     Started: 14:22:07.118432
    Duration: 21.436 ms
     Changes:
              ----------
              diff:
                  <show_changes=False>
 
Summary for web1
------------
Succeeded: 1 (unchanged=1, changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  21.436 ms

Targeting decides who gets the plaintext

Encryption protects the file at rest. It does nothing about delivery. The master decrypts whatever the top file says a minion should have, then sends the plaintext to whichever minion matched. So the match itself is the access control decision, the guard checking a badge at the door. Target secret pillar by minion ID or nodegroup, never by grain. Grains are facts the minion reports about itself, and a minion with root on it can report whatever it likes. grains.setval writes to /etc/salt/grains and takes effect on the next pillar refresh. One command, and a compromised web server claims to be a database server.

sudo salt 'web1' grains.item role
 
# now, as root on the compromised minion itself:
sudo salt-call grains.setval role database

web1:
    ----------
    role:
        web
 
local:
    ----------
    role:
        database

Minion IDs cannot be rewritten that way, because each one is bound to a public key the master explicitly accepted. The second half of the same rule is publisher_acl, the master's list of which users may run which functions. Anyone that ACL (access control list) permits to run pillar.items or pillar.get receives decrypted values, so a generous ACL turns every helpdesk account into a secrets reader and nothing in the logs looks unusual.

When GPG runs out: Vault through ext_pillar and SDB

For secrets with a lifecycle, stop storing them in the Salt tree at all, encrypted or otherwise, and fetch them on demand instead. Think of it as a hotel key card that expires at checkout rather than a cut brass key that works forever. Salt gives you two hooks. An ext_pillar (external pillar) module merges data from an outside system into pillar at compile time, so minions consume it exactly like ordinary pillar keys. SDB (Salt Database) is a small URI scheme, sdb://profile/path/key, that you can drop into master config, pillar values or state arguments, resolved at render time by whichever process reads it. For HashiCorp Vault, use the extension rather than the built-in modules: Salt 3007 deprecated the bundled Vault code in favour of saltext-vault, and it is removed from Salt core in 3009.

# install into the master's bundled Python (onedir packages ship salt-pip)
sudo salt-pip install saltext-vault
sudo systemctl restart salt-master

Collecting saltext.vault
  Downloading saltext.vault-1.7.0-py3-none-any.whl (203 kB)
Installing collected packages: saltext.vault
Successfully installed saltext.vault-1.7.0

vault:
  auth:
    method: approle
    role_id: e5a7b66e-5d08-da9c-7075-71984634b882
    secret_id: 841771dc-11c9-bbc7-bcac-6a3945a69cd9
  server:
    url: https://vault.example.com:8200
  issue:
    type: approle              # mint a per-minion AppRole instead of sharing this one
 
ext_pillar:
  - vault: salt/minions/{minion}     # merged into that minion's pillar at compile time
 
myvault:                       # an SDB profile name you invent
  driver: vault

# resolve a secret from the master. Last path segment is the key inside the secret.
sudo salt-run sdb.get 'sdb://myvault/secret/salt/db/password'

hunter2-rotated-2026-07

That same URI works as a pillar value or a state argument, so you can move one credential at a time out of GPG without rewriting your states. Vault then covers what GPG cannot: leases that expire on their own, policies scoped per role, database credentials minted fresh and never issued twice, and an audit log naming who read what and when.

Picking between the two is an operations decision, not a security absolute. GPG pillar needs no extra service, keeps secrets versioned next to the states that use them, and reviews like any other pull request, which makes it a sane default for a small estate. Vault costs you a service to run, back up and unseal, and buys you rotation, revocation and audit in return. A workable split is GPG for values that barely ever change, a license key or an SMTP password, and Vault for anything with a clock on it. Whichever you land on, make sudo salt-run pillar.show_pillar 'web1' the last thing you type after editing an encrypted file. It is the only step that tells you the master can still open the envelope before a minion discovers it cannot.

Try this

Run sudo cat /srv/pillar/secrets.sls on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: one passphrase-less key opens every envelope. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related