Securing the Salt infrastructure

Key acceptance, the master, ACLs.

A Salt master is the key-cutting bench for a whole building. Ask it for a key to any door and it cuts one in about a second, with nothing downstream asking a second question. Whoever sits at that bench owns the building. From a master you run commands as root on every machine you manage, all of them, at the same time.

Securing Salt splits into three separate jobs, and none of them can cover for the other two. First, deciding which machines get a key cut at all: key acceptance. Second, bolting down the bench itself: the master host. Third, deciding who may sit at the bench and which keys they are allowed to cut: access control lists, or ACLs, which are written rules about who may run what against which machines, checked against your real accounts.

Every control in this lesson hangs off the same piece of machinery, so start with how a machine joins in the first place. A minion (the Salt agent, a small background program running on each managed box) makes itself a keypair the first time it starts. That is RSA, 2048 bits by default, controlled by the keysize setting. Public-key cryptography works like a padlock you hand out: the public half is the open padlock, anybody may take a copy, and the private half is the only thing that opens it. The private half never leaves that machine.

The minion posts its public half to the master's request port, TCP 4506, tagged with whatever name it decided to call itself. That name is the minion ID. The master files the key in an unaccepted pile and then does nothing at all. No states, no pillar, no commands, however long the minion waits. Once a human accepts the key, the master hands over the shared AES key (the Advanced Encryption Standard, the fast cipher that scrambles the real traffic), wrapped so only that minion's private half can open it. From then on jobs go out on TCP 4505 and results come back on 4506.

The Lab, And What test.ping Really Proves

You need something to break before hardening means anything. The salt-bootstrap script works out your distribution and pulls packages from the Salt Project repositories, which have lived on Broadcom infrastructure since Broadcom bought VMware. The -M flag installs the master service alongside the minion. The -A flag writes the master's address into /etc/salt/minion.d/99-master-address.conf, which is how a brand new minion knows where to post its key.

# on the master VM: master and minion services, pinned to the 3007 series
curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh -o bootstrap-salt.sh
sudo sh bootstrap-salt.sh -M stable 3007
 
# on each minion VM: install the agent and point it at the master
sudo sh bootstrap-salt.sh -A salt-master.acme.internal stable 3007

 *  INFO: sh bootstrap-salt.sh -- Version 2024.10.16
 *  INFO: System Information:
 *  INFO:   CPU:          GenuineIntel
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   5.15.0-113-generic
 *  INFO:   Distribution: Ubuntu 22.04
 *  INFO: Installing minion
 *  INFO: Installing master
 *  INFO: Found function install_ubuntu_stable_deps
 *  INFO: Found function install_ubuntu_stable
 *  INFO: Running install_ubuntu_stable_deps()
 *  INFO: Running install_ubuntu_stable()
 *  INFO: Running install_ubuntu_check_services()
 *  INFO: Running install_ubuntu_restart_daemons()
 *  INFO: Running daemons_running()
 *  INFO: Salt installed!

Nothing is manageable yet. Each fresh minion has posted its public key and is sitting in the waiting room, knocking again every ten seconds (acceptance_wait_time, which defaults to 10). Run salt-key -L and the master shows you the four piles it keeps. In this lab web1 and web2 were accepted last week, and web3 is the machine you just built.

sudo salt-key -L

Accepted Keys:
web1.acme.internal
web2.acme.internal
Denied Keys:
Unaccepted Keys:
web3.acme.internal
Rejected Keys:

Three of those piles explain themselves. Denied is the one worth understanding early, because the master files a key there for exactly one reason: a machine claiming an ID you have already accepted has turned up holding a different key. The master log does not mince words about it, saying the public keys did not match and that this may be an attempt to compromise the Salt cluster. Sometimes it really is a host you rebuilt and forgot to delete first. Sometimes it is not. Either way you find out which before you touch anything.

Check The Fingerprint Before You Accept

A name proves nothing. Anything that can reach port 4506 can send a key and call itself web3.acme.internal, including a laptop on hotel wifi belonging to somebody who guessed your naming scheme. Accept that key and you have handed the machine your states, the pillar secrets that belong to that name, and a permanent seat on the bus.

Identity comes from the fingerprint instead. A fingerprint is a short hash of the public key, computed with SHA-256 by default and controlled by the hash_type setting, and a hash is a fixed-length summary you cannot run backwards, so two different keys will not produce the same string. Using one is like reading a long serial number down the phone to check two people are holding the same object. Read it on the master. Read it on the machine itself. Accept only when the two strings match character for character, and take that second reading out of band, meaning through a channel an attacker sitting on your network cannot touch: the serial console, the cloud-init log, the output of the provisioning job that built the host.

# on the master: the fingerprint of the key that showed up
sudo salt-key -f web3.acme.internal

Unaccepted Keys:
web3.acme.internal:  d2:41:96:0b:a1:88:52:c9:6e:f3:0d:27:b4:7c:3e:19:8a:5d:c0:44:ef:72:31:9b:06:e8:5f:aa:13:c7:60:2d

# on web3's own console: the fingerprint the minion really generated
sudo salt-call --local key.finger

local:
    d2:41:96:0b:a1:88:52:c9:6e:f3:0d:27:b4:7c:3e:19:8a:5d:c0:44:ef:72:31:9b:06:e8:5f:aa:13:c7:60:2d

The --local flag tells salt-call to answer from the minion's own files without contacting the master, which is exactly what you need for a machine whose key nobody has accepted yet. Matching strings mean one machine sent one key. Different strings mean two machines, and one of them is not yours. Compare first, accept second, then prove the whole chain end to end.

sudo salt-key -a web3.acme.internal
sudo salt 'web3*' test.ping

The following keys are going to be accepted:
Unaccepted Keys:
web3.acme.internal
Proceed? [n/Y] y
Key for minion web3.acme.internal accepted.
 
web3.acme.internal:
    True

test.ping has nothing to do with the ping in your shell, which throws ICMP (Internet Control Message Protocol) packets at an address and times the echo. This one takes a full lap through the security machinery. The master publishes an encrypted job on 4505. Only a minion holding an accepted key can decrypt it. The answer travels back over the return channel on 4506. A True at the end means authentication, encryption and transport are all working, which is worth considerably more than a name that resolves.

Two more subcommands belong in your fingers. salt-key -d deletes a key and salt-key -r rejects one. Delete is for decommissioned machines: the master forgets them, and if that host ever comes back it lands in the unaccepted pile like a stranger. Reject is for keys you have decided are hostile: the master files them under Rejected, and the minion logs that the Salt Master has rejected this minion's public key on every single retry, forever. Either action also rotates the shared AES key, because rotate_aes_key is on by default, so the machine you removed cannot read anything published afterwards. What neither action can do is claw back pillar the host already received. That data is on its disk. Rotate those secrets by hand, the same day.

Trust has to run in both directions, and this is the half people forget. Minions find their master by name, and DNS (the Domain Name System, the phone book that turns salt-master.acme.internal into an address) can be rewritten. Anybody who wins a DNS race, poisons a resolver or edits a hosts file can point your machine at an impostor master, which then feeds it states of its choosing, as root, with nothing else checking. The fix is to pin the real master's public key fingerprint into the minion config as master_finger. Read it with salt-key -F master and take the master.pub line. Never master.pem: that is the private half, and it never leaves the box.

sudo salt-key -F master

Local Keys:
master.pem:  1e:5b:c4:33:80:a9:6d:12:f7:04:be:29:57:d1:8f:3a:62:cc:0e:75:91:4d:ab:20:e8:36:7f:c5:18:b3:44:d9
master.pub:  6f:2c:9a:41:0d:b8:77:e3:15:aa:52:c0:38:9f:64:1b:d7:2e:83:50:c9:16:ff:4a:0b:71:e6:35:8c:d2:19:a7
Accepted Keys:
web1.acme.internal:  3a:88:d0:17:6b:f2:45:9c:2e:71:b5:08:cd:3f:a6:14:52:e9:07:bb:96:2d:c1:78:40:1f:d3:6a:e5:9b:22:0c
web2.acme.internal:  7c:19:ae:60:33:d8:52:0b:94:6f:21:c7:4e:a0:15:8d:b6:39:72:e4:0a:5c:83:f1:26:db:47:90:1a:68:ff:3e
web3.acme.internal:  d2:41:96:0b:a1:88:52:c9:6e:f3:0d:27:b4:7c:3e:19:8a:5d:c0:44:ef:72:31:9b:06:e8:5f:aa:13:c7:60:2d
Denied Keys:
Unaccepted Keys:
Rejected Keys:

# Written at build time by your image or cloud-init, never by hand on a live host.
master: salt-master.acme.internal
 
# The master.pub fingerprint from 'salt-key -F master'. A minion that reaches an
# impostor logs a CRITICAL mismatch and refuses to finish authenticating: it keeps
# retrying rather than taking orders from the wrong master.
master_finger: '6f:2c:9a:41:0d:b8:77:e3:15:aa:52:c0:38:9f:64:1b:d7:2e:83:50:c9:16:ff:4a:0b:71:e6:35:8c:d2:19:a7'

Autoscale Without Propping The Gate Open

Accepting by hand stops working the night an autoscaling group replaces twelve machines while you are asleep. The tempting fix is auto_accept: True, which is a front door with the latch taped over. Price it from the attacker's side. Reach 4506. Pick a name your top file matches, say web9.acme.internal. Get accepted automatically. Receive that role's pillar. Read the database password out of it. Elapsed time: seconds. And open_mode: True is worse again, because it switches off key checking at both ends. It exists so you can dig yourself out of a mangled keypair, not so you can run a fleet.

The workable middle is grain-based autosign, which is a doorman with a guest list. The minion presents a pre-shared value as a grain inside its authentication request, and the master signs the key only if that value appears in a file the master controls. The name on the request is still chosen by whoever is knocking. The password is not.

# Bind the publish and request ports to the address minions actually use,
# rather than every address the box happens to have.
interface: 10.20.0.5
 
auto_accept: False                            # the default. Keep it.
open_mode: False                              # the default. Really keep it.
 
# Sign a new key only if the auth request carries a grain whose value is listed
# in a file under this directory. File name = grain name, one value per line.
#   /etc/salt/autosign_grains/autosign_key   (0600, owned by the master's user)
autosign_grains_dir: /etc/salt/autosign_grains

# Dropped in by cloud-init user data at first boot.
 
# Only grains named here travel with the auth request, so a compromised minion
# cannot smuggle extra facts into the master's decision.
autosign_grains:
  - autosign_key
 
grains:
  autosign_key: 7f2a0c9d4b1e8a35c6f0d2b7e94a1c58

Be honest about what that string is: a bearer token, one secret that works for anybody holding it, like a numbered cloakroom ticket. Two consequences follow. Bake it into a golden image (a prebuilt disk image you clone for every new host) and it lives as long as that image does, on every copy, including the one somebody snapshots out of your cloud account. And the authentication request carrying it crosses the network on a clear channel, because no shared key exists yet at that point in the handshake. Only the token field inside is wrapped in the master's public key, so anybody sniffing that segment reads your autosign value in plain text. Use a different value per environment, keep it out of Git, rotate it with your other build-time secrets, and do not run enrollment across a network you would not trust with a password. It buys unattended enrollment without handing the front door key to the whole internet, which is the trade auto_accept makes.

Harden The Master Host

The master runs arbitrary code as root on every machine it manages, which makes it the most valuable host you own. Treat it the way you treat a domain controller. Nothing else runs on it. It gets patched on a days clock rather than a quarters clock. And it does not listen to the whole world: set interface in /etc/salt/master to the address minions actually use, then put packet filters in front of 4505 and 4506 as well. A bind address controls one program's behaviour. A filter is enforced by the kernel for everything, including the service you forgot was listening.

Write the filtering as a state rather than a runbook step, so it is reviewable, repeatable, and lands identically on the staging master. Pull the allowed range from pillar so one file covers both networks, and make the state refuse to render when that value is missing, because a firewall rule with a blank source address is how you cut an entire fleet off from its master in one command. The range is written in CIDR notation (Classless Inter-Domain Routing), the 10.20.0.0/16 form, where the number after the slash says how many leading bits of the address are fixed.

{%- set allowed = salt['pillar.get']('salt:minion_cidr', '') %}
 
{%- if not allowed %}
# No CIDR in pillar. Fail loudly instead of writing a rule that locks the fleet out.
master_hardening_needs_pillar:
  test.fail_without_changes:
    - comment: 'set salt:minion_cidr in pillar before applying master_hardening'
 
{%- else %}
 
iptables-persistent:
  pkg.installed: []          # Debian family: replays /etc/iptables/rules.v4 at boot
 
{%- for port in [4505, 4506] %}
allow_minions_{{ port }}:
  iptables.append:
    - table: filter
    - chain: INPUT
    - jump: ACCEPT
    - proto: tcp
    - dport: {{ port }}
    - source: {{ allowed }}
    - save: True
    - require:
      - pkg: iptables-persistent
 
drop_others_{{ port }}:
  iptables.append:
    - table: filter
    - chain: INPUT
    - jump: DROP
    - proto: tcp
    - dport: {{ port }}
    - save: True
    - require:
      - iptables: allow_minions_{{ port }}
{%- endfor %}
{%- endif %}

# dry run first, then the real thing
sudo salt 'salt-master*' state.apply master_hardening test=True
sudo salt 'salt-master*' state.apply master_hardening

salt-master.acme.internal:
----------
          ID: iptables-persistent
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 09:41:11.882014
    Duration: 812.443 ms
     Changes:
----------
          ID: allow_minions_4505
    Function: iptables.append
      Result: True
     Comment: Set iptables rule for allow_minions_4505 to: -p tcp -s 10.20.0.0/16 --dport 4505 -j ACCEPT for ipv4
              Saved iptables rule allow_minions_4505 for ipv4
     Started: 09:41:12.704203
    Duration: 61.204 ms
     Changes:
              ----------
              locale:
                  allow_minions_4505
 
(3 more results: drop_others_4505, allow_minions_4506, drop_others_4506)
 
Summary for salt-master.acme.internal
-------------
Succeeded: 5 (changed=4)
Failed:    0
-------------
Total states run:     5
Total run time:   1.093 s

Read the summary, then stop trusting it. A changed=4 on the first run and no changed count on any run after it proves the state is idempotent (running it twice changes nothing the second time). It does not prove a single socket is closed. Verify that from outside Salt, with tools that have no idea Salt exists.

# is the master still listening on every address?
sudo ss -lntp | grep -E '450[56]'
 
# and did the rules actually land, in the right order?
sudo iptables -S INPUT

LISTEN 0  1000   10.20.0.5:4505   0.0.0.0:*  users:(("salt-master",pid=1412,fd=25))
LISTEN 0  1000   10.20.0.5:4506   0.0.0.0:*  users:(("salt-master",pid=1418,fd=27))
 
-P INPUT ACCEPT
-A INPUT -s 10.20.0.0/16 -p tcp -m tcp --dport 4505 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4505 -j DROP
-A INPUT -s 10.20.0.0/16 -p tcp -m tcp --dport 4506 -j ACCEPT
-A INPUT -p tcp -m tcp --dport 4506 -j DROP

Order is the whole game in a packet filter. The kernel reads the chain top to bottom and the first match wins, so the ACCEPT for your minion range has to sit above the DROP for everyone else. The require lines guarantee the relative order of those two, because states run in sequence and each append lands at the bottom of the chain. What they cannot guarantee is what sits above them. A broad ACCEPT installed earlier by Docker or by your cloud image will match first and your DROP will never be reached, so read the whole chain rather than only your own two rules. On a current distribution these iptables commands are a front end onto the nftables backend anyway, and Salt ships nftables and firewalld state modules if you would rather write your platform's native language.

Scope Who May Publish What

Out of the box Salt offers two access levels: root on the master, which is root on every machine in the fleet, and nothing whatsoever. Real teams need the floor in between. The build pipeline should deploy web servers and never go near the database fleet. The on-call engineer should be able to restart a service at 3am without being able to read pillar.

Two settings give you that floor. publisher_acl grants named Linux users specific functions against specific targets, the way a building key can be cut for three doors instead of all of them. external_auth, usually shortened to eauth, does the same for real people through PAM (Pluggable Authentication Modules, the machinery that already checks Linux logins) or LDAP (the Lightweight Directory Access Protocol, which is what your company's user directory speaks). Wire it to LDAP and Salt access follows the joiners and leavers process you already run, instead of drifting quietly in a config file nobody reads.

publisher_acl:                   # local Linux users running the 'salt' command
  ci-deploy:                     # the account the pipeline runs as
    - 'web*':
      - state.apply
      - service.restart
 
publisher_acl_blacklist:         # checked for every publisher, root included
  modules:
    - cmd.*
    - module.*
 
external_auth:                   # real identities, checked by PAM
  pam:
    jchen:
      - 'web*':
        - test.ping
        - service.restart
      - '@runner':               # master-side runners, for reading job history
        - jobs.list_jobs
        - jobs.lookup_jid
    'ops-oncall%':               # a trailing % means a Linux group, not a user
      - 'web*':
        - test.ping
        - service.restart
 
# then:  salt -a pam 'web*' service.restart nginx
#        username: jchen
#        password: ********
# add -T to cache a token in ~/.salt_token for token_expire seconds (43200 default)

Restart the master after editing that file. Then deal with the detail that eats an afternoon: a non-root user also needs read access to the master's runtime directories, or every command dies with a message about failing to authenticate and possibly not being permitted to execute commands. That reads like an ACL problem and is actually a file permission problem. The documented fix is one line: chmod 755 /var/cache/salt /var/cache/salt/master /var/cache/salt/master/jobs /var/run/salt /var/run/salt/master. With that done, compare an allowed publish against a refused one.

# as the pipeline's own Linux user, not root
sudo -u ci-deploy salt 'db1*' state.apply postgres
sudo -u ci-deploy salt 'web1*' service.restart nginx

Authorization error occurred.
 
web1.acme.internal:
    True

The refusal never becomes a job, so it leaves no trace in the job cache at all. It lands in /var/log/salt/master as a warning reading: Authentication failure of type "user" occurred. That wording is misleading. The user authenticated fine, and the ACL then refused the target. Two different failures wear similar words, so tell them apart by what the client prints. An authorization error means the master heard you and said no. A failure to authenticate usually means permissions on those cache directories.

Now the limitation, stated plainly. publisher_acl is an authorization filter, not a security boundary. Every function it allows still runs as root on the minion, so allowlisting cmd.run or module.run hands out a fleet-wide root shell with a little extra typing in front of it. That is why the blacklist above blocks both for everybody, root included. And the filter is only the master process reading its own config file, so anyone with a shell on the master can edit that file, restart the service, or skip Salt entirely and use the master's keys directly. Shell on the master is root everywhere. Keep the list of people who have it short enough to say out loud.

Grains Lie, Pillar Does Not

Salt's two data channels look alike and have opposite trust properties. Mixing them up is how a secret ends up on a machine that should never have seen it. Grains are what a visitor writes on the sign-in sheet at reception: facts a minion works out about itself and reports upward, such as operating system, memory, and whatever role labels you set. Pillar is the sealed envelope the front desk addresses to one room number: data compiled on the master and handed to exactly one minion, matched against the ID that minion's accepted key vouches for.

A rooted minion can write anything it likes on the sign-in sheet. A different role, a different datacenter, somebody else's hostname. The one thing it cannot forge is the ID its key was accepted under, because the master checks the key, not the claim.

sudo salt 'web3*' grains.item roles
sudo salt 'web3*' pillar.items

web3.acme.internal:
    ----------
    roles:
        - webserver
 
web3.acme.internal:
    ----------
    app_env:
        production
    db_password:
        Fs9!kQx2-vLp

The first block is the machine's claim about itself. The second was assembled on the master, for that one machine, and handed to nobody else. Write your pillar top file accordingly.

base:
  # Identity vouched for by an accepted key. Safe to hang secrets off.
  'web3.acme.internal':
    - secrets.web3
 
  # Never do this for secret data. The minion chooses its own grains, so a
  # compromised web server can relabel itself and ask for the vault pillar.
  # 'roles:vault':
  #   - match: grain
  #   - secrets.vault

The same thinking covers the GPG key from the encrypted-pillar lesson. GPG (GNU Privacy Guard) is the tool that encrypts individual values inside your pillar files, and the private key under /etc/salt/gpgkeys decrypts every one of them. So that directory is mode 0700, owned by the user the master runs as, it stays out of ordinary backups along with /etc/salt/pki/master/master.pem, and it gets rotated when anybody who could have read it leaves the team. While you are in the config, leave pillar_opts at its default of False, or the master's entire configuration rides along inside every minion's pillar for anyone on any minion to read.

Keep The Receipts, And Keep A Way In

Every publish that survives the ACL check is written to the master's job cache. That is your ledger for the most privileged tool in the building, and like any ledger it is worth exactly what its retention and its tamper-resistance are worth.

sudo salt-run jobs.list_jobs

20260721094112034521:
    ----------
    Arguments:
        - master_hardening
    Function:
        state.apply
    StartTime:
        2026, Jul 21 09:41:12.034521
    Target:
        salt-master*
    Target-type:
        glob
    User:
        sudo_jchen

Look at the User field. Running the command line under sudo records sudo_jchen rather than a flat root, so the cache can tell you which human published what, and eauth jobs record the eauth username the same way. Two settings decide how much that is worth. keep_jobs_seconds defaults to 86400, one day, which is far too short for an investigation that opens on a Monday about something that happened over the weekend. And master_job_cache can point at an external returner, so the record lives somewhere an attacker holding root on the master cannot quietly edit it.

Enrollment attempts show up on the event bus live. Leave this running for a minute on any master with a public DNS name and watch who knocks.

sudo salt-run state.event pretty=True

salt/auth	{
    "_stamp": "2026-07-21T09:39:58.113344",
    "act": "pend",
    "id": "web3.acme.internal",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A...\n-----END PUBLIC KEY-----\n",
    "result": true
}
salt/auth	{
    "_stamp": "2026-07-21T09:40:08.204551",
    "act": "pend",
    "id": "backup01",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8A...\n-----END PUBLIC KEY-----\n",
    "result": true
}

An act of pend means an unaccepted key has asked to join, and accept or reject events follow when you act on it. Coming from a name you provisioned, that is your cue to go and compare a fingerprint. Coming from backup01, a name nobody on your team created, arriving every ten seconds like clockwork, it is somebody probing your enrollment gate. This is the exact event the reactor lesson wires up to an alert.

One tool sits outside all of this on purpose, and it is your fire escape. salt-ssh needs no minion, no accepted key and no open 4505 or 4506. It needs SSH (Secure Shell, the encrypted remote login you already use) and a roster, which is a plain inventory file listing hosts and how to log into them. That covers the two situations this lesson creates: the master is stopped while you patch it, or an incident has forced you to stop trusting the bus entirely. Give the master its own roster entry and push the same hardening state over SSH. The first run against a host whose SSH host key you have not accepted stops and tells you to rerun with -i.

salt-master:
  host: 10.20.0.5
  user: ops
  sudo: True
  priv: /home/ops/.ssh/id_ed25519

sudo salt-ssh 'salt-master' state.apply master_hardening

salt-master:
----------
          ID: allow_minions_4505
    Function: iptables.append
      Result: True
     Comment: iptables rule for allow_minions_4505 already set (-p tcp -s 10.20.0.0/16 --dport 4505 -j ACCEPT) for ipv4
     Started: 10:07:44.118902
    Duration: 74.311 ms
     Changes:
 
(4 more results)
 
Summary for salt-master
-------------
Succeeded: 5
Failed:    0
-------------
Total states run:     5
Total run time: 312.884 ms

Run that today, while everything is healthy and boring. A recovery path nobody has executed is a guess, and the afternoon you discover the roster carries last year's username is the afternoon the bus is already down. Then write one more line into the runbook: the master.pub fingerprint, kept somewhere the person rebuilding a minion at 2am can read it without asking the master to vouch for itself.

Try this

Work through “Keep The Receipts, And Keep A Way In” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: an exposed master is a fleet-wide root shell, and it has already happened. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related