What Salt is & the master/minion model

The message bus and real-time control.

A taxi dispatcher with a phone works one driver at a time: dial, talk, hang up, dial the next. A dispatcher with a radio keys the mic once and every cab in the city hears the same sentence in the same instant. SSH-based tools like Ansible are the phone. Salt, which plenty of people still call SaltStack after the company that built it, is the radio. That one design choice explains nearly everything about how Salt behaves, including the parts that can hurt you.

Salt is open-source software doing two jobs with one set of machinery. Configuration management is the first: you write down the state a machine should be in (these packages installed, this file with these contents, this service running and set to start at boot) and Salt changes whatever does not match. Remote execution is the second: run this function on two hundred machines now, and show me every answer. Salt does both over a connection that is already open before you start typing, which is why it stays quick on a fleet of ten thousand.

One Open Channel Instead of a Thousand Knocks

That connection is a message bus, which behaves like an office intercom rather than a telephone: one speaker, many listeners, one message. Salt's default bus runs on ZeroMQ, a small messaging library that shifts messages between processes and machines with no separate broker service sitting in the middle. Salt also ships a plain TCP transport and a newer WebSocket one, both of which can wrap themselves in TLS (Transport Layer Security, the encryption behind the padlock in your browser). Most people run ZeroMQ and never think about it again.

The salt-master daemon is a family of processes, and two of them bind a TCP port. Port 4505 is the publisher: everything the master says to the fleet leaves through here, once. Port 4506 is the request server, where minions hand back results, pull files from the master's file server and collect their pillar data. Every managed machine runs a salt-minion agent, and the minion opens both connections outward. Nothing dials into a minion, so there is no inbound firewall rule to write on it and no trouble with NAT (network address translation, where a crowd of machines shares one public address and none of them is reachable from outside). The one exception is salt-ssh, at the end of this lesson, and that rides ordinary SSH.

# On the master: which ports are open, and who owns them
sudo ss -lntp '( sport = :4505 or sport = :4506 )'
 
# On a minion: which way does the connection point?
sudo ss -tnp '( dport = :4505 or dport = :4506 )'

State   Recv-Q  Send-Q    Local Address:Port    Peer Address:Port  Process
LISTEN  0       1000            0.0.0.0:4505         0.0.0.0:*      users:(("salt-master",pid=1487,fd=25))
LISTEN  0       1000            0.0.0.0:4506         0.0.0.0:*      users:(("salt-master",pid=1502,fd=31))
 
State   Recv-Q  Send-Q    Local Address:Port    Peer Address:Port  Process
ESTAB   0       0         10.0.0.21:41288       10.0.0.10:4505     users:(("salt-minion",pid=903,fd=19))

Read the peer addresses. The minion holds exactly one long-lived connection, to 4505, and sits there listening. Connections to 4506 come and go: the minion opens one when it has a result to hand back or a file to fetch, then drops it. So the firewall rule is short. Allow 4505 and 4506 inbound to the master from your minion networks only, and leave the minions with no inbound rules at all. One trap on the master itself: the salt command talks to the master daemon over the loopback interface, so a firewall that blocks loopback traffic on those ports breaks your own commands while the minions carry on fine.

A job is never routed to a host. The master publishes it to everyone connected and lets each minion decide whether the job was meant for it. Salt's architecture notes put it flatly: the master always publishes commands to all connected minions, and the minions decide if the command is meant for them by checking themselves against the target. The master separately works out which minions it expects to answer, so it knows when a job is finished, but the match happens on the minion. That is how one publish scales to thousands of hosts. It is also the first thing to hold on to when you start asking who can lie to whom.

Build a Lab You Can Trust

Two virtual machines or containers is enough: one master, one minion. Pin the version on purpose. In July 2026 the long-term support line is 3008, with 3008.0 released on 27 May 2026 and 3008.2 on 1 July. The previous long-term line, 3006, is still getting patch releases (3006.27 shipped the same day as 3008.2), and the short-term 3007 line has gone quiet since 3007.14 in April. Build the lab on 3008 and hold it there. A lab that upgrades itself overnight is a lab whose results you cannot trust.

Salt packages are onedir builds, which means Salt carries its own bundled Python runtime instead of fighting your distribution's Python. That has been standard since 3006. The packages also changed address in 2024, from repo.saltproject.io to packages.broadcom.com, after Broadcom bought VMware, which had bought SaltStack in 2020. The code is still Apache-2.0 open source. Keep the repository URL and the version pin in files you commit, not in your shell history.

# --- Master VM, Debian 12 ---
sudo mkdir -m 0755 -p /etc/apt/keyrings
curl -fsSL https://packages.broadcom.com/artifactory/api/security/keypair/SaltProjectKey/public \
  | gpg --dearmor | sudo tee /etc/apt/keyrings/salt-archive-keyring.pgp > /dev/null
curl -fsSL https://github.com/saltstack/salt-install-guide/releases/latest/download/salt.sources \
  | sudo tee /etc/apt/sources.list.d/salt.sources > /dev/null
 
# Refuse anything outside the 3008 LTS line
printf 'Package: salt-*\nPin: version 3008.*\nPin-Priority: 1001\n' \
  | sudo tee /etc/apt/preferences.d/salt-pin-1001 > /dev/null
 
sudo apt-get update && sudo apt-get install -y salt-master
systemctl is-active salt-master

Get:1 https://packages.broadcom.com/artifactory/saltproject-deb stable InRelease [2,880 B]
Get:2 https://packages.broadcom.com/artifactory/saltproject-deb stable/main amd64 Packages [24.1 kB]
Fetched 27.0 kB in 1s (25.4 kB/s)
Reading package lists... Done
The following NEW packages will be installed:
  salt-common salt-master
Setting up salt-common (3008.2) ...
Setting up salt-master (3008.2) ...
Created symlink /etc/systemd/system/multi-user.target.wants/salt-master.service -> /lib/systemd/system/salt-master.service.
active

X-Repolib-Name: Salt
Types: deb
URIs: https://packages.broadcom.com/artifactory/saltproject-deb/
Suites: stable
Components: main
Architectures: amd64 arm64
Signed-By: /etc/apt/keyrings/salt-archive-keyring.pgp

Two lines in that file are doing security work. Signed-By names exactly one key, and apt will only trust this repository's index if that index carries a signature from that key, so a hostile mirror cannot hand you a counterfeit salt-master. The pin file tells apt to refuse to move you off the 3008 line, and priority 1001 is the number that matters: anything above 1000 lets apt downgrade a package to honour the pin rather than quietly ignoring it.

There is a one-line path as well. salt-bootstrap is a shell script the Salt Project maintains that works out your distribution, wires up the same repository and installs the packages. Its flags earn their keep: -M also installs the master, -N skips the minion on that host, -A points a minion at its master, -i sets the minion id. The last two are not magic, they write plain files. -A drops /etc/salt/minion.d/99-master-address.conf and -i writes /etc/salt/minion_id.

# --- Minion VM ---
curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh \
  -o bootstrap-salt.sh
# read it before you run it: a few thousand lines, executed as root
sudo sh bootstrap-salt.sh -A 10.0.0.10 -i web01 stable 3008
systemctl is-active salt-minion

 *  INFO: sh bootstrap-salt.sh -- Version 2026.07.10
 *  INFO: System Information:
 *  INFO:   CPU:          GenuineIntel
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   6.1.0-23-amd64
 *  INFO:   Distribution: Debian 12
 *  INFO: Installing minion
 *  INFO: Running install_debian_stable()
 *  INFO: Running install_debian_stable_post()
 *  INFO: Running daemons_running()
 *  INFO: Salt installed!
active

Configuration lives in /etc/salt/master and /etc/salt/minion, and both read drop-in files from a matching .d directory. That is where your settings belong, so package upgrades never argue with them. Drop-ins merge in alphabetical order and the last file to set a key wins. A minion strictly needs one setting, master. Set id anyway. Left blank, Salt guesses in a documented order: the cached /etc/salt/minion_id file if one exists, then the fully qualified hostname, then /etc/hostname, then a non-loopback entry in /etc/hosts, then an IP address, and finally the literal string localhost. Guessing is how two production boxes end up both calling themselves localhost.

# Which master to talk to. No inbound ports are opened on this host.
master: 10.0.0.10
 
# Pin the identity. Re-imaging the box must not change who Salt thinks it is.
# An id set here outranks the cached /etc/salt/minion_id file.
id: web01
 
# Refuse the first handshake unless the master's public key matches this.
# Read it on the master with:  salt-key -F master
master_finger: 'a1:44:0f:2d:9c:38:7e:b6:51:ca:03:8d:22:fe:97:40:6b:15:e2:79:c8:34:ab:50:1d:66:f3:82:07:b9:4e:d5'

Trust Before Traffic: The Key Handshake

A contractor turns up at a building's front desk, hands over identification, and waits while somebody who knows the job confirms they were expected. No badge, no access. Salt runs the same procedure. On first start salt-minion generates an RSA keypair, two mathematically linked halves: a private key it keeps and never sends, and a public key it can hand to anybody. The default size is 2048 bits (keysize). The public half goes to the master, which parks it in /etc/salt/pki/master/minions_pre/ and does nothing else with it. No jobs, no files, no pillar. When you accept the key, the master moves it to minions/ and sends that minion the shared AES key (Advanced Encryption Standard, the cipher protecting traffic on the bus), wrapped so only the holder of the matching private key can unwrap it. One AES key covers the whole fleet, which is exactly why deleting a minion's key forces a rotation (rotate_aes_key, on by default). It rotates on a timer too, every 24 hours as shipped (publish_session). A revoked host stops being able to read the bus.

salt-key files every key it has ever seen into four buckets. Accepted keys can work. Unaccepted keys are waiting for a human. Rejected keys are ones you refused on purpose with salt-key -r. Denied keys were refused automatically by the master, which happens when a minion presents a different public key for an id that already has an accepted one: a duplicate id, or a rebuilt machine whose stale key nobody deleted.

Before you accept anything, compare fingerprints out of band. A fingerprint is a short hash of a key, a string you can read down a phone line without reading out the key itself. What the master shows for a pending key must match what the machine itself reports, checked over a channel that is not the one you are busy trying to secure: a console session, your provisioning system, a colleague reading digits aloud.

# On the master
sudo salt-key -L
sudo salt-key -f web01
 
# On the minion, over a separate console session
sudo salt-call --local key.finger
 
# Only once the two strings match, character for character
sudo salt-key -a web01

Accepted Keys:
Denied Keys:
Unaccepted Keys:
web01
Rejected Keys:
 
Unaccepted Keys:
web01:  3f:1c:9a:24:d0:7b:8e:55:6a:c3:11:be:47:92:0d:af:5c:e8:63:70:19:d4:2b:fc:88:a5:31:6e:04:9b:d7:52
 
local:
    3f:1c:9a:24:d0:7b:8e:55:6a:c3:11:be:47:92:0d:af:5c:e8:63:70:19:d4:2b:fc:88:a5:31:6e:04:9b:d7:52
 
The following keys are going to be accepted:
Unaccepted Keys:
web01
Proceed? [n/Y] y
Key for minion web01 accepted.

The minion config above closes the other half of the loop. master_finger makes the minion refuse to finish its first handshake unless the master's public key matches the fingerprint you pinned, and you read that value on the master with salt-key -F master. Without it, a minion booting into a network where something else answers on 4505 will trust the first master it meets.

First Words on the Bus

test.ping is Salt's hello. It sends no ICMP packet, the kind of packet the ordinary ping command uses. It asks each targeted minion to return True, which proves in one round trip that the key is accepted, the bus is up, the minion process is alive and a whole job cycle works end to end. Learn the shape of the command, because it never changes: salt, then a target, then module.function, then any arguments. The '*' is a glob, a wildcard pattern matching every accepted minion id, and you quote it so your shell does not expand it into a list of filenames first.

sudo salt '*' test.ping
sudo salt '*' test.version
sudo salt web01 grains.get os_family
sudo salt web01 pillar.items
sudo salt-run manage.status

db01:
    True
web01:
    True
 
db01:
    3008.2
web01:
    3008.2
 
web01:
    Debian
 
web01:
    ----------
 
down:
up:
    - db01
    - web01

Two data sources turned up there. Grains are facts a minion reports about itself: operating system family, kernel, CPU count, addresses, plus any custom ones you set. They are the name badge the machine writes for itself. Pillar runs the other way, data the master compiles and hands down to named minions, more like a sealed envelope the front desk prepares for one guest. That is where secrets belong. Yours came back empty because pillar is opt-in. Keep the direction straight in your head. Grains travel up from the machine, pillar travels down from the master.

Listening to the Bus Itself

The bus is not something you can only reason about in the abstract. Every job, every authentication attempt and every minion restart crosses it as an event, and you can sit and watch the traffic go by. Open a second terminal on the master, start the event stream, then fire a job from the first.

# terminal 1, on the master
sudo salt-run state.event pretty=True
 
# terminal 2, on the master
sudo salt '*' test.ping

salt/job/20260721141055123456/new	{
    "_stamp": "2026-07-21T14:10:55.130361",
    "arg": [],
    "fun": "test.ping",
    "jid": "20260721141055123456",
    "minions": [
        "db01",
        "web01"
    ],
    "missing": [],
    "tgt": "*",
    "tgt_type": "glob",
    "user": "sudo_ops"
}
salt/job/20260721141055123456/ret/web01	{
    "_stamp": "2026-07-21T14:10:55.256772",
    "cmd": "_return",
    "fun": "test.ping",
    "fun_args": [],
    "id": "web01",
    "jid": "20260721141055123456",
    "retcode": 0,
    "return": true,
    "success": true
}

That is the whole model in a dozen lines. One new event carrying the target expression, the job id and the list of minions the master expects to hear from, then one ret event per minion that answered (db01's has been trimmed here to keep the block short). The user field records who asked, prefixed with sudo_ when the command came through sudo. Authentication events land on the salt/auth tag, minion restarts on tags like salt/minion/web01/start. So this stream is an audit trail: ship it somewhere your minions cannot reach and you have a record of every command run against the fleet and the human behind it. Anything that can read events can also be wired to act on them, which is the reactor system, later in this course. Finished jobs stay queryable with salt-run jobs.list_jobs and salt-run jobs.lookup_jid.

Two Modes, One Tool

Everything so far has been remote execution: imperative, immediate, forgotten as soon as it prints. Shouting an order across a kitchen. The second mode is state management, and that is the recipe card taped to the wall. You write an SLS file (SaLt State, YAML that gets passed through the Jinja templating language before the YAML itself is parsed) describing the end state you want, and Salt finds the gap and closes it.

{%- set pkg = 'nginx' if grains['os_family'] == 'Debian' else 'httpd' %}
 
{{ pkg }}:
  pkg.installed: []
  service.running:
    - enable: True
    - require:
      - pkg: {{ pkg }}

The order of operations is where people trip. Salt renders and compiles states on the minion, not on the master. The minion pulls the SLS text from the master's file server over 4506 and does the work locally, the opposite of Puppet's server-side catalog compile. So that set line runs on web01, reads web01's own grains, picks nginx on a Debian-family host or httpd on a Red Hat one, and writes the name into the text before a single line of YAML is parsed. The require line is load-bearing: it tells Salt the service cannot start until the package state has succeeded, which is how you get ordering without writing a script.

One consequence of that design deserves your attention before you put anything sensitive in a state file. The master's file server hands out everything under /srv/salt to any minion with an accepted key, with no per-minion filtering. Pillar is the part that gets compiled separately for each minion. Secrets go in pillar, never in the state tree.

sudo salt web01 state.apply webserver

web01:
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: nginx
     Started: 14:12:11.482910
    Duration: 8412.003 ms
     Changes:
              ----------
              nginx:
                  ----------
                  new:
                      1.22.1-9
                  old:
----------
          ID: nginx
    Function: service.running
      Result: True
     Comment: Service nginx has been enabled, and is running
     Started: 14:12:19.897342
    Duration: 341.221 ms
     Changes:
              ----------
              nginx:
                  True
 
Summary for web01
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:   8.753 s

Run it a second time and both entries come back Result: True with an empty Changes block and a comment saying the package is already installed. That is idempotence (running it again changes nothing the second time). It is what makes a state safe to put on a schedule, and it is how you check that a change really landed. A clean second run is evidence. A second run that changes things again means something on that box is fighting you.

# Same machinery, run on the minion itself, reporting only what it would do
sudo salt-call state.apply webserver test=True
 
# No agent possible? Same states over plain SSH, driven by a roster file
salt-ssh legacy01 test.ping

local:
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 14:31:07.115330
    Duration: 612.884 ms
     Changes:
----------
          ID: nginx
    Function: service.running
      Result: True
     Comment: The service nginx is already running
     Started: 14:31:07.744118
    Duration: 41.006 ms
     Changes:
 
Summary for local
------------
Succeeded: 2
Failed:    0
------------
Total states run:     2
Total run time: 653.890 ms
 
legacy01:
    True

salt-call runs the same code on the minion itself, with no bus involved. It is the most useful debugging tool Salt has: when a state behaves differently from what the master reports, log in and run salt-call -l debug state.apply webserver to watch it render, fetch and execute. It is also the foundation of masterless setups, where a machine carries its own state tree and never talks to a master at all. Where an agent is impossible, on an appliance or a locked-down legacy host, salt-ssh pushes the same states over ordinary SSH using a roster file (a plain list of hosts and how to reach them) instead of keys on a bus. Same language, SSH speed rather than bus speed.

What It Costs You

Salt earns its place where you want a configuration tool and a live control plane in one system. A single tuned master handles a few thousand minions comfortably, and syndics (masters of masters) tier it further for bigger estates. The bill is real. Every minion is software you now patch, monitor and secure, and one that dies quietly stops being managed without telling anybody. The master is stateful, high-value infrastructure with an ugly CVE history. YAML plus Jinja plus grains plus pillar is a steeper climb than Ansible's flatter playbooks. For a small fleet where agentless matters more than speed, Ansible is the better first tool. Puppet and Chef converge on a timer, with on-demand push bolted on rather than built into the core transport.

Some habits worth forming now. Keep the master config and the whole state tree in git, so the fleet's desired state has a history and a review step. Treat salt-key -a as a change, not a chore: fingerprint verified, ticket referenced, hostname you recognise. And wire salt-run manage.status into monitoring, so a minion that goes quiet raises an alarm instead of drifting in silence.

You have one targeting tool so far, '*', and it is a shotgun. Production work wants a scalpel and a brake. salt -b 10 'web*' state.apply webserver rolls a change across ten minions at a time, so a bad state cannot take out every web server in the same second. Globs, lists, regular expressions, grain matching and compound expressions are the next lesson.

Try this

Run sudo ss -lntp '( sport = :4505 or sport = :4506 )' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the master is the whole fleet. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related