Remote execution & targeting

Run commands across the fleet instantly.

A warehouse floor has a loudspeaker system. The supervisor keys the microphone, says "night shift, forklifts to bay four", and every person on the floor hears it. Only the night shift moves. Salt's remote execution works like that. Your control machine (the master) announces one job. Every managed machine runs a small agent program (the minion) that hears every announcement. Only the minions matching your target act on it and send an answer back. Three machines or three thousand, the master does the same amount of work: it speaks once.

The shape of the command never changes: salt '<target>' <module>.<function> [arguments]. Those functions live in execution modules, the batteries-included libraries that ship with Salt: pkg for packages, service for daemons, user for accounts, cmd for raw shell, test for diagnostics. This is the imperative half of Salt (imperative means you name an action and it happens right now), sitting next to the declarative states you will write in the next lesson. On its own it already replaces a folder of brittle SSH (Secure Shell) loops.

# Is every minion alive, authenticated, and executing code?
$ sudo salt '*' test.ping
 
# One fact from the whole fleet, one round trip
$ sudo salt '*' pkg.version openssl

web1:
    True
web2:
    True
web3:
    True
db1:
    True
db2:
    True
 
web1:
    3.0.13-0ubuntu3.4
web2:
    3.0.13-0ubuntu3.4
web3:
    3.0.13-0ubuntu3.4
db1:
    3.0.15-1~deb12u1
db2:
    3.0.15-1~deb12u1

Two things in that output are worth slowing down for. test.ping sends no ICMP packet (ICMP, the Internet Control Message Protocol, is what the ordinary ping command uses). It is a real function call that travels the whole path, so a True proves the minion is connected, its key was accepted, and it can execute code on request. That is a far stronger signal than a network ping. And every reply comes back as structured data keyed by minion id, which is why Salt output is easy to filter, store, diff against yesterday, or hand to another program.

A master and two minions in five minutes

Salt publishes an official bootstrap script from the salt-bootstrap project on GitHub. It works out your distribution, wires up the right package repository (Salt's packages moved to Broadcom infrastructure in 2024, and the script spares you memorising URLs), and installs the release you pin. The -M flag adds the master service to that host. The -A flag points a minion at its master by name or address. Pin the version on purpose: stable 3007 gets you the 3007.x line rather than whatever ships tomorrow. You are about to run a downloaded shell script as root, so check it first. Each release carries a SHA-256 checksum file and a detached signature next to the script.

# On the box that will be the master (installs the master plus a local minion)
$ curl -fsSL -o bootstrap-salt.sh \
    https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh
$ sudo sh bootstrap-salt.sh -M stable 3007
 
# On each managed box: -A points the minion at its master
$ sudo sh bootstrap-salt.sh -A salt.acme.internal stable 3007

 *  INFO: System Information:
 *  INFO:   CPU Arch:      x86_64
 *  INFO:   OS Name:       Linux
 *  INFO:   Distribution:  Ubuntu 24.04
 *  INFO: Installing minion
 *  INFO: Installing master
 *  INFO: Found function install_ubuntu_stable_deps
 *  INFO: Found function config_salt
 *  INFO: Found function install_ubuntu_stable
 *  INFO: Found function install_ubuntu_stable_post
 *  INFO: Found function install_ubuntu_restart_daemons
 *  INFO: Found function daemons_running
 *  INFO: Running install_ubuntu_stable_deps()
 *  INFO: Running config_salt()
 *  INFO: Running install_ubuntu_stable()
 *  INFO: Running install_ubuntu_stable_post()
 *  INFO: Running install_ubuntu_restart_daemons()
 *  INFO: Running daemons_running()
 *  INFO: Salt installed!

Connections only ever flow one way: minion to master, on TCP (Transmission Control Protocol) ports 4505 and 4506. Managed machines open no inbound ports at all, which is a real advantage over SSH-based tooling in locked-down networks and an easy thing to defend in a design review. The bill comes due in one place. Your master is now the single box that can run code as root on everything you own, so it earns the same protection you give a domain controller or a secrets vault.

The key ceremony that decides who joins

A new starter turns up at the badge office. They bring identification, a human checks it against a record, and only then does a badge come out. Salt runs the same ritual with cryptography. Each minion generates its own RSA keypair (RSA is a public-key algorithm: the private half never leaves the minion, the public half goes to the master). Until the master accepts that public key, the minion gets nothing at all: no jobs, no states, no secret data. salt-key is where you perform the ceremony.

# On the master: the new key is queued and inert
$ sudo salt-key -L
 
# On web1 itself: read the fingerprint of the key it actually generated
$ sudo salt-call --local key.finger
 
# Back on the master: compare the two by eye, then accept
$ sudo salt-key -f web1
$ sudo salt-key -a web1

Accepted Keys:
Denied Keys:
Unaccepted Keys:
web1
Rejected Keys:
 
local:
    27:5c:9a:44:8e:17:b0:6f:2d:c3:51:98:0a:7e:e1:3f:62:d0:4b:a9:76:35:c8:1e:90:57:2b:fa:83:6c:d4:19
 
Unaccepted Keys:
web1:  27:5c:9a:44:8e:17:b0:6f:2d:c3:51:98:0a:7e:e1:3f:62:d0:4b:a9:76:35:c8:1e:90:57:2b:fa:83:6c:d4:19
 
The following keys are going to be accepted:
Unaccepted Keys:
web1
Proceed? [n/Y] y
Key for minion web1 accepted.

Compare that fingerprint somewhere other than the network you are worried about: the cloud console, the provisioning log, an out-of-band console session. Modern Salt hashes fingerprints with SHA-256 by default (a hashing algorithm that boils the key down to a fixed 64-character summary), which is why they are so long. The check protects you against an attacker who reaches port 4506 first and offers a key under the hostname you were expecting. Trust runs both directions, too. On the minion, key.finger_master prints the fingerprint of the master public key it was handed, so you can catch an impostor master before the minion starts taking orders from it.

Anatomy of a command

Every call is <module>.<function>, and Salt documents itself from the machine that would run the code. sys.doc reads the docstring out of the module actually loaded on that minion, which matters more than it sounds: "service" means the systemd module on Ubuntu 24.04 and a different module on an older box, and their options differ. Asking the minion beats guessing from a blog post.

$ sudo salt 'web1' sys.doc test.ping

test.ping:
 
    Used to make sure the minion is up and responding. Not an ICMP ping, this
    just sends a command down the pipe and asks for a response.
 
    Returns ``True``.
 
    CLI Example:
 
    .. code-block:: bash
 
        salt '*' test.ping
 

Reach for a purpose-built module before cmd.run. service.restart nginx behaves the same on systemd and OpenRC, hands back a True or False your script can test, and records a readable intent in the job log. cmd.run 'systemctl restart nginx' is an opaque string that happens to work on today's distribution and returns whatever the command printed. Then verify rather than assume: service.status answers the question you actually care about. When you do need a shell, cmd.run_all is the grown-up version, returning exit code, standard output and standard error as separate fields.

$ sudo salt 'web*' service.restart nginx
$ sudo salt 'web*' service.status nginx
$ sudo salt 'web1' cmd.run_all 'nginx -t'

web1:
    True
web2:
    True
web3:
    True
 
web1:
    True
web2:
    True
web3:
    True
 
web1:
    ----------
    pid:
        24601
    retcode:
        0
    stderr:
        nginx: the configuration file /etc/nginx/nginx.conf syntax is ok
        nginx: configuration file /etc/nginx/nginx.conf test is successful
    stdout:

Notice that nginx -t writes its success message to standard error and leaves standard output empty. cmd.run would have handed you an empty string and you would have called it a failure. retcode: 0 is the field that tells the truth. When one machine misbehaves, stop shouting from the master and go stand next to it. salt-call runs the same functions locally on the minion, -l debug prints every step of what it loaded and returned, and --local skips the master completely, reading everything off the minion's own disk. That flag settles a lot of "is it the network or my code" arguments.

# Standing on web1: the same functions, run locally, master not involved
$ sudo salt-call --local grains.item os osrelease

local:
    ----------
    os:
        Ubuntu
    osrelease:
        24.04

Targeting: deciding who answers

With no flag at all, the target is a shell-style glob against the minion id: 'web*' catches web1 and web2, '*' catches everything. Flags open up the rest. -L takes an explicit comma-separated list. -E is a regular expression (PCRE, the Perl-compatible flavour). -S matches an IP address or a whole subnet written in CIDR notation (Classless Inter-Domain Routing, the 10.20.3.0/24 form). -G matches grains, the facts each minion collects about itself. -I matches pillar, the data the master assigns to a minion. -P and -J are the regular-expression versions of those last two. -N names a node group defined once in the master config, and -C composes any of them into one boolean expression using and, or and not.

Grains and pillar deserve to be seen side by side, because they look alike in a command line and behave nothing alike underneath.

# Grains: what the minion says about itself
$ sudo salt 'web1' grains.item os osrelease datacenter ipv4
 
# Pillar: what the master says about that minion
$ sudo salt 'web1' pillar.items

web1:
    ----------
    datacenter:
        us-east
    ipv4:
        - 10.20.3.11
        - 127.0.0.1
    os:
        Ubuntu
    osrelease:
        24.04
 
web1:
    ----------
    role:
        webserver
    tls_key_passphrase:
        keep-me-secret

Precise targeting is the whole game. It is the difference between restarting nginx on the web tier and restarting it on the payment gateway at the same time. Build the habit of reading the guest list before you send the invitation: --preview-target prints the minions an expression matches and issues no command whatsoever. In a compound expression each source gets a letter, so G@ is a grain, I@ is pillar, S@ a subnet, L@ a list, E@ a regular expression on the id, and a bare word is a glob. One caveat worth carrying: the preview is answered by the master from its cached copy of each minion's grains, not by asking the minions live, so a host whose facts changed since it last checked in can still surprise you.

$ sudo salt --preview-target -G 'os:Ubuntu'
$ sudo salt --preview-target -I 'role:webserver'
$ sudo salt --preview-target -S '10.20.3.0/24'
$ sudo salt --preview-target -C 'web* and G@datacenter:us-east and not web3'

- web1
- web2
- web3
 
- web1
- web2
- web3
 
- web1
- web2
 
- web1
- web2

Once an expression earns its keep, name it in the master config instead of retyping it. A node group is a saved compound target, and node groups can reference each other with N@ inside that config file (on the command line you pick one with -N, not with N@). Edit the file, restart salt-master so it rereads the config, and from then on the whole team says -N webservers and means exactly the same set of machines.

nodegroups:
  # pillar half is assigned by the master; grain half is reported by the host
  webservers: 'I@role:webserver and G@datacenter:us-east'
  edge: 'L@lb1,lb2'
  frontline: 'N@webservers or N@edge'      # node groups can nest
 
# systemctl restart salt-master
# then:  sudo salt -N webservers test.ping

Now the question a defender has to ask about every one of those selectors: who gets to decide the answer? Grains are self-reported. A minion sets its own os, its own ipv4, and any custom grains sitting in /etc/salt/grains, so a host an attacker already owns can rewrite that file, claim role: harmless-cache, and quietly walk into (or out of) your grain-targeted jobs. Pillar is different. The master compiles it from the master's own files, one minion at a time, and hands it back over that minion's private request channel instead of announcing it to the room. A minion can neither forge its pillar nor read anyone else's.

Minion ids sit in between. An id is bound to a key you accepted, so a glob or an -L list is exactly as trustworthy as your key acceptance process. Practical rule: target by grain when you want convenience, and gate anything sensitive on pillar or an explicit minion id.

How one job actually travels

Salt's speed and its sharp edges come from the same design. The master does not walk a list of hosts the way an SSH tool does. It publishes the job once onto a message bus built on ZeroMQ (a fast messaging library) on port 4505. Every connected minion receives that message, decrypts it, evaluates the target expression against itself, and drops the job on the floor if it does not match. The ones that match run the function as root and push results back on port 4506, where the command-line tool collects and prints them. Ten minions and ten thousand feel identical at the prompt, and a mistyped target spreads at exactly the same speed.

One detail in that picture has real consequences. Every accepted minion holds the same shared symmetric key for the publish channel, so every accepted minion can decrypt every job the master publishes, including jobs aimed at other machines. Targeting is a filter, not a fence. Two rules follow. Never pass a secret as a command argument, because salt 'db1' cmd.run 'mysql -pHUNTER2' hands that password to the whole fleet and writes it into the job cache. And never treat a target expression as an access-control boundary. Data that must reach one machine and no other belongs in pillar. (The zmq_filtering option narrows who receives a publish for id-based targets, but it is a scaling knob, not a security control.)

# The local user "deploy" may publish only these functions, only at web*
publisher_acl:
  deploy:
    - web*:
      - state.apply
      - service.restart
      - test.*
 
# Blacklists are separate and absolute. Anyone listed under users: is blocked
# from publishing anything at all, and any module listed here is blocked for
# every caller, root included. Do not repeat the allowed user here.
publisher_acl_blacklist:
  modules:
    - cmd.run
    - cmd.script
 
# Without this, a publish made with "sudo salt ..." arrives as root and skips
# publisher_acl entirely. On, the master checks the name behind the sudo_ prefix.
sudo_acl: True
 
# A non-root CLI user also needs read access to the master's socket and key
# directories (see permissive_pki_access) before publisher_acl does anything.

This caution is not theoretical. Two bugs fixed at the very end of April 2020, CVE-2020-11651 (an authentication bypass that leaked the master's root key) and CVE-2020-11652 (a directory traversal in the wheel modules), meant anyone who could reach a salt master could run commands as root on it and on every minion behind it. Roughly six thousand masters were reachable from the public internet. Within days of the patch going public, LineageOS, Ghost, DigiCert and Xen Orchestra had all been hit, mostly with cryptominers. CVE-2021-25281 repeated the pattern in salt-api, which ignored authentication for one of its clients. Patch promptly, and keep 4505, 4506 and salt-api's 8000 on a management network the internet cannot touch. An exposed master exposes every machine it manages.

Silence is not success

A powered-off minion does not argue with you. It says nothing. Salt tries to make that visible: the master compares the returns against the minions it expected to hear from and prints "Minion did not return. [No response]", or "[Not connected]" when it knows the minion is gone. That expectation comes from the master's cached view of the world (accepted keys plus cached grains), so it can be stale or incomplete. The command-line tool waits -t seconds (default 5) and prints what it has, and --summary adds a scoreboard at the end.

$ sudo salt --summary '*' test.ping

db1:
    True
web1:
    True
web2:
    True
web3:
    True
db2:
    Minion did not return. [No response]
 
The minions may not have all finished running and any remaining minions will return upon completion. To look up the return data for this job later, run the following command:
 
salt-run jobs.lookup_jid 20260721142238105412
 
-------------------------------------------
Summary
-------------------------------------------
# of minions targeted: 5
# of minions returned: 4
# of minions that did not return: 1
# of minions with errors: 0
-------------------------------------------

When the answer matters, ask a question that cannot be answered by silence. salt-run manage.status pings everything the master knows about and sorts it into up and down. For a job that runs for minutes, do not hold your terminal hostage: publish with --async, take the job id it prints (a JID, the timestamp-shaped identifier Salt stamps on every job), and collect results later. That same job cache is your audit trail. jobs.list_job shows who published what, at what target, with which arguments, and a publish made through sudo is stamped sudo_ plus the login name, so the record survives the fact that everything ran as root. The master keeps that history for keep_jobs_seconds (86400 by default, so one day) unless you point master_job_cache at something more permanent.

$ sudo salt-run manage.status
$ sudo salt --async 'db*' cmd.run 'pg_basebackup -D /backup/base'
$ sudo salt-run jobs.list_job 20260721142905331877

down:
    - db2
up:
    - db1
    - web1
    - web2
    - web3
 
Executed command with job ID: 20260721142905331877
 
----------
Arguments:
    - pg_basebackup -D /backup/base
Function:
    cmd.run
Minions:
    - db1
Result:
    ----------
    db1:
        ----------
        return:
StartTime:
    2026, Jul 21 14:29:05.331877
Target:
    db*
Target-type:
    glob
User:
    sudo_ana
jid:
    20260721142905331877

Two more knobs keep a fast tool from becoming a fast outage. Batch mode (-b) runs the job in waves instead of all at once, so a restart rolls through the tier rather than taking it down together, and --batch-safe-limit flips into batch mode automatically when a target turns out bigger than you meant. Batch pings the target before it starts, so it only ever acts on minions that answered. For machines that cannot run a minion at all (appliances, ancient boxes, anything you are not allowed to install software on) salt-ssh runs the same functions over plain SSH using a roster file, slower and agentless, with its own lesson later.

# Restart in waves of 25% instead of all at once
$ sudo salt -b 25% -N webservers service.restart nginx
 
# No minion on the box? Same function, over plain SSH
$ sudo salt-ssh 'legacy1' test.ping      # host defined in /etc/salt/roster

 
Executing run on ['web1']
 
web1:
    True
 
Executing run on ['web2']
 
web2:
    True
 
Executing run on ['web3']
 
web3:
    True
 
legacy1:
    True

From one-off verbs to declared state

Ad-hoc execution has a ceiling. cmd.run happens once, leaves no record of what you meant, and does nothing to stop the machine drifting the moment someone else logs in. States invert that: you declare what should be true in an SLS file (SaltStack State, written in YAML, a plain-text format that uses indentation instead of brackets, with Jinja templating on top) and Salt makes it so, idempotently (running it twice changes nothing the second time). The bridge between the two worlds is that state.apply is itself an execution function published over the same bus. Add test=True and you get a fleet-wide dry run that reports what would change and touches nothing.

# The master only serves this file. The minion renders the Jinja itself,
# against its own grains, before applying anything.
nginx:
  pkg.installed:
    - name: {{ 'nginx-full' if grains['os_family'] == 'Debian' else 'nginx' }}
 
nginx-service:
  service.running:
    - name: nginx
    - enable: True
    - require:
      - pkg: nginx

$ sudo salt 'web1' state.apply nginx test=True

web1:
----------
          ID: nginx
    Function: pkg.installed
        Name: nginx-full
      Result: None
     Comment: The following packages would be installed/updated: nginx-full
     Started: 09:41:07.318260
    Duration: 412.732 ms
     Changes:   
----------
          ID: nginx-service
    Function: service.running
        Name: nginx
      Result: None
     Comment: Service nginx is set to start
     Started: 09:41:07.731542
    Duration: 38.914 ms
     Changes:   
 
Summary for web1
------------
Succeeded: 2 (unchanged=2)
Failed:    0
------------
Total states run:     2
Total run time: 451.646 ms

Result: None is Salt telling you "I would have done this", and unchanged=2 in the summary counts exactly those would-have states. Nothing was installed and nothing was started. That gives you a three-step routine worth making automatic on every change you publish: --preview-target to see who is in scope, test=True to see what would happen to them, then the real run, with jobs.list_job afterwards as the receipt. Keep one thing straight about that {{ ... }} while you go: unlike tools that template on the control node, Salt ships the raw file and the minion renders it locally, which is why a template can read that host's grains and even call execution modules as it renders. One file, the right package name on every distribution.

Try this

Run sudo salt '*' test.ping on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: auto_accept: True is a door with no lock. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related