salt-ssh & masterless

Agentless and standalone modes.

The master/minion model rests on two assumptions: that you can install an agent on every machine, and that the machine will hold a connection open to the master forever. A dispatcher with a radio in every van. Real fleets have vans without radios. A storage appliance whose vendor voids your support contract the moment you add a daemon (a background program that runs all the time). A card-processing host where every resident process has to be defended to an auditor. A build machine that lives for ninety seconds. Salt covers those gaps with two other ways of running the same states, and both of them take the master's network listener out of the picture entirely.

salt-ssh is the courier. It carries a toolbox to the door over an ordinary SSH session (Secure Shell, the encrypted remote-login protocol you already use to get a terminal on a server), does the work, and leaves with the toolbox. Masterless is the remote cabin with its own toolshed: the machine keeps a copy of the state tree on local disk and converges itself, with no master anywhere on the network. Everything you already learned still applies. States, the top file, pillar (Salt's store of per-host data, usually secrets), grains (facts a host reports about itself) and Jinja (the templating language Salt renders your SLS files through, where SLS is short for SaLt State, the YAML files your states live in) all behave identically. What changes is the wire underneath, and where trust lives.

# Two hosts, and neither one has a minion installed
salt-ssh '*' test.ping

web1:
    True
db1:
    True

One Control Node, Zero Agents

salt-ssh ships as part of Salt. Distributions package it separately as salt-ssh, and pip install salt pulls it in too. You install it on exactly one machine, a laptop, a bastion, or a CI runner (CI means continuous integration, the system that builds and tests your code on every commit), and that machine becomes the control node. No salt-master daemon has to run. salt-ssh reads the master configuration file, so file_roots (default /srv/salt) and pillar_roots (default /srv/pillar) are read straight off local disk.

# Control node only. The targets get nothing installed.
sudo apt install salt-ssh          # Salt Project repo, Debian/Ubuntu
 
# ...or keep it out of the system Python entirely:
python3 -m venv ~/.salt && ~/.salt/bin/pip install salt
 
salt-ssh --version

salt-ssh 3007.1

Since 3006 the official packages are onedir builds, meaning a private Python interpreter and Salt bundled together in one directory, published by the Salt Project repositories that Broadcom inherited from VMware. A virtualenv install behaves the same for a personal control node. If you would rather not touch /etc/salt, pass -c ~/.salt-ssh and salt-ssh reads its master config and roster from there. Be careful with that one: -c moves the config, not the state. The generated key still lands in pki_dir and the cache still lands in cachedir, both of which keep their system defaults until you set them inside that config file. Set all three and a non-root runner writes nothing outside its own home directory.

A target needs two things: a reachable sshd (the SSH server daemon listening for logins) and a Python 3 interpreter, version 3.8 or newer for Salt 3007. For raw shell commands, even Python is optional. If a box has neither, the roster can run a bootstrap script before anything else lands, which is the next section.

The Roster Replaces Minion Keys

With no minions there are no minion keys, so salt-key and the whole accept-or-reject dance disappears. In its place sits the roster, an inventory file at /etc/salt/roster written in YAML (a plain-text format for structured data, all indentation and colons, the same one your states use). The master's key database is a guest list on a clipboard at the door: who is allowed to talk to me? The roster is the opposite object, an address book with a door key taped to each page: who am I allowed to talk to, and how do I get in? Trust now points outward from the control node, the way it does with Ansible.

# Connection details. Never credentials.
web1:
  host: 10.0.1.5
  user: deploy
  sudo: True
  priv: /etc/salt/pki/master/ssh/salt-ssh.rsa
  identities_only: True
 
db1:
  host: 10.0.1.9
  port: 2222
  user: deploy
  sudo: True
 
appliance7:
  host: 10.0.9.7
  user: svc-salt
  ssh_pre_flight: /srv/salt-ssh/install-python.sh
  thin_dir: /opt/salt-thin
  timeout: 60

Three of those keys earn their place. identities_only: True stops your SSH agent from offering every key it holds to a box you only half trust, which is one of the quieter ways keys get harvested. ssh_pre_flight points at a script on the control node that salt-ssh copies over and runs on the target before Salt's runtime is deployed, and that is the fix for the appliance with no Python 3. It fires only when the target has no Salt directory yet, unless you force it with --pre-flight. thin_dir moves Salt's working directory off /var/tmp, which matters on hosts where /var/tmp is mounted noexec (a mount option telling the kernel to refuse to run any program stored there).

That file is only the default flat roster. salt-ssh --roster=scan 10.0.1.0/24 test.ping sweeps a subnet and builds the list on the fly. salt-ssh --roster=ansible --roster-file=/etc/ansible/hosts '*' test.ping reads an existing Ansible inventory instead, dynamic inventory scripts included as long as Ansible itself is installed to provide ansible-inventory, so you are not maintaining two lists of the same machines. Targeting still uses globs, with -E for a regular expression and -L for an explicit comma-separated list, matched against roster IDs rather than live minions.

# First contact. salt-ssh generates its own keypair if none exists,
# then pushes the public half after a single password prompt.
salt-ssh --key-deploy --askpass '*' test.version

Password: 
web1:
    3007.1
db1:
    3007.1

That keypair lands at /etc/salt/pki/master/ssh/salt-ssh.rsa, and every later run authenticates with it. You will also meet the same machinery by accident, because salt-ssh offers to deploy its key whenever a host answers with a refusal.

# No --key-deploy this time. salt-ssh still notices and offers.
salt-ssh 'db1' test.ping

Permission denied for host db1, do you want to deploy the salt-ssh key? (password required):
[Y/n] 
Password for deploy@db1: 
db1:
    True

What Actually Crosses The Wire

salt-ssh's trick is the thin bundle, and the honest way to describe it is flat-pack furniture. It is a small tarball holding a stripped-down Salt runtime plus its Python dependencies. On first contact salt-ssh copies it across, unpacks it into a hidden directory under /var/tmp, and runs salt-call inside it using the target's own Python. That directory is named from the user you log in as and six hex characters derived from the control node's fully qualified hostname, so it looks like .deploy_3f2a1c_salt. A minion that exists for sixty seconds and then stops existing.

Plain execution modules such as cmd.run and grains.item run inside that thin runtime on the target. State runs work differently, and the difference is the single most useful thing to understand here. salt-ssh first calls the thin to read the target's grains. Then it compiles the entire state run back on the control node, Jinja and all, using those grains and the pillar it compiled locally. Only the finished result crosses: a second tarball holding the low state (the flattened list of instructions Salt actually executes, saved as lowstate.json), every salt:// file the run references, and the compiled pillar as pillar.json. The thin unpacks that and executes it.

salt-ssh 'web1' state.apply nginx

web1:
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: nginx
     Started: 14:22:03.118422
    Duration: 8471.913 ms
     Changes:   
              ----------
              nginx:
                  ----------
                  new:
                      1.24.0-2ubuntu7.3
                  old:
----------
          ID: /etc/nginx/conf.d/hardening.conf
    Function: file.managed
      Result: True
     Comment: File /etc/nginx/conf.d/hardening.conf updated
     Started: 14:22:11.601233
    Duration: 62.114 ms
     Changes:   
              ----------
              diff:
                  New file
----------
          ID: nginx
    Function: service.running
      Result: True
     Comment: Service restarted
     Started: 14:22:11.712004
    Duration: 1044.221 ms
     Changes:   
              ----------
              nginx:
                  True
 
Summary for web1
------------
Succeeded: 3 (changed=3)
Failed:    0
------------
Total states run:     3
Total run time:   9.578 s

So {{ grains['os'] }} reflects the real machine, because the grains were read from the real machine, while the rendering happened on your control node. Results travel back as JSON (JavaScript Object Notation, a plain-text data format that machines parse without ambiguity) over the same SSH channel, bracketed by a long random-looking sentinel line so login banners and MOTD text (message of the day, the greeting a shell prints at login) cannot corrupt the payload. What salt-ssh cannot absorb is a target whose login shell is not Bourne-compatible, because the first thing it sends is a small shell script. One more sharp edge: the thin tarball is built once and cached on the control node, so after you add a custom grain, module or state, pass -t (--regen-thin) to force a rebuild. Salt's own help text calls that out, and skipping it is the classic "my module works from the master but not over salt-ssh" afternoon.

Proving It Left Nothing Behind

Raw mode (-r) pushes a command straight down the SSH session with no thin involved at all, which makes it the honest way to inspect what a run left on disk. A normal salt-ssh call would deploy a fresh runtime and answer its own question.

# -r runs a plain shell command, so this call deploys nothing itself
salt-ssh -r 'web1' 'ls -d /var/tmp/.*_salt'

web1:
    ----------
    retcode:
        0
    stderr:
    stdout:
        /var/tmp/.deploy_3f2a1c_salt

There it is, cached so later runs skip the copy. On a host you fully own that cache is a feature. On a shared or half-trusted box it is a liability, because the state tarball you shipped contains pillar.json, and pillar.json holds that host's compiled secrets in plaintext. -w (--wipe) removes the deployment when the run finishes.

salt-ssh -w 'web1' state.apply nginx > /dev/null
salt-ssh -r 'web1' 'ls -d /var/tmp/.*_salt'

web1:
    ----------
    retcode:
        2
    stderr:
        ls: cannot access '/var/tmp/.*_salt': No such file or directory
    stdout:

-W (--rand-thin-dir) goes further: it unpacks into a randomly named directory under /tmp rather than the predictable one under /var/tmp, and it turns wiping on for you. Watch the seam between the two. Because -W uses a different path, it cleans up after itself but never touches a /var/tmp directory that an earlier ordinary run cached. Switch a pipeline to -W and the old cache, pillar and all, sits there untouched until something removes it. Two operational facts to file away alongside that. Runs are per-host SSH sessions rather than a broadcast, so salt-ssh '*' across two hundred hosts takes minutes instead of seconds, and --max-procs raises the default of 25 parallel connections. And pillar is compiled on the control node, which means pillar files never need to exist on the target at all.

Masterless: The Node Converges Itself

Masterless goes the other direction: nothing connects in, and nothing connects out. Think of a recipe card taped inside the cupboard door, so the kitchen never has to phone anyone to find out what dinner is. The machine runs salt-call --local, which skips the master lookup entirely and reads states and pillar from its own disk. Setting file_client: local in the minion config makes that the permanent default for every salt-call. This is the mode for machines where the central dependency is itself the risk. Baking images with Packer (HashiCorp's image builder), so states run once at build time and the image ships already converged. cloud-init on first boot. Air-gapped hosts. Runners that must never hold a credential capable of reaching production.

file_client: local
 
file_roots:
  base:
    - /srv/salt
 
pillar_roots:
  base:
    - /srv/pillar

# Jinja renders first, then YAML parses whatever it produced.
/etc/motd:
  file.managed:
    - mode: '0644'
    - contents: |
        {{ grains['id'] }} ({{ grains['os'] }} {{ grains['osrelease'] }})
        Environment: {{ salt['pillar.get']('app_env', 'unset') }}
        Managed by Salt. Local edits are overwritten on the next run.

sudo salt-call --local state.apply motd
sudo salt-call --local pillar.items

local:
----------
          ID: /etc/motd
    Function: file.managed
      Result: True
     Comment: File /etc/motd updated
     Started: 09:14:22.883119
    Duration: 31.882 ms
     Changes:   
              ----------
              diff:
                  --- 
                  +++ 
                  @@ -1 +1,3 @@
                  -
                  +build-runner-01 (Ubuntu 24.04)
                  +Environment: staging
                  +Managed by Salt. Local edits are overwritten on the next run.
 
Summary for local
------------
Succeeded: 1 (changed=1)
Failed:    0
------------
Total states run:     1
Total run time:  31.882 ms
local:
    ----------
    app_env:
        staging

What you give up is everything that needed the bus: the event system, reactors, the mine, peer publishing and cross-node orchestration. Each machine converges alone, so coordinating a rolling restart across twenty masterless nodes becomes your problem again. Check one more thing on any masterless box. If the salt-minion package is installed and its service enabled, that daemon is still out there dialing a master on a timer, which quietly undoes the whole point. Mask it with systemctl mask salt-minion, or do not install the package.

Making Masterless Fail Loudly

salt-call exits 0 even when states fail. Read that twice, because it is exactly how hardening stops being applied without anyone noticing. The image builds, the pipeline goes green, and sshd_config was never touched. --retcode-passthrough makes the process exit code reflect what the run actually did instead of whether the binary managed to start.

sudo salt-call --local --retcode-passthrough state.apply hardening
echo "exit=$?"

local:
----------
          ID: sshd-no-root-login
    Function: file.replace
        Name: /etc/ssh/sshd_config
      Result: False
     Comment: File /etc/ssh/sshd_config not found
     Started: 09:20:11.402118
    Duration: 2.914 ms
     Changes:   
 
Summary for local
------------
Succeeded: 0
Failed:    1
------------
Total states run:     1
Total run time:   2.914 ms
exit=2

Exit code 2 means at least one state failed. In a Packer build that single flag is the difference between shipping a hardened image and shipping a broken one with a tick next to it. Add --state-output=changes to cut the noise down to what moved, --out=json when something downstream has to parse the run, and test=True for a dry run that reports what would change without touching anything.

The state tree usually arrives as a pinned Git clone or is baked into the image. That delivery path is now your configuration supply chain and deserves the scrutiny you would give a master. Masterless can also read gitfs directly, which is Salt's Git-backed file server: install pygit2 or GitPython on the node, set fileserver_backend: [gitfs] and gitfs_remotes in the minion config, point gitfs_base at a tag rather than a branch that keeps moving, and salt-call --local state.apply fetches the tree itself. Salt does not verify signatures on that tag, so the pin buys you repeatability and your repository controls do the rest of the work.

Where The Trust Moves

Rule of thumb. The bus for fleets you own and can agent, where instant fan-out across thousands of nodes is the whole point. salt-ssh for the dozens you cannot touch: appliances, vendor boxes, anything with no agent on it yet. Masterless for machines that should be self-contained: images, edge devices, CI runners. The modes compose, and the most common combination bootstraps a new machine with salt-ssh whose first state installs a real minion, which then joins the bus.

Here is why this sits in the hardening section. Both alternate modes remove the master's listening ports (4505 for publishing commands, 4506 for returns) from your attack surface, and those ports have history. CVE-2020-11651 (a CVE, or Common Vulnerabilities and Exposures identifier, is the public catalogue number for a known security hole) was an authentication bypass in the master's request server. CVE-2020-11652 was a directory traversal sitting right beside it. Together they handed unauthenticated attackers root on every minion of any master reachable from the internet, and Shodan scans at the time counted thousands of reachable masters. CVE-2021-25281 did something comparable through salt-api by failing to honor authentication on the async wheel client. A control node that opens outbound SSH connections and listens on nothing at all cannot be hit that way.

The trust does not disappear, though. It relocates. That auto-generated salt-ssh.rsa is root on every host in your roster, so the control node earns the same care a master would: full-disk encryption, no shared accounts, key access logged, and ideally the whole thing running from CI so there is one audited entry point instead of five laptops carrying god-keys. For masterless, anyone who can write to the repository feeding /srv/salt has remote code execution (their code, running as root, on your machines) across every node that pulls it.

One constant survives all three transports: grains are self-reported by the target, so a host that lies about grains['role'] receives whatever states that role gets. Never gate a secret on a grain, in any mode. And before your next salt-ssh run, check the one file that decides everything. stat -c '%a %U' /etc/salt/pki/master/ssh/salt-ssh.rsa should print 600 root, and that key should exist in exactly one place on earth.

Try this

Run salt-ssh '*' test.ping on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the roster is not a password file. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related