The event system & reactor

Event-driven automation.

A smoke detector that only beeps is a notification. Wire that same detector to the sprinkler valves and it becomes automation. Most configuration management stops at the beep: a check fails, a dashboard turns red, a human reads it and types a command. Salt ships the plumbing for the second half, and this lesson is about wiring it up without burning the house down.

Every interesting thing that happens in a Salt deployment gets announced. A minion (the small agent Salt installs on each managed machine) connects to the master (the one box that gives the orders). A job finishes. A key is accepted. Your deploy script says it is done. Each announcement is an event, and an event is shaped like a piece of mail: an address line and a letter. The address line is the tag, a slash-separated string such as salt/minion/web3/start that says what kind of thing happened. The letter is the payload, which Salt calls data, a JSON dictionary (JavaScript Object Notation, plain text holding nested key/value pairs) carrying the details.

Both parts travel on the event bus. Think of the public-address system in a warehouse: one person speaks into the microphone, everyone in earshot hears it, and the speaker never needs to know who is listening. That arrangement is called publish/subscribe, and on a Salt master it is a local socket that every master-side process can read from. The reactor is one of those listeners, and it comes with standing orders taped to the wall. The orders read: when an announcement matching this description comes over the speaker, run that. Nothing cleverer than that. All the care goes into what you match and what you run.

Watch the bus before you automate anything

You cannot react to events you have never seen, so the first skill is tailing the bus and reading what actually goes past. If you no longer have the master and minions from the earlier lessons, the official bootstrap script stands up a master plus a local minion on a scratch VM in about a minute. -M adds the master to the default minion install, and -A writes the master's address into that minion's config. Pin the version so your lab matches this lesson. Then prove the transport works before you wire anything on top of it, because reactor debugging is miserable when the real problem is the connection underneath.

# fresh lab? one script installs a master plus a local minion
# (bootstrap prints a few hundred INFO lines; trimmed below)
curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh -o bootstrap-salt.sh
sudo sh bootstrap-salt.sh -M -A 127.0.0.1 stable 3007
 
# which minions is this master actually willing to talk to?
sudo salt-key -L
 
# never debug a reactor over a broken transport: prove reachability first
sudo salt '*' test.ping

 *  INFO: Running install_debian_restart_daemons()
 *  INFO: Running daemons_running()
 *  INFO: Salt installed!
 
Accepted Keys:
db1
saltmaster
web1
web2
web3
Denied Keys:
Unaccepted Keys:
Rejected Keys:
 
web1:
    True
web2:
    True
web3:
    True
db1:
    True
saltmaster:
    True

salt-run state.event is the tool you will keep open in a second terminal for the rest of your Salt career. It prints every tag and every payload as they land, which is how you find the exact tag to match instead of guessing at one. pretty=True indents the JSON and sorts the keys so a human can read it. Start it, then restart a minion or kick off a job, and watch what turns up.

# tail the live event bus; leave this running in a second terminal
# (the minion public key and the state return below are trimmed for width)
sudo salt-run state.event pretty=True

salt/auth	{
    "_stamp": "2026-07-14T09:12:41.482715",
    "act": "accept",
    "id": "web3",
    "pub": "-----BEGIN PUBLIC KEY-----\nMIIBIjANBgkqhkiG9w0BAQ...",
    "result": true
}
salt/minion/web3/start	{
    "_stamp": "2026-07-14T09:12:44.103502",
    "cmd": "_minion_event",
    "data": "Minion web3 started at Tue Jul 14 09:12:43 2026",
    "id": "web3",
    "pretag": null,
    "tag": "salt/minion/web3/start"
}
salt/job/20260714091530123456/ret/web1	{
    "_stamp": "2026-07-14T09:15:31.220981",
    "cmd": "_return",
    "fun": "state.apply",
    "fun_args": [
        "nginx"
    ],
    "id": "web1",
    "jid": "20260714091530123456",
    "out": "highstate",
    "retcode": 0,
    "return": {
        "pkg_|-nginx_|-nginx_|-installed": "(state result trimmed)"
    },
    "success": true
}

Three events, three stories. salt/auth fired when web3 authenticated, and its act field says what the master did with that minion's key: accept when the key is already trusted, pend when it is sitting in the queue waiting for a human, reject when you have turned it away. This one fires on every authentication, including the routine re-auth an established minion performs, so it is far noisier than it first looks. salt/minion/web3/start landed three seconds later, when the minion process finished starting and connected. The third is a job return, and its tag carries the JID (job ID), a timestamp accurate to the microsecond that Salt uses to tie one published command to all the answers it gets back. Notice _stamp: the master writes that itself at publish time, so a sender cannot set it or forge it.

Two arguments turn that firehose into something usable. tagmatch= takes a glob (wildcard matching, the same * you already use on filenames) and drops everything that does not match. count=1 prints one matching event and then exits, which is exactly what you want inside a test script that would otherwise hang forever.

Fire your own events

The bus is not read-only. event.send lets a minion publish an event of your own design, and that is the hook connecting the rest of your world to Salt. The last line of a deploy pipeline. A small shim that catches a monitoring webhook. A cron job that spots drift. Anything that can run salt-call can raise its hand. Give your tags a namespace of your own, the way a company stamps its name on the front of a filing cabinet, so they can never collide with Salt's.

# on web1: the final line of a deploy script
sudo salt-call event.send 'myco/deploy/done' '{"app": "shop", "version": "2.4.1"}'

local:
    True

# back on the master: catch only your namespace, print one event, exit
sudo salt-run state.event tagmatch='myco/*' count=1 pretty=True

myco/deploy/done	{
    "_stamp": "2026-07-14T09:18:02.771349",
    "cmd": "_minion_event",
    "data": {
        "__pub_fun": "event.send",
        "__pub_jid": "20260714091802112233",
        "__pub_pid": 14503,
        "__pub_tgt": "salt-call",
        "app": "shop",
        "version": "2.4.1"
    },
    "id": "web1",
    "pretag": null,
    "tag": "myco/deploy/done"
}

Read that shape carefully, because getting it wrong is the number one beginner bug. When a minion sends the event, your fields sit one level down under data['data'], alongside the __pub_* bookkeeping keys that salt-call staples on. The sender's identity lives at data['id'], and that one field is trustworthy in a narrow but useful sense: before the master republishes anything, it checks the token in the message against the public key it holds for that minion, so id really is the machine that spoke. The tag is a different story. An accepted minion chooses its own tag and the master passes it through unchanged, so a tag on its own proves nothing about who sent it. And everything inside data['data'] is whatever the sender felt like typing.

Beacons are the other half of this idea, and they are the smoke detector from the top of the page. A beacon is a small watcher that runs on the minion and fires an event when something local changes: a config file edited, a service that died, a disk crossing ninety percent. You configure one once and get a stream of events with no glue code of your own.

Wire a reactor: tag to SLS

The reactor lives on the master. A reactor: block in the master config maps tag patterns to reactor SLS files (SLS is short for SaLt State, a plain YAML file, YAML being a readable text format for nested lists and key/value pairs). Patterns are glob-matched, so salt/minion/*/start catches every minion in the fleet, and a single tag can point at several files. Drop it in master.d, the include directory the master reads on startup, rather than editing the main config.

# a list of single-key maps: tag glob -> the SLS files that answer it
reactor:
  - 'salt/minion/*/start':
    - /srv/reactor/onboard.sls
  - 'myco/deploy/done':
    - /srv/reactor/deploy-verify.sls

A reactor SLS file looks like a state file and runs through Jinja (a templating language that fills values into the file before anything else parses it) the same way. Two extra variables get handed to the template: tag, the tag that matched, and data, the event payload. The file declares actions rather than states, and there are four kinds. local. runs an execution module on minions, the same thing that happens when you type salt at the master. runner. calls a runner, a module that executes on the master itself. wheel. reaches the master's own key and config functions, the machinery behind salt-key. caller. runs an execution module on the minion, and it applies only when the reactor is configured minion-side, where it is the one type available.

# re-rendered from scratch on every matching event; `tag` and `data` are injected
{% if data['id'].startswith('web') %}
onboard_new_web_minion:
  local.state.apply:
    - tgt: {{ data['id'] }}
    - args:
      - mods: webserver
{% endif %}

The Jinja guard means only minions whose id starts with web get touched. For anything else the file renders to nothing at all and no action is queued. That render-time branching is the idiom that stops one trigger from becoming a shotgun. Notice also that this fires on every minion restart, not only the first one, so the state you call had better be idempotent (running it twice changes nothing the second time). The args list is the current calling convention, from 2017.7.2 onward. You will still meet arg and kwarg blocks in old formulas; write new reactors with args.

# the mapping lives in master config, so restart the master after editing it
sudo systemctl restart salt-master
 
# then prove the master loaded the triggers you meant to give it
sudo salt-run reactor.list

|_
  ----------
  salt/minion/*/start:
      - /srv/reactor/onboard.sls
|_
  ----------
  myco/deploy/done:
      - /srv/reactor/deploy-verify.sls

That runner only answers when the reactor system is actually running, so an error from it is its own useful signal. While you iterate, salt-run reactor.add 'myco/deploy/done' reactors='/srv/reactor/deploy-verify.sls' and salt-run reactor.delete change the live mapping with no restart, and they vanish on the next restart because they never touch the config file. The SLS files behave differently: they are re-read and re-rendered from scratch on every matching event, so an edit takes effect on the very next one.

Now prove the reaction happened instead of assuming it did. A reaction that runs an execution module publishes an ordinary Salt job, so the evidence turns up on the same bus you were already watching.

# on the master: wait for the next published job, then exit
sudo salt-run state.event tagmatch='salt/job/*/new' count=1 pretty=True
 
# meanwhile, on web3: make the trigger happen for real
sudo systemctl restart salt-minion

salt/job/20260714093012775431/new	{
    "_stamp": "2026-07-14T09:30:12.776019",
    "arg": [
        {
            "__kwarg__": true,
            "mods": "webserver"
        }
    ],
    "fun": "state.apply",
    "jid": "20260714093012775431",
    "minions": [
        "web3"
    ],
    "missing": [],
    "tgt": "web3",
    "tgt_type": "glob",
    "user": "root"
}

There is the whole reaction, out in the open. The reactor published state.apply webserver at web3 as user: root. On the wire it is indistinguishable from an operator typing that command at the master keyboard, which is the thing to hold in your head before you widen a tag glob.

Keep reactor files thin and hand off to orchestrate

The rules of thumb make sense once you know the machinery. Matching and rendering happen sequentially in a single process, like one clerk working a ticket queue, and the rendered result is handed to a pool of worker threads (reactor_worker_threads, default 10, with the queue capped by reactor_worker_hwm, default 10000, where hwm is short for high water mark). One slow Jinja loop that calls out to a runner delays every event stacked up behind it. The bus itself is fire and forget: events are never stored, never replayed, and gone for good if the master was down when they fired. And a reactor SLS is not a state file, whatever it looks like. By design it has no requisites, no ordering guarantee between actions, no onlyif or unless, and no test=True dry run (the mode that reports what would change without changing it).

All those limits point the same way. Keep the reactor file down to a few lines that pull out the fields you need and hand off to the orchestrate runner, where the real multi-step logic lives in an ordinary state file with requisites, readable output, and a dry run you can actually use. Orchestration is the difference between shouting an instruction across a room and handing someone a numbered recipe.

# thin by design: validate, quote, hand off, stop
{% set payload = data.get('data', {}) %}
{% set version = payload.get('version', '') %}
 
{% if version | regex_match('^([0-9]+[.][0-9]+[.][0-9]+)$') %}
verify_deploy:
  runner.state.orchestrate:
    - args:
      - mods: orch.verify_deploy
      - pillar:
          target: {{ data['id'] | yaml_encode }}
          version: {{ version | yaml_encode }}
{% endif %}

Two details in that file are doing real work. yaml_encode turns a value into a properly escaped YAML scalar, which matters because Jinja renders before YAML parses: an unquoted payload containing a newline can close your reaction and open a second one you never wrote. For a whole structure rather than a single value, the json filter does the same job, as in {{ data['data']|json }}. The parentheses in the regular expression are load-bearing too. Salt's regex_match filter hands back the captured groups, so a pattern with no capture group returns an empty tuple, and Jinja reads an empty tuple as false even when the string matched perfectly.

# a normal orchestration: requisites work, output is readable, test=True works
{% set target = salt['pillar.get']('target') %}
{% set version = salt['pillar.get']('version') %}
 
verify_running_version:
  salt.function:
    - name: cmd.run
    - tgt: {{ target }}
    - arg:
      - curl -fsS http://127.0.0.1:8080/version
 
record_result:
  salt.function:
    - name: file.append
    - tgt: {{ target }}
    - arg:
      - /var/log/myco-deploys.log
      - verified {{ version }}
    - require:
      - salt: verify_running_version

One spelling trap worth pointing at: the reactor takes args, while the salt.function state inside an orchestration takes arg. Different layers, different parameter names, and mixing them up produces a state that runs with no arguments at all rather than an error you can see. Now exercise the orchestration by hand, with pillar (the per-minion data the master hands out) that you type yourself and no event anywhere in sight.

# run the real logic with hand-written pillar, no forged event needed
sudo salt-run state.orchestrate orch.verify_deploy \
  pillar='{"target": "web1", "version": "2.4.1"}'

saltmaster_master:
----------
          ID: verify_running_version
    Function: salt.function
        Name: cmd.run
      Result: True
     Comment: Function ran successfully. Function cmd.run ran on web1.
     Started: 09:41:22.118904
    Duration: 812.44 ms
     Changes:   
              ----------
              web1:
                  2.4.1
----------
          ID: record_result
    Function: salt.function
        Name: file.append
      Result: True
     Comment: Function ran successfully. Function file.append ran on web1.
     Started: 09:41:22.941657
    Duration: 402.117 ms
     Changes:   
              ----------
              web1:
                  Wrote 1 lines to "/var/log/myco-deploys.log"
 
Summary for saltmaster_master
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:   1.215 s

That split fixes debuggability. You test the interesting part with pillar you typed, and the reactor file shrinks until there is nowhere left for a bug to hide. Two more hazards live in the machinery. A reactor that matches job-return tags and answers by running another job generates fresh return events, which is the classic feedback loop; match on specific functions, or better, on custom tags you control. And if you run more than one master, both reactors see the same event and both react, which is why salt-run reactor.is_leader and salt-run reactor.set_leader False exist to keep the standby quiet.

What an attacker does with this

The reactor concentrates risk in two places: what you match, and what you trust. Start with the master itself, because its ports have a hard history. Salt uses 4505 to publish commands and 4506 for returns and file transfer. CVE-2020-11651 (CVE stands for Common Vulnerabilities and Exposures, the public catalogue of known security bugs) was an authentication bypass in the master, and CVE-2020-11652 was a directory traversal in the wheel modules. In May 2020, within days of disclosure, the pair was used at scale against internet-facing masters to take root across entire fleets; Ghost and LineageOS were among the public casualties. CVE-2021-25281 later let unauthenticated salt-api requests reach wheel functions, which is the same key-management surface a wheel. reaction uses. Patch, and keep 4505 and 4506 off the public internet, behind a firewall only your minions can cross.

Know also what publisher_acl does and does not do. It restricts which non-root users on the master may run which commands against which minions, and that is all. It has no say over a minion firing events, and none over the reactor, which runs as root and around it. Three standing rules follow. Never auto-accept keys from a reactor: wiring wheel.key.accept to salt/auth recreates auto_accept: True by hand, which means any host that can reach your master joins the fleet and starts receiving pillar data. Check a fingerprint with salt-key -f <id> and accept by hand, or gate acceptance on signed provisioning data you control. Distrust every field in data, and write reactions that stay safe when the payload is a lie. Match narrow tags, because a glob like salt/job/*/ret/* hands every job on the bus a lever on your root-level automation.

Before you turn any of this on in production, run salt-run reactor.list on the master and read every SLS file it names out loud, as if it were a script somebody left in root's crontab. That is exactly what it is. Next up: salt-ssh and masterless minions, where you give up the bus entirely and get a much smaller attack surface in exchange.

Try this

Work through “What an attacker does with this” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: reactor actions run as root with nobody watching. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related