States & the SLS format

Declarative desired state in YAML.

A thermostat never hands your furnace a script. It does not say: light the pilot, run for nine minutes, shut off. It says twenty-one degrees, and the hardware works out how to get there and how to hold it. Salt states are the thermostat side of Salt. The execution modules from the last lesson are the imperative side: run this command, right now, on these boxes. A state describes the finished result instead. This package installed. This file, with this content and these permissions. This service running and enabled at boot. Salt then converges each minion (the small agent Salt installs on a managed machine) to that description, whatever condition the machine started in.

The unit of declaration is an SLS file (short for SaLt State). It is YAML, a plain-text format for nested lists and key-value data, and on the way in it gets run through Jinja, a template engine that behaves like a mail merge: it fills in the blanks and expands the loops before anyone reads the letter. The default renderer chain is written jinja|yaml and reads left to right. Render with Jinja, then parse the result as YAML. These files live on the master under /srv/salt, the default file_roots (the list of directories the master serves state files from). Every entry has exactly three parts: a unique ID, a state function such as pkg.installed or file.managed or service.running, and that function's arguments. Those functions are idempotent, meaning running one twice changes nothing the second time, because each one measures the machine first and writes only where reality disagrees with the declaration.

A Lab, and the Handshake That Guards It

You cannot learn this from prose, so build a two-box lab. The official salt-bootstrap script detects your distribution and installs the current onedir packages (a self-contained bundle that ships its own Python, so Salt stops fighting whatever Python your distro shipped). This lesson uses the 3007 line throughout. The -M flag installs a master alongside the local minion; without it you get a minion only. The -A flag writes the master's address into the minion config for you. Read the script before you run it, the same way you would read anything else you are about to hand to root.

# On the box that will be the master (master plus a local minion):
curl -fsSL -o bootstrap-salt.sh https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh
less bootstrap-salt.sh
sudo sh bootstrap-salt.sh -M stable 3007
 
# On every machine you want to manage, pointed at the master (same download first):
sudo sh bootstrap-salt.sh -A 10.0.0.5 stable 3007
 
# Back on the master, confirm both daemons came up:
systemctl is-active salt-master salt-minion

active
active

Nothing runs yet. On first start a minion generates an RSA keypair (a matched pair of numbers where anything locked with the public half can only be opened with the private half) and posts the public half to the master, which parks it in a waiting room. Until you accept that key, the minion is a stranger at the door. Accept it the way you accept an SSH host key: read the fingerprint at both ends and compare them by eye. The only thing identifying that machine is a name it chose for itself, so the fingerprint is the whole of your evidence.

# On the master: who has knocked, and what is that key's fingerprint?
sudo salt-key -L
sudo salt-key -f web1

Accepted Keys:
Denied Keys:
Unaccepted Keys:
web1
Rejected Keys:
 
Unaccepted Keys:
web1:  6f:2a:8c:41:d0:7b:33:9e:15:aa:c2:58:04:e6:9b:71:3f:cd:52:88:1a:64:b9:07:de:36:f2:80:4c:a5:19:e3

# On web1 itself: the fingerprint of the key it actually sent
sudo salt-call --local key.finger

local:
    6f:2a:8c:41:d0:7b:33:9e:15:aa:c2:58:04:e6:9b:71:3f:cd:52:88:1a:64:b9:07:de:36:f2:80:4c:a5:19:e3

# Only once those two strings match, back on the master:
sudo salt-key -a web1 -y
salt 'web1' test.version

The following keys are going to be accepted:
Unaccepted Keys:
web1
Key for minion web1 accepted.
 
web1:
    3007.1

One SLS File, Read Line by Line

States live in a tree under /srv/salt. A directory holding an init.sls is addressable by the directory name alone, so /srv/salt/nginx/init.sls is the state called nginx. Every public formula follows that convention (a formula is a state tree somebody else wrote and published, the Salt equivalent of an Ansible role). Here is a complete, runnable one.

{% set workers = grains['num_cpus'] %}
 
nginx_installed:
  pkg.installed:
    - name: nginx
 
nginx_config:
  file.managed:
    - name: /etc/nginx/nginx.conf
    - source: salt://nginx/nginx.conf.jinja
    - template: jinja
    - user: root
    - group: root
    - mode: '0644'
    - context:
        workers: {{ workers }}
    - check_cmd: /usr/sbin/nginx -t -c
    - require:
      - pkg: nginx_installed
    - watch_in:
      - service: nginx_running
 
nginx_running:
  service.running:
    - name: nginx
    - enable: True
    - reload: True

Three declarations. nginx_installed hands the work to the pkg state module, which drives apt on Debian and dnf on RHEL without you writing either. nginx_config manages a file. The source: salt://nginx/nginx.conf.jinja line pulls the template from the master's built-in file server (the salt:// scheme maps onto file_roots), template: jinja renders that template on the minion, and context feeds the render a CPU count read from grains, which are the facts a minion reports about itself. The ID on the left, nginx_config, is the handle other states point at. The - name: argument is what the module actually operates on. Leave name out and Salt uses the ID as the name, which is why you often see terse states whose ID is a file path.

Two details in that file earn their keep. First, mode: '0644' is quoted because YAML reads a bare 0644 as an octal literal and hands Salt the integer 420, and your config lands with permissions nobody asked for. Quote every mode, every time. Second, check_cmd is the seatbelt. Salt renders the template to a temporary path, runs /usr/sbin/nginx -t -c /tmp/__salt.tmp.xxxxxxxx against that copy, and moves the file into place only when the check exits zero. A typo in the template then fails one state instead of taking nginx down on five hundred machines at once.

Requisites Turn a List into a Dependency Graph

A recipe that puts "ice the cake" three lines below "bake the cake" is trusting you to read top to bottom. A recipe that says "ice the cake once it has cooled" has written the real dependency down. Salt gives you both. With no requisites it runs declarations in the order they appear in the file, because the state_auto_order setting stamps a sequence number onto each one as it renders. That is deterministic, and it is fragile. Somebody tidies the file during a refactor, the declarations move, and your deploy order quietly moves with them.

Requisites are the dependency written down. require runs a state only after another one has succeeded. watch does everything require does and then reacts when the watched state reported changes. Reacting means Salt calls that module's mod_watch function, and for service that is a restart, or a graceful reload when the state sets reload: True, as this one does. A state module with no mod_watch treats watch exactly like require, silently. onchanges fires only when the referenced state actually changed something, which is the correct guard for work like rebuilding a cache after its config moved. onfail fires only when the referenced state failed, which is where alerting and rollback hang. Each of these has an _in mirror (require_in, watch_in, onchanges_in, onfail_in) that declares the same edge from the other end, which is what you need when the state you want to point at lives in a formula you do not own.

Read the Compiled Form Before You Run It

Most broken states are broken renders. A Jinja tag that never closed, a tab where YAML wanted spaces, a variable that came back empty on one host and nowhere else. Salt hands you both intermediate stages for free, and --out=yaml makes them readable instead of a wall of dashes. state.show_sls prints the high data, which is the recipe as written: the structure the YAML parsed into once Jinja had finished with it. state.show_low_sls prints the low chunks, which are the line cook's version: one flat dictionary per function call, the exact form a state module receives. Neither command applies anything. Both still render, which matters more than it sounds, and the warning below explains why.

salt 'web1' state.show_low_sls nginx --out=yaml

web1:
- __env__: base
  __id__: nginx_installed
  __sls__: nginx
  fun: installed
  name: nginx
  order: 10000
  state: pkg
- __env__: base
  __id__: nginx_config
  __sls__: nginx
  check_cmd: /usr/sbin/nginx -t -c
  context:
    workers: 4
  fun: managed
  group: root
  mode: '0644'
  name: /etc/nginx/nginx.conf
  order: 10001
  require:
  - pkg: nginx_installed
  source: salt://nginx/nginx.conf.jinja
  state: file
  template: jinja
  user: root
  watch_in:
  - service: nginx_running
- __env__: base
  __id__: nginx_running
  __sls__: nginx
  enable: true
  fun: running
  name: nginx
  order: 10002
  reload: true
  state: service
  watch:
  - file: nginx_config

Three things worth noticing there. {{ workers }} has already resolved to 4 for this particular host, which is your proof that Jinja ran on the minion and not on the master. The order keys were assigned automatically, starting at 10000 and counting up in file order, which is the fallback ordering you get when you write no requisites at all. And the watch_in you wrote on nginx_config now also shows up as a plain watch on nginx_running. The compiler rewrites every _in requisite into an ordinary requisite on the target before anything executes, which is why you can debug them here rather than at runtime. The list itself stays in declaration order. Requisites do not reshuffle it. They get resolved as Salt walks the list and pulls each unmet dependency forward.

Dry Run, Then Apply

test=True is Salt's preview. Every state reports what it would do and writes nothing. The line to read is Result. True means already correct, False means it would fail, and None means it would change something.

salt 'web1' state.apply nginx test=True

web1:
----------
          ID: nginx_installed
    Function: pkg.installed
        Name: nginx
      Result: True
     Comment: All specified packages are already installed
     Started: 14:22:31.482910
    Duration: 812.334 ms
     Changes:
----------
          ID: nginx_config
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: None
     Comment: The file /etc/nginx/nginx.conf is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 14:22:32.295244
    Duration: 41.339 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,4 +1,4 @@
                   user www-data;
                  -worker_processes auto;
                  +worker_processes 4;
                   pid /run/nginx.pid;
                   include /etc/nginx/modules-enabled/*.conf;
----------
          ID: nginx_running
    Function: service.running
        Name: nginx
      Result: None
     Comment: Service is set to be reloaded
     Started: 14:22:32.336583
    Duration: 12.008 ms
     Changes:
 
Summary for web1
------------
Succeeded: 3 (unchanged=2, changed=1)
Failed:    0
------------
Total states run:     3
Total run time: 865.681 ms

Nothing on the box moved. file.managed still computed the diff so you can read it, and service.running reported that the watch would fire. Note unchanged=2 in the summary: that is the count of states whose result came back None, and it only shows up in test mode. Note also that check_cmd never ran, because file.managed returns early under test=True, long before it builds the temporary copy. Now the real thing.

salt 'web1' state.apply nginx

web1:
----------
          ID: nginx_installed
    Function: pkg.installed
        Name: nginx
      Result: True
     Comment: All specified packages are already installed
     Started: 14:26:04.911238
    Duration: 786.221 ms
     Changes:
----------
          ID: nginx_config
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: True
     Comment: File /etc/nginx/nginx.conf updated
     Started: 14:26:05.697811
    Duration: 231.905 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,4 +1,4 @@
                   user www-data;
                  -worker_processes auto;
                  +worker_processes 4;
                   pid /run/nginx.pid;
                   include /etc/nginx/modules-enabled/*.conf;
----------
          ID: nginx_running
    Function: service.running
        Name: nginx
      Result: True
     Comment: Service reloaded
     Started: 14:26:05.929716
    Duration: 142.556 ms
     Changes:
              ----------
              nginx:
                  True
 
Summary for web1
------------
Succeeded: 3 (changed=2)
Failed:    0
------------
Total states run:     3
Total run time:   1.161 s

Two states changed, one was already correct, and the reload happened because the file changed and watch_in wired that edge. Nobody scripted the restart. While you are iterating on a single box, stop publishing jobs through the master and drive the minion directly: sudo salt-call state.apply nginx test=True runs the same pipeline on that host, still pulling files and pillar from the master but skipping the job bus entirely. Add -l debug and a Jinja error gives you the full traceback and the template it died in, instead of one terse line.

What a Failure Looks Like, and What You Gate On

Somebody fat-fingers the template. worker_procesess. Here is the same command against that broken tree, with check_cmd doing its job.

salt 'web1' state.apply nginx

web1:
----------
          ID: nginx_installed
    Function: pkg.installed
        Name: nginx
      Result: True
     Comment: All specified packages are already installed
     Started: 14:31:01.884270
    Duration: 780.114 ms
     Changes:
----------
          ID: nginx_config
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: False
     Comment: check_cmd execution failed
              nginx: [emerg] unknown directive "worker_procesess" in /tmp/__salt.tmp.k2b1x9jz:2
              nginx: configuration file /tmp/__salt.tmp.k2b1x9jz test failed
     Started: 14:31:02.664930
    Duration: 63.221 ms
     Changes:
----------
          ID: nginx_running
    Function: service.running
        Name: nginx
      Result: False
     Comment: One or more requisite failed: nginx.nginx_config
     Changes:
 
Summary for web1
------------
Succeeded: 1
Failed:    2
------------
Total states run:     3
Total run time: 843.335 ms

The live /etc/nginx/nginx.conf was never touched, because the check ran against the temporary copy in /tmp and Salt deleted that copy on the way out. nginx_running never ran at all, and Salt names the requisite that stopped it. That is the outcome you want. Without check_cmd, the broken file lands, the reload fires, and nginx refuses to come back on every host in the batch.

For a pipeline, the summary text is useless. You need an exit code. salt-call --retcode-passthrough state.apply nginx exits non-zero when a state fails or the SLS will not compile, and that is the signal a CI job (continuous integration, the pipeline that builds and tests your changes automatically) can gate on. Pair it with state.show_low_sls on every pull request. A render error caught in two seconds by a command that applies nothing beats the same error caught by a pager.

The Escape Hatch, and What It Costs

Idempotence belongs to the state modules, not to Salt itself, and cmd.run is the proof. It runs an arbitrary command on every single apply unless you guard it. creates skips when a given path already exists. unless and onlyif skip based on a shell test. An onchanges requisite skips unless something upstream genuinely moved. An unguarded cmd.run in an SLS is imperative thinking wearing declarative clothes, and it is the single most useful pattern to grep for when you are reviewing somebody else's state tree.

release_dir:
  file.directory:
    - name: /opt/app/2.4.1
    - makedirs: True
 
extract_release:
  cmd.run:
    - name: tar -xzf /opt/releases/app-2.4.1.tar.gz -C /opt/app/2.4.1
    - creates: /opt/app/2.4.1/bin/app
    - require:
      - file: release_dir
 
rebuild_search_index:
  cmd.run:
    - name: /opt/app/2.4.1/bin/reindex --quiet
    - runas: app
    - onchanges:
      - cmd: extract_release

One more thing about that tree, and it is the part people underrate. The minion runs as root, so everything under /srv/salt runs as root on every machine that receives it. Write access to the state tree is root on the fleet, no exploit required, no CVE needed. Treat /srv/salt the way you treat production credentials: kept in version control, reviewed before merge, tight filesystem permissions on the master, and shell accounts on the master handed out like keys to the safe rather than like wifi passwords. The attacker who lands a commit in your state tree has already won.

The honest limits. YAML plus Jinja stays compact right up until real logic accumulates, then it turns into template soup; the Jinja lesson covers how to hold that line. Every minion converges on its own, so a plain SLS cannot express "drain this host from the load balancer, upgrade it, put it back", because that is a sequence across machines and belongs to the orchestrate runner, which drives a workflow from the master instead. IDs must be unique across the whole highstate (the full set of states the master assigns to one machine) rather than per file, and Salt will tell you so with Detected conflicting IDs, SLS IDs need to be globally unique, naming both files. Keep IDs stable once they exist, too. They are the keys in every job return, so renaming one scrambles the change history for that resource.

Everything so far named the state on the command line, which means the desired shape of your fleet currently lives in your shell history rather than in Git. Put that map somewhere a reviewer can see it. The file that assigns states to machines is called the top file, top.sls, it sits at the root of /srv/salt, and one command applies the whole map. Build that next.

Try this

Work through “The Escape Hatch, and What It Costs” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: auto_accept turns the waiting room into an open door. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related