The top file & highstate

Mapping states to minions.

A fire station keeps a binder by the door with one procedure per page. How to work the pump. How to raise the ladder. What to do when a tanker spills something that burns. The binder does nothing on its own. What actually runs the station is the duty roster taped up beside it, the sheet that says who is doing what today. Salt splits the same way. Your SLS files (SaLt State files, YAML documents where indentation does the structuring, each one describing how some part of a machine should be configured) are the binder. The top file, top.sls, is the roster. It sits at the root of each environment in your state tree and maps states onto minions using the same targeting expressions you already type on the command line.

A minion's highstate is its whole shift: every state the top file hands it, compiled and enforced in one run. This is where Salt stops being a fast way to run commands and becomes configuration management. Instead of remembering that web boxes need nginx and Debian boxes need your apt hardening, you write the mapping down once and run state.highstate. Each minion works out what it is supposed to be and converges on it, meaning it keeps changing things until reality matches the description and then changes nothing further. Put that run on a timer and drift repairs itself, so the edit somebody made over SSH (Secure Shell, the usual way a person logs into a server by hand) last Tuesday quietly disappears at the next cycle. The bargain is that one line in one file now decides what hundreds of machines become. That is why half of this lesson is about seeing the blast radius before you cause it.

A One Box Lab And The Trust It Hands Out

Everything below runs on a single throwaway virtual machine playing both roles, master and minion. The official bootstrap script works out your distribution and installs current packages. Keep one production habit even here, because Salt's entire trust model is a single list: the accepted keys on the master. Accepting a key is like cutting a spare key to the building and posting it to whoever asked for one. A minion whose key you accept can pull any file from the master's file server, and it will run whatever the master publishes to it, as root. So compare fingerprints before you accept anything. salt-key -f <minion-id> prints what the master received. salt-call --local key.finger prints what the minion actually holds. If the two strings differ, something is sitting in the middle.

# One throwaway box plays both roles: a master, plus a minion pointed at itself
$ curl -fsSL -o bootstrap-salt.sh \
    https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh
$ sudo sh bootstrap-salt.sh -M -A 127.0.0.1 -i salt-lab stable 3007
 
# The master trusts nothing until you accept a key. Compare the two fingerprints.
$ sudo salt-key -f salt-lab
$ sudo salt-call --local key.finger
$ sudo salt-key -a salt-lab -y
$ sudo salt 'salt-lab' test.ping

Unaccepted Keys:
salt-lab:  a1:5c:9f:03:7d:2e:b8:44:6a:19:cf:80:2b:57:e3:1d:9a:64:f2:0c:71:38:d5:be:47:e0:16:8c:23:fa:95:60
 
local:
    a1:5c:9f:03:7d:2e:b8:44:6a:19:cf:80:2b:57:e3:1d:9a:64:f2:0c:71:38:d5:be:47:e0:16:8c:23:fa:95:60
 
The following keys are going to be accepted:
Unaccepted Keys:
salt-lab
Key for minion salt-lab accepted.
 
salt-lab:
    True

-M installs a master next to the minion. -A 127.0.0.1 writes that address into the minion config so the pair talk over the loopback interface. -i salt-lab fixes the minion ID instead of letting it inherit the hostname, which is why the fingerprint command above knows what to ask for. One more lab habit worth forming now: salt-master binds to 0.0.0.0 out of the box, so even this throwaway box is listening on every interface it has. Put interface: 127.0.0.1 in /etc/salt/master and restart it. Then avoid two shortcuts from day one, a blind salt-key -A (accept every pending key) buried in a provisioning script, and auto_accept: True in the master config. Both mean anything that can reach the master's request port enrols itself into your fleet, and every fleet member is a reader of your entire state tree. Accepting a key is the security decision. Everything after it is detail.

Anatomy Of A Top File

The top file is YAML nested three levels deep. The outermost keys are environments, named branches of your state tree declared by file_roots in the master config. base is the default and the only one you need at first. Under each environment come targets. Under each target sits the list of states to hand out. Dotted names are directory paths, so apt.hardening means the file apt/hardening.sls, or apt/hardening/init.sls if you organised that state as a directory.

One rule catches everybody exactly once. Think of a target as the address written on an envelope, and of the matcher as the rule the sorting office uses to read it. Inside a top file that rule is compound by default, the same syntax as salt -C on the command line, where a bare pattern with no prefix letter is a shell-style glob against minion IDs. Write os_family:Debian as a target and Salt does not read it as a grain (a grain is a fact the minion reports about itself: its operating system family, its processor count, the role it says it plays). Salt reads it as a minion ID pattern, matches nobody, and says nothing about it, because matching zero minions is a perfectly legal outcome. The - match: line below is load-bearing, not decoration. Valid values include glob, pcre, grain, grain_pcre, pillar, pillar_exact, list, ipcidr, nodegroup, data and compound.

base:                              # an environment, from the master's file_roots
  '*':                             # bare pattern = glob on minion IDs = everyone
    - core                         # /srv/salt/core.sls or /srv/salt/core/init.sls
 
  'web*':
    - nginx
    - app
 
  'os_family:Debian':
    - match: grain                 # without this line it is read as a minion-ID glob
    - apt.hardening                # /srv/salt/apt/hardening.sls
 
  'web* and G@os_family:Debian':
    - match: compound              # already the default here; say it for the reader
    - certbot

A minion can match several targets at once, and its assignment is the union of all of them, in the order they appear in the file. If two targets both hand out core, Salt compiles that state once, not twice. Notice how quickly the mental arithmetic gets hard. Four targets and a fleet of mixed operating systems, and you are already guessing. Do not do this arithmetic in your head.

Ask The Minion, Never Guess

state.show_top runs the real top-matching code on the real minion and returns exactly what a highstate would assign it, environment merging and all. It writes nothing. It is safe on production at any hour, and it is the most useful command in this lesson.

$ sudo salt 'web1' state.show_top

web1:
    ----------
    base:
        - core
        - nginx
        - app
        - apt.hardening
        - certbot

Two neighbours are worth learning at the same time. state.show_highstate returns the full compiled state data for everything assigned, which is long but definitive. state.show_lowstate returns the flat list of chunks in the exact order they will execute, which is what you read when you suspect an ordering problem. All three run on the minion, so salt-call state.show_top from the box itself gives the same answer without publishing a job to the fleet. One caveat: salt-call still pulls the top file down from the master, so it is not an offline check. For that you need a masterless setup, --local plus a file_roots on the box itself.

What A Highstate Actually Does

From the shell, state.highstate looks like one command, so the machinery underneath deserves spelling out. The master publishes a job to the targeted minions and then steps back. Almost all the work happens minion-side. Each minion fetches top.sls from the master's file server, checks every target against its own ID and its own grains, and pulls down the SLS files it matched. Rendering happens there too. Jinja (the templating language Salt runs over your files before anything parses them, so a file can say "if this host is Debian, use that package name") executes with that minion's grains and pillar in scope. The result is parsed as YAML into what Salt calls high data. Everything assigned is merged into one structure, requisites such as require, watch and onchanges are resolved into an ordered list of low chunks, and each chunk is handed to a state module. Results come back to the master, which prints the per-minion report.

Two consequences fall out of that picture. The first is ordering. With the default state_auto_order: True, the compiler stamps every state declaration with an incrementing order number starting at 10000, following the sequence you wrote them in. Treat that as a tiebreaker and nothing more. An explicit order: argument beats it, and requisites beat both, because they are resolved after the sort. Numbered pages are not the same as "do this one before that one". Express real dependencies with requisites, never with position in the file.

The second is that rendering is per-minion, so the honest way to debug a template is to render it as one specific host. state.show_sls does exactly that. In the state below the package name is decided by a grain, and the --out=yaml view shows the Jinja already resolved for that host plus the order numbers Salt assigned.

# Jinja runs on the minion, with that minion's grains, before the YAML is parsed
nginx:
  pkg.installed:
    - name: {{ 'nginx-full' if grains['os_family'] == 'Debian' else 'nginx' }}
  service.running:
    - enable: True
    - require:
      - pkg: nginx

$ sudo salt 'web1' state.show_sls nginx --out=yaml

web1:
  nginx:
    __env__: base
    __sls__: nginx
    pkg:
    - name: nginx-full
    - installed
    - order: 10000
    service:
    - enable: true
    - require:
      - pkg: nginx
    - running
    - order: 10001

Read each list as three things stacked: the arguments you wrote, then the function name Salt split off the pkg.installed shorthand, then the order entry the compiler appended. nginx-full proves the grain resolved the way you expected on this host. If it came out wrong, the fault is in your template or in that minion's grains, and you found it without touching a package manager.

Preview, Then Converge

Highstate has a rehearsal built in. test=True does the identical fetch, render and compile, then runs every state in no-change mode. States that would alter something report Result: None with a comment written in the conditional: would be installed, is set to be changed. Nothing is written to disk.

$ sudo salt 'web*' state.highstate test=True

web1:
----------
          ID: nginx
    Function: pkg.installed
        Name: nginx-full
      Result: None
     Comment: The following packages would be installed/updated: nginx-full
     Started: 14:02:11.482913
    Duration: 512.3 ms
     Changes:
----------
          ID: nginx_config
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: None
     Comment: The file /etc/nginx/nginx.conf is set to be changed
     Started: 14:02:12.004871
    Duration: 41.9 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,4 +1,4 @@
                  -worker_processes 1;
                  +worker_processes auto;
...
 
Summary for web1
-------------
Succeeded: 14 (unchanged=3, changed=1)
Failed:     0
-------------
Total states run:     14
Total run time:   1.221 s

That summary line repays close reading, because the two counters overlap rather than dividing the fourteen between them. unchanged=3 counts the states that came back None, so three of the fourteen would change something if you let them. changed=1 counts the states that produced a preview diff, and that one is also inside the three. Run it for real and the same host prints Succeeded: 14 (changed=3). On anything bigger than a lab, add --state-output=changes so clean states collapse into a single line each and only the ones that moved print in full, or --state-verbose=False to drop the clean ones from the output entirely.

$ sudo salt 'web*' state.highstate --state-output=changes

web1:
 Name: chrony - Function: pkg.installed - Result: Clean Started: - 14:07:43.980112 Duration: 6.593 ms
 Name: /etc/motd - Function: file.managed - Result: Clean Started: - 14:07:43.994308 Duration: 4.117 ms
...
----------
          ID: nginx_config
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: True
     Comment: File /etc/nginx/nginx.conf updated
     Started: 14:07:44.113905
    Duration: 88.4 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,4 +1,4 @@
                  -worker_processes 1;
                  +worker_processes auto;
 
Summary for web1
-------------
Succeeded: 14 (changed=3)
Failed:     0
-------------
Total states run:     14
Total run time:   8.667 s

When a highstate fails to render at all, the master-side report hands you a compile error with the useful part swallowed. Run salt-call state.highstate -l debug on the minion itself and you get the template error with a line number. Two more facts about the command. state.apply with no arguments is an exact alias for state.highstate. state.apply nginx applies that one SLS while ignoring the top file, which is handy while you iterate and corrosive as a habit, because minions quietly accumulate configuration that no top file ever assigned, that no highstate will ever repair, and that nobody will think to look for.

A minion also refuses to run two state jobs at the same time. The lock lives on the minion, not the master, and it surprises people the first time a scheduled highstate collides with the one they typed.

$ sudo salt 'web1' state.highstate

web1:
    Data failed to compile:
----------
    The function "state.highstate" is running as PID 3421 and was started at 2026, Jul 21 14:30:02.118344 with jid 20260721143002118344
ERROR: Minions returned with non-zero exit code

The message gives you the PID (process ID) of the run that beat you to it and the jid (job ID, the timestamp Salt uses to name a job), so you can watch that run instead with salt-run jobs.lookup_jid 20260721143002118344. Pass queue=True and the second run waits its turn rather than failing. Resist concurrent=True, which lets both proceed and lets two state modules fight over the same file at the same moment.

Convergence On A Schedule

A highstate you have to remember to run is a highstate that stops running. The minion carries its own scheduler, a kitchen timer on the machine itself, so put the loop there rather than in a cron entry on the master that nobody updates when the fleet grows.

schedule:
  converge:
    function: state.highstate
    minutes: 30
    splay: 300          # random 0-300s offset, so the fleet does not stampede
    maxrunning: 1       # never stack two runs of this job on this minion
 
  drift_report:         # changes nothing, reports everything that has drifted
    function: state.highstate
    kwargs:
      test: True
    hours: 1
    splay: 600
    maxrunning: 1

splay adds a random delay of up to that many seconds so a thousand minions do not hit the file server in the same second. maxrunning: 1 stops runs stacking up when one overruns its window. The second entry is the one security teams care about. An hourly highstate in test mode writes nothing and reports every state whose result is None, which is a rolling drift report for the whole fleet, published on the event bus, for the price of some processor time. Stagger the two entries in the hour, because the state lock you met above covers every state function rather than one job at a time, so a converge run and a drift report that land together mean one of them dies. Schedules written into a config file need a minion restart to take effect. Check what actually landed with salt 'web1' schedule.list, and remember that schedule blocks can live in pillar as well as the minion config, which lets you change convergence cadence centrally instead of editing a thousand files.

Environments Are Not A Security Boundary

Several environments (base, qa, prod) turn into promotion stages: three binders on three shelves, with changes moving up a shelf once somebody has reviewed them. Each maps to its own directory in file_roots, or its own gitfs branch, and production minions pin themselves with saltenv: prod in their config so they read one shelf and nothing else. One surprise to defuse early. For any minion that has not pinned an environment, Salt merges the top files from every environment when working out assignments, which is top_file_merging_strategy: merge, the default. Set it to same and each environment's top file only assigns states inside its own environment, with default_top naming the fallback when a minion's environment has no top file of its own. That option is read wherever the top file gets compiled. For a normal highstate that is the minion, so it belongs in the minion config; for pillar it is the master.

Now the part people get badly wrong. Environments organise the tree. They do not guard it. The file server is an unlocked supply cupboard: any accepted minion that asks for a path gets the file, whatever the top file did or did not assign. Below is web1, a host the top file never gave the vault state to, listing the whole tree and then reading a private key out of it.

$ sudo salt-call cp.list_master
$ sudo salt-call cp.get_file salt://vault/tls/server.key /tmp/proof.key

local:
    - apt/hardening.sls
    - core/init.sls
    - nginx/init.sls
    - nginx/nginx.conf.jinja
    - top.sls
    - vault/init.sls
    - vault/tls/server.key
local:
    /tmp/proof.key

That is the design, not a bug. Minions have to be able to fetch the files their states reference, and the file server performs no per-minion authorisation on the way out. The rule that follows is absolute. No private keys, no passwords, no customer data anywhere in the state tree, in any environment, ever. Anything a state needs and a stranger must not read has to arrive by a different road.

The different road is pillar: a second tree with its own top file, compiled on the master and delivered over the encrypted channel only to the minions its own targeting selected. A notice on the wall versus a sealed envelope with one name on it. A database password, an API token (the credential one program uses to prove itself to another), the single tuning value that differs per host, all of that belongs there and none of it belongs in the world-readable tree you have been building. Before you write a line of pillar, run state.show_top against a production minion and read what your current top file is already handing out.

Try this

Run sudo sh bootstrap-salt.sh -M -A 127.0.0.1 -i salt-lab stable 3007 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: one line under '*' converges everywhere, and grains are self-reported. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related