Your nginx formula's `map.jinja` sets `conf: /etc/nginx/nginx.conf`, but one site keeps its config under `/opt/nginx`. What does the formula's design expect you to do?

Set `nginx:lookup:conf` in that site's pillar, which the map's `merge=` argument overlays on top of the matched entry. `merge=salt['pillar.get']('nginx:lookup', {})` exists precisely so one site can override one key without touching the formula.

Why should `map.jinja` never branch on a grain to decide something security-relevant, such as which sudoers file or firewall policy a host receives?

Grains are reported by the minion itself, so a compromised host can claim a false grain and steer itself into a more privileged entry of the lookup table. Grains are self-reported facts the master accepts on trust, while pillar is compiled on the master against the minion ID its accepted key is bound to.

`salt 'edge01' jinja.load_map nginx/map.jinja nginx` returns `conf: /opt/nginx/nginx.conf`, while every other minion on the same OS returns `/etc/nginx/nginx.conf`. Your `fileserver_backend` is `gitfs` then `roots`. Most likely cause?

Pillar for edge01 sets `nginx:lookup:conf` and `merge=` overlays it on the matched entry; confirm with `salt 'edge01' pillar.items`. Pillar is the only per-minion input to that map, so `pillar.items` will show the override and what it was targeted at.

Courses SaltFormulas & state organization

Formulas & state organization

Reusable, shareable state trees.

Intermediate12 min · lesson 8 of 12

Copy your nginx states into a second project and you have two copies. By the fifth project you have five slightly different nginx setups and no honest way to say which one is right. A restaurant solves this with a printed recipe card: one card, five cooks, instead of five handwritten versions taped inside five cupboards. A formula is that card. It is a reusable, parameterised state tree (a directory of SLS files, short for SaltStack State, which is YAML with Jinja mixed in; Jinja is a templating language that fills values into text before anything reads it) covering exactly one piece of software, and you depend on it instead of duplicating it. Like a recipe card, it never names your oven. Portions, temperature and the local word for the ingredient all stay off the card.

A formula is a contract with three clauses. First, a conventional directory layout, so any engineer can open it and find the part that installs the package without reading the whole thing. Second, a map.jinja file that pens every operating-system difference into one lookup table. Third, pillar-driven configuration. Pillar is Salt's per-minion data store: the master compiles it and hands each slice only to the machine it was meant for, which is where site-specific values live so they never end up inside the formula's code. The Salt community publishes hundreds of these at github.com/saltstack-formulas. The conventions matter far more than the catalogue does, because they are how you should lay out your own private state tree even if you never install a single third-party formula.

A lab you can render against

Formulas need a master and at least one minion before you can render them, so build something disposable first. The official salt-bootstrap script works out your distribution and wires up the right package repository. The -M flag installs a master alongside the minion on the same virtual machine, and -A 127.0.0.1 tells that minion where its master lives. Check the script's published checksum before running it as root, because feeding an unverified installer to a root shell is the exact habit this lesson spends the rest of its time arguing against. Then Salt insists on one manual step: key acceptance. Every minion generates a keypair the first time it starts, and the master ignores it until you say otherwise. Treat that step the way you treat cutting someone a key to the building, because accepting a key decides which machines receive your rendered states and your pillar data.

terminal

curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh -o bootstrap-salt.sh
curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh.sha256 -o bootstrap-salt.sh.sha256
sha256sum -c bootstrap-salt.sh.sha256
 
sudo sh bootstrap-salt.sh -M -A 127.0.0.1 stable 3007
salt --version

output

bootstrap-salt.sh: OK
 
 *  INFO: sh bootstrap-salt.sh -- Version 2024.11.14
 *  INFO: System Information:
 *  INFO:   CPU:          AuthenticAMD
 *  INFO:   CPU Arch:     x86_64
 *  INFO:   OS Name:      Linux
 *  INFO:   OS Version:   6.8.0-45-generic
 *  INFO:   Distribution: Ubuntu 24.04
 *  INFO: Installing minion
 *  INFO: Installing master
 *  INFO: Running install_ubuntu_stable_deps()
 *  INFO: Running install_ubuntu_stable()
 *  INFO: Running install_ubuntu_stable_post()
 *  INFO: Running install_ubuntu_restart_daemons()
 *  INFO: Salt installed!
 
salt 3007.1 (Chlorine)

terminal

sudo salt-key -L
sudo salt-key -f lab01
sudo salt-key -a lab01 -y
sudo salt '*' test.ping

output

Accepted Keys:
Denied Keys:
Unaccepted Keys:
lab01
Rejected Keys:
 
Unaccepted Keys:
lab01:  3d:96:1b:9f:07:c4:a2:58:6e:11:bd:4f:90:2c:e7:35:8a:64:d0:19:fb:73:5c:2e:81:a7:46:cf:0b:d8:92:14
 
The following keys are going to be accepted:
Unaccepted Keys:
lab01
Key for minion lab01 accepted.
 
lab01:
    True

auto_accept is a lab-only convenience

Setting auto_accept: True in /etc/salt/master enrols any host that can reach the master's request port, 4506, and an enrolled host receives your rendered states plus whatever pillar its minion ID matches, delivered to a minion process that runs as root by default. Accept keys one at a time. On a real fleet, compare salt-key -f web1 on the master against salt-call --local key.finger typed on the box itself before you type -a. And keep both ports off the public internet: 4505 is the publish bus every minion subscribes to, 4506 carries requests, returns and file transfers. CVE-2020-11651 and CVE-2020-11652 together let a stranger run commands on the master with no password and no accepted key, and CVE-2021-25281 did much the same through salt-api by skipping the authentication check on wheel calls. All three were mass-exploited within days of disclosure, and every one of those breaches started with a master port someone outside could reach.

Anatomy of a formula

By convention a formula ships one directory named after the software, and inside it the work splits into small SLS files. install.sls puts the package on disk. config.sls manages files. service.sls keeps the daemon running and reloads it when the config changes. init.sls stitches them together with an include list. The split is not cosmetic. It shrinks your blast radius, meaning how much of a machine a single command is able to disturb: salt 'web*' state.apply nginx.config rewrites configuration without touching package state, which is the difference between a ten-second change and an unplanned package upgrade in the middle of an incident.

nginx-formula/ (repo layout)

nginx-formula/                 # one repo, one piece of software
  FORMULA                      # metadata: name, version, supported os / os_family
  README.rst
  pillar.example               # every tunable this formula reads (its manual)
  nginx/                       # the state tree the master actually serves
    init.sls                   # include list; `state.apply nginx` runs it all
    install.sls                # pkg.installed
    config.sls                 # file.managed + templates
    service.sls                # service.running, watches the config
    map.jinja                  # the per-platform lookup table
    files/
      nginx.conf.jinja         # the template itself

nginx/init.sls + nginx/install.sls

# nginx/init.sls
include:
  - nginx.install
  - nginx.config
  - nginx.service
 
# nginx/install.sls
{% from "nginx/map.jinja" import nginx with context %}
 
nginx-pkg:
  pkg.installed:
    - name: {{ nginx.pkg }}

Two conventions are doing quiet work here. The first is how the master resolves a state name. Ask for nginx and it looks for nginx.sls, then for nginx/init.sls. A directory holding an init.sls behaves the way a Python package behaves when it holds an __init__.py file: the folder itself becomes the thing you can name, so the directory *is* the state. The second convention is that each file includes what it needs. config.sls includes nginx.install, service.sls includes nginx.config, and that is why any one of them can be applied on its own. Drop those includes and state.apply nginx.config comes back with Result: False and the comment The following requisites were not found: require: pkg: nginx-pkg, because the state it points at was never rendered into that run. Then there is pillar.example, where a well-built formula writes down every pillar key it reads. That single file is the formula's manual, and it is the first thing to read before you adopt anything.

map.jinja: one substitution table for every platform

International cookbooks carry a table at the back: aubergine is eggplant, bicarbonate of soda is baking soda. Same dish, different local words. map.jinja is that table. Debian-family systems run nginx as the www-data user; RedHat-family systems run it as nginx. Package names, config paths and service names drift the same way. Rather than sprinkling {% if grains['os_family'] == ... %} through six files, a formula puts every platform difference in one dictionary and has every state read from that dictionary. Grains, in case you have not met them yet, are the facts a minion reports about itself when it starts: operating system, OS family, CPU count, hostname, network addresses.

nginx/map.jinja

{#- one table, every platform difference, nothing site-specific -#}
{%- set nginx = salt['grains.filter_by']({
      'common': {
          'pkg': 'nginx',
          'service': 'nginx',
          'conf': '/etc/nginx/nginx.conf',
          'worker_connections': 1024,
      },
      'Debian': {'user': 'www-data'},
      'RedHat': {'user': 'nginx'},
      'Suse':   {'user': 'nginx'},
    },
    grain='os_family',
    base='common',
    default='Debian',
    merge=salt['pillar.get']('nginx:lookup', {})
) %}

/srv/pillar/nginx.sls

# the site's own data, outside the formula, compiled on the master
nginx:
  lookup:
    worker_connections: 4096

nginx/config.sls + nginx/service.sls

# nginx/config.sls
{% from "nginx/map.jinja" import nginx with context %}
 
include:
  - nginx.install
 
nginx-conf:
  file.managed:
    - name: {{ nginx.conf }}
    - source: salt://nginx/files/nginx.conf.jinja
    - template: jinja
    - mode: '0644'
    - context:
        user: {{ nginx.user }}
        worker_connections: {{ nginx.worker_connections }}
    - require:
      - pkg: nginx-pkg
 
# nginx/service.sls
{% from "nginx/map.jinja" import nginx with context %}
 
include:
  - nginx.config
 
nginx-service:
  service.running:
    - name: {{ nginx.service }}
    - enable: True
    - reload: True
    - watch:
      - file: nginx-conf

grains.filter_by takes four arguments worth knowing by heart. grain= picks which minion fact to match on, and defaults to os_family. base= names an entry that gets merged underneath every match, so shared defaults are written once and the matched entry wins any collision. default= names the entry to fall back on when a minion's grain matches no key at all, which is what stops an unexpected distribution from producing an empty dictionary and dying with a baffling Jinja error three lines later. merge= overlays a dictionary on top of the result, and pointing it at pillar.get('nginx:lookup', {}) is the hinge of the whole design: one site can change one value without forking the formula.

Rendering happens on the minion, once per minion, before a single change is made. Jinja runs first and produces plain text. YAML then parses that text into state data. The same formula therefore turns into different concrete states on different machines, and you can read exactly what one machine would do before it does it.

terminal

sudo salt 'lab01' grains.item os_family
 
# render the map on that minion and show the merged result
sudo salt 'lab01' jinja.load_map nginx/map.jinja nginx

output

lab01:
    ----------
    os_family:
        Debian
 
lab01:
    ----------
    conf:
        /etc/nginx/nginx.conf
    pkg:
        nginx
    service:
        nginx
    user:
        www-data
    worker_connections:
        4096

terminal

# the fully compiled state data this minion would execute, nothing run
sudo salt 'lab01' state.show_sls nginx.config

output

lab01:
    ----------
    nginx-conf:
        ----------
        __env__:
            base
        __sls__:
            nginx.config
        file:
            |_
              ----------
              name:
                  /etc/nginx/nginx.conf
            |_
              ----------
              source:
                  salt://nginx/files/nginx.conf.jinja
            |_
              ----------
              template:
                  jinja
            |_
              ----------
              mode:
                  0644
            |_
              ----------
              context:
                  ----------
                  user:
                      www-data
                  worker_connections:
                      4096
            |_
              ----------
              require:
                  |_
                    ----------
                    pkg:
                        nginx-pkg
            - managed
            |_
              ----------
              order:
                  10000
    nginx-pkg:
        ----------
        __env__:
            base
        __sls__:
            nginx.install
        pkg:
            |_
              ----------
              name:
                  nginx
            - installed
            |_
              ----------
              order:
                  10001

Those two commands answer different questions. jinja.load_map renders the map on that minion and hands back the merged dictionary, which makes it the fastest way to ask where a value came from. Here worker_connections came back as 4096 rather than the 1024 sitting in the common block, so pillar won. state.show_sls goes further and prints the compiled state data, includes and all. Notice nginx-pkg in the output even though you asked for nginx.config. The include pulled it in, which is why the require has something to point at. Each state also picked up an order number counting up from 10000, assigned in the sequence Salt read the declarations. Do not read those numbers as the running order. Requisites outrank them, so the explicit require is what puts the package on disk before the file is written, whichever number happens to be lower.

Never branch on a grain for a security decision

Grains are facts the minion reports about itself, and the master takes them on faith. A compromised host can write whatever it likes into /etc/salt/grains and re-report, so a map.jinja entry keyed on grains['role'] or grains['env'] lets that host steer itself into the more privileged branch of your lookup table: the sudoers file meant for bastion hosts, the firewall policy meant for the trusted subnet. Portability decisions on os_family are fine. Anything that grants access belongs in pillar, which the master compiles against the minion ID that the accepted key is bound to. One catch worth internalising: a pillar top file can itself match on grains, and the moment you write match: grain in /srv/pillar/top.sls you have handed that decision straight back to the minion. Target sensitive pillar by minion ID, or by a compound match anchored on ID.

Serving the tree: file_roots, environments and gitfs

The master hands out files from file_roots, a map of *environments* to directories. base is the default; add dev or prod and minions pick one with saltenv=. That works. The pattern that survives an audit is gitfs, short for the Git fileserver, where the master reads the state tree straight out of a Git repository instead of a local folder. Every change to what converges on your fleet becomes a reviewed commit, and an environment becomes a branch or a tag. The master needs a Python Git library for this: pygit2 is the one to reach for, with GitPython as a fallback. Backends are consulted in the order you list them and the first one holding the file wins, so putting roots after gitfs leaves you a local escape hatch for the day your Git server is down, without letting a stray file dropped in /srv/salt quietly shadow a formula.

/etc/salt/master

fileserver_backend:
  - gitfs
  - roots                       # local escape hatch, consulted second
 
gitfs_provider: pygit2
gitfs_update_interval: 120      # seconds between fetches, per remote
gitfs_ref_types:
  - tag                         # branches never become environments
gitfs_saltenv_whitelist:
  - base                        # and only this name is exposed at all
 
gitfs_remotes:
  - https://git.acme.internal/salt/nginx-formula.git:
    - base: v2.1.0              # the pin: the base env IS this immutable tag
  - https://git.acme.internal/salt/states.git:
    - base: v1.14.3
    - root: states              # serve only this subdirectory of the repo
 
file_roots:
  base:
    - /srv/salt

terminal

sudo systemctl restart salt-master
sudo salt-run fileserver.update
sudo salt-run fileserver.envs backend=gitfs
sudo salt-run fileserver.file_list saltenv=base

output

True
 
- base
 
- FORMULA
- README.rst
- baseline/init.sls
- baseline/sshd.sls
- nginx/config.sls
- nginx/files/nginx.conf.jinja
- nginx/init.sls
- nginx/install.sls
- nginx/map.jinja
- nginx/service.sls
- pillar.example
- top.sls

The per-remote base: v2.1.0 line is the whole game. Without it, your base environment is whatever the default branch happened to say the last time the master fetched, which means anyone who can push to that repository changes what runs as root across your fleet without touching a line of your code. With it, base is a fixed tag, and upgrading becomes a deliberate act: move the pin, read the diff, dry-run, apply. The two settings above it close the side doors. gitfs_ref_types: [tag] stops branches becoming environments at all, and gitfs_saltenv_whitelist limits which environment names exist, so nobody can run saltenv=v1.0.0 and quietly converge a host onto a two-year-old formula carrying a two-year-old TLS configuration. One operational footnote: if a master is killed mid-fetch, gitfs leaves an update lock behind and warns about it on every fetch afterwards. salt-run cache.clear_git_lock gitfs type=update clears it.

A formula is code that runs as root

A minion runs as root unless you have gone out of your way to change that, so every state in a formula executes as root on every machine your top file targets. The top file, /srv/salt/top.sls, is the master's map of which states go to which hosts. Treat a third-party formula the way you treat a package dependency, because that is exactly what it is. Pin each gitfs remote to a tag, read the states before adopting them, and dry-run every pin bump. An unpinned tracking branch means whoever holds push access to that repo, or whoever steals a token that does, silently decides what converges on your fleet, and nothing in your own change history will show it.

How a pinned formula reaches a minion

1Tag pushed in Git

v2.1.0, a reviewed commit

2Master fetches gitfs

salt-run fileserver.update

3Top file targets web*

decides which hosts get it

4Minion downloads the SLS

over port 4506

5Jinja renders on the minion

map.jinja plus that minion's pillar

6States execute as root

or stop here, with test=True

Rendering happens on the minion, so one tag compiles into different concrete states on Debian and RedHat. test=True stops the sequence one step before the end.

Dry-run before you trust someone else's code

test=True is a dress rehearsal with no audience. The minion renders everything (the map, its pillar, the templates), works out what would change, and writes nothing. Result: None marks a pending change. Result: True with an empty Changes block means that state is already where it should be.

terminal

sudo salt 'lab01' state.apply nginx test=True

output

lab01:
----------
          ID: nginx-pkg
    Function: pkg.installed
        Name: nginx
      Result: True
     Comment: All specified packages are already installed
     Started: 09:41:02.113455
    Duration: 812.44 ms
     Changes:
----------
          ID: nginx-conf
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: None
     Comment: The file /etc/nginx/nginx.conf is set to be changed
     Started: 09:41:02.930118
    Duration: 41.318 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -4,7 +4,7 @@
                   error_log /var/log/nginx/error.log;
 
                   events {
                  -    worker_connections 1024;
                  +    worker_connections 4096;
                   }
 
                   http {
----------
          ID: nginx-service
    Function: service.running
        Name: nginx
      Result: None
     Comment: Service is set to be reloaded
     Started: 09:41:02.972901
    Duration: 15.204 ms
     Changes:
 
Summary for lab01
------------
Succeeded: 3 (unchanged=2, changed=1)
Failed:    0
------------
Total states run:     3
Total run time: 868.962 ms

Read that summary slowly, because it counts two different things. unchanged=2 is how many states came back with Result: None, meaning nothing was written: the config file and the service reload behind it. changed=1 is how many states had a populated Changes block, which here is the single diff you can see above. Different columns, different numbers, and they separate again on a real apply. Make the dry run a habit, then make it a gate. --out=json gives you machine-readable output, and failing a pipeline whenever a state reports "result": null on a host you did not expect to change is a cheap, honest guard. That goes double for a formula pin bump, because the code you are re-reading belongs to someone else.

terminal

sudo salt 'lab01' state.apply nginx --state-output=changes --state-verbose=False

output

lab01:
----------
          ID: nginx-conf
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: True
     Comment: File /etc/nginx/nginx.conf updated
     Started: 09:44:19.220317
    Duration: 58.882 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -4,7 +4,7 @@
                   error_log /var/log/nginx/error.log;
 
                   events {
                  -    worker_connections 1024;
                  +    worker_connections 4096;
                   }
 
                   http {
----------
          ID: nginx-service
    Function: service.running
        Name: nginx
      Result: True
     Comment: Service reloaded
     Started: 09:44:19.301155
    Duration: 132.401 ms
     Changes:
              ----------
              nginx:
                  True
 
Summary for lab01
------------
Succeeded: 3 (changed=2)
Failed:    0
------------
Total states run:     3
Total run time: 942.559 ms

Those two flags do different jobs, and you want both. --state-output=changes prints full detail for the states that changed and collapses the rest to one terse line each. --state-verbose=False drops the quiet, already-correct states from the printout altogether, which is why nginx-pkg is missing above. Counting happens before printing, so the summary still says three states ran. That is the view to paste into a change ticket. Two more checks close the loop. pillar.items shows exactly what the map's merge= was handed on that minion, and it is the first thing to run when a value renders wrong. And for boxes that cannot carry an agent (appliances, DMZ hosts, anything where installing a daemon needs its own change ticket) salt-ssh runs the same formula over plain SSH, reading its targets from a roster file, which is a short list of hosts and how to log into each one. No minion installed and no key to accept, though the far end still needs a working SSH server and a Python 3 interpreter.

terminal

sudo salt 'lab01' pillar.items
 
# agentless: same formula over SSH, targets read from /etc/salt/roster
sudo salt-ssh 'edge01' state.apply nginx test=True

output

lab01:
    ----------
    nginx:
        ----------
        lookup:
            ----------
            worker_connections:
                4096
 
edge01:
----------
          ID: nginx-pkg
    Function: pkg.installed
        Name: nginx
      Result: None
     Comment: The following packages would be installed/updated: nginx
     Started: 10:02:41.552310
    Duration: 3241.118 ms
     Changes:
----------
          ID: nginx-conf
    Function: file.managed
        Name: /etc/nginx/nginx.conf
      Result: None
     Comment: The file /etc/nginx/nginx.conf is set to be changed
     Started: 10:02:44.801221
    Duration: 88.204 ms
     Changes:
              ----------
              newfile:
                  /etc/nginx/nginx.conf
----------
          ID: nginx-service
    Function: service.running
        Name: nginx
      Result: None
     Comment: Service is set to be started
     Started: 10:02:44.891533
    Duration: 22.117 ms
     Changes:
 
Summary for edge01
------------
Succeeded: 3 (unchanged=3, changed=1)
Failed:    0
------------
Total states run:     3
Total run time:   3.351 s

Monorepo, formula-per-repo, and what to vendor

Two ways of organising this compete once you pass a handful of formulas. Formula-per-repo, the saltstack-formulas style, gives every formula its own version and its own pin, at the price of many gitfs remotes. Each remote is fetched on its own schedule (gitfs_update_interval, 60 seconds by default), so fifty remotes means fifty fetches and a fileserver that feels sluggish on every state run. A monorepo, meaning one repository holding all your states, gives you a single reviewed tree and a single version, at the price of coarser pinning: you cannot hold nginx at v2.1.0 while moving postgres forward. Most teams land in the middle, with a monorepo for their own states plus a small set of pinned third-party formulas.

Vendor rather than trust. Vendoring means keeping your own copy on your own Git server instead of pointing the master at somebody else's. Community formulas vary wildly in how well they are looked after, and an abandoned formula still runs as root on every host your top file matches. A pin-bump routine that holds up: mirror the formula into your own Git server, move the tag there, run salt-run fileserver.update, then state.apply nginx test=True against one canary minion (a single host, targeted by its ID, that you are willing to break) and read the diff line by line before widening the target to web*. Keep real credentials out of the state repo entirely, because a formula repo tends to be the most widely readable thing your team owns. Encrypted pillar is where those belong. What decides *when* all of this converges is the event bus, which can fire a state.apply the second a key is accepted or a service dies.

Try this

Work through “Monorepo, formula-per-repo, and what to vendor” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: auto_accept is a lab-only convenience. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Jinja & templating in statesThe event system & reactor

What Ansible is & the push modelAnsible Inventory and variable securityAnsible for secure automation What Chef is & the pull modelChef What Puppet is & the modelPuppet

Formulas & state organization

A lab you can render against

Anatomy of a formula

map.jinja: one substitution table for every platform

Serving the tree: file_roots, environments and gitfs

Dry-run before you trust someone else's code

Monorepo, formula-per-repo, and what to vendor

Try this

Takeaway

Related