Grains: minion facts

Static data for targeting and logic.

Every machine in a Salt fleet writes its own name badge. It fills in the operating system, the kernel version, how much memory it has, which network cards it owns, and whether it is running on bare metal or inside a virtual machine. Then it clips the badge on and walks in. The master (the central server that hands out work) reads the badge and routes jobs by what it says. Nobody checks the badge against a passport. That is grains in one paragraph, and the last sentence is the one that decides how you are allowed to use them.

Here is the precise version. The salt-minion daemon (a daemon is a program that runs quietly in the background; this one is the agent Salt installs on every managed host) starts up and runs probe code from salt/grains/core.py. That code reads /etc/os-release, /proc/cpuinfo and /proc/meminfo, shells out to dmidecode and systemd-detect-virt to work out the hardware and the hypervisor underneath it, and walks every network interface. What comes back is a flat dictionary of roughly sixty key/value pairs. One pair is one grain. That dictionary then sits in the daemon's memory for as long as it keeps running.

Grains do two jobs. They target, so you can say "run this on every Debian-family box tagged roles:web" without hardcoding a list of hostnames. And they make states portable, so a single SLS file (SaLt State, Salt's YAML state format) installs apache2 on Ubuntu and httpd on Rocky Linux by branching on grains['os_family']. If you have used Ansible facts or Puppet facts, this is the same idea, and it carries the same catch: the data describing a machine comes from that machine.

Salt pushes that catch further than most people expect. Target by grain and the master does not work out a recipient list and post the job only to the matching minions. It shouts down the corridor. The job goes out to every connected minion with your target expression stapled to it, and each minion decides for itself whether that expression describes it. The match runs on the machine being matched.

A two-minute lab

You need a master and a couple of minions to follow along. The bootstrap script is the fastest route. Pull it from the project's GitHub releases page, which is the URL Salt documents now, and download it to disk rather than piping it straight into a shell: you are about to run several thousand lines as root, and reading them first costs a minute. The script works out your distribution and wires up the right package repository, hosted on packages.broadcom.com since the Salt project moved onto Broadcom-run infrastructure. Nothing talks to anything until you accept each minion's key on the master. Picture a tenant cutting their own front-door key and the landlord deciding whether it goes on the lock. Key acceptance is the real trust decision in Salt, and we come back to it.

# on the master node: -M also installs salt-master, "stable 3007.14" pins the release
curl -fsSL https://github.com/saltstack/salt-bootstrap/releases/latest/download/bootstrap-salt.sh -o bootstrap-salt.sh
less bootstrap-salt.sh          # you are about to run this as root
sudo sh bootstrap-salt.sh -M stable 3007.14
 
# on each minion node: -A writes the master address into
# /etc/salt/minion.d/99-master-address.conf
sudo sh bootstrap-salt.sh -A 10.0.12.1 stable 3007.14
 
# back on the master: who is knocking?
sudo salt-key -L

Accepted Keys:
Denied Keys:
Unaccepted Keys:
db1
web1
Rejected Keys:

sudo salt-key -A -y
sudo salt '*' test.ping

The following keys are going to be accepted:
Unaccepted Keys:
db1
web1
Key for minion db1 accepted.
Key for minion web1 accepted.
web1:
    True
db1:
    True

Reading what a minion says about itself

Four functions cover most of the daily work. grains.items dumps the entire dictionary. grains.item returns only the keys you name. grains.get returns a single value. And grains.ls prints the key names on their own, which helps when you are hunting for the exact spelling of something like osmajorrelease. The dump below is abridged. A stock Ubuntu 24.04 minion reports around sixty keys, and one of them, cpu_flags, is a list of well over a hundred processor feature strings that swamps everything else on screen.

sudo salt 'web1' grains.items

web1:
    ----------
    cpuarch:
        x86_64
    fqdn:
        web1.prod.acme.internal
    id:
        web1
    init:
        systemd
    ip_interfaces:
        ----------
        eth0:
            - 10.0.12.7
        lo:
            - 127.0.0.1
    kernel:
        Linux
    kernelrelease:
        6.8.0-51-generic
    mem_total:
        7937
    num_cpus:
        4
    os:
        Ubuntu
    os_family:
        Debian
    osfinger:
        Ubuntu-24.04
    osrelease:
        24.04
    saltversion:
        3007.14
    virtual:
        kvm

Two details are worth burning in. First, grains.get takes a colon-separated path, so ip_interfaces:eth0 walks down into the nested interface map. Pillar lookups (pillar is Salt's separate store for per-host secret data, compiled on the master and handed only to the minion it belongs to) use exactly the same syntax, so the habit transfers. Second, salt-call --local runs the function on the minion itself against its own local config, with no round trip to the master. That is the right tool when a host's grains look wrong, because you see what the minion believes about itself with nothing from the master's cache mixed in.

# one value; the colon walks into a nested structure
sudo salt 'web1' grains.get 'ip_interfaces:eth0'
 
# ask the minion directly; the master is not involved at all
sudo salt-call --local grains.item os_family osfinger

web1:
    - 10.0.12.7
local:
    ----------
    os_family:
        Debian
    osfinger:
        Ubuntu-24.04

Where a grain value actually comes from

Four sources feed that dictionary, and the minion stacks them like stickers on one line of a form: whatever ends up on top is what you read. Core grains go down first, from the probe code. Then /etc/salt/grains, a plain YAML file (YAML is a whitespace-indented text format for structured data) sitting on the minion. Then any Python grain modules you wrote and synced down from the master's _grains directory. Last, the grains: block inside /etc/salt/minion, the main minion config file. Later sources cover earlier ones on a name clash, so core grains always lose. If your os grain is lying to you, someone put it in one of the last three places.

The tail of that order has a wrinkle, because the published precedence table and the shipping code disagree. Salt's documentation lists custom grain modules last, as the winner. The loader itself (the grains() function in salt/loader/__init__.py) merges the config file's grains: block after every grain module has finished running, which makes the config block the winner instead. Do not build anything that leans on which one takes it. Set a given grain name in exactly one place and the argument never comes up.

Two commands keep this honest, and they are not interchangeable. saltutil.sync_grains copies new or changed Python grain modules from the master down to the minion. saltutil.refresh_grains re-runs the whole load without touching any code, which is what you want after hand-editing a file behind the daemon's back; it rereads /etc/salt/minion from disk, so a config edit does get picked up. Watch the keyword names, because they differ between the two: sync_grains takes refresh=False to skip the pillar recompile that otherwise follows it, while refresh_grains takes refresh_pillar=False. Both refresh pillar by default, which on a fleet of a few thousand is a pile of master work you never asked for.

Tagging your own taxonomy

Core grains describe hardware and operating system. Real targeting runs on your vocabulary instead: role, environment, datacenter, owning team. grains.setval writes that vocabulary into /etc/salt/grains and it takes effect immediately, with no restart. Salt parses the argument as YAML on its way in, so '[web, php]' arrives as a real list rather than the literal string "[web, php]".

sudo salt 'web1' grains.setval roles '[web, php]'
sudo salt 'web1' cmd.run 'cat /etc/salt/grains'

web1:
    ----------
    roles:
        - web
        - php
web1:
    roles:
    - web
    - php

grains.append adds one entry to a list grain without rewriting the whole thing. Taking a grain back off is where people get bitten, so watch what the next two commands really do.

sudo salt 'web1' grains.append roles cache
 
# now remove the grain... or so you would think
sudo salt 'web1' grains.delval roles

web1:
    ----------
    roles:
        - web
        - php
        - cache
web1:
    ----------
    changes:
        ----------
    comment:
        The key 'roles' exists but is a dict or a list. Use 'force=True' to overwrite.
    result:
        False

That refusal is deliberate. Under the hood delval hands off to grains.set, which will not clobber an existing list or dictionary unless you say force=True. It is a guard rail, and it is also the reason people think a grain is gone when it is not. Note that destructive=True on its own does not get you past it; the type check runs first.

# delkey is delval with destructive already set; force gets you past the type check
sudo salt 'web1' grains.delkey roles force=True
sudo salt 'web1' cmd.run 'cat /etc/salt/grains'

web1:
    ----------
    changes:
        ----------
        roles:
            None
    comment:
    result:
        True
web1:
    {}

When a tag has to be computed rather than typed, write a grain module. Any public function in a file under the master's _grains directory gets called on the minion, and the dictionary it returns is merged in. The function name is yours to pick; the keys in the returned dictionary are what become grains. Ten lines can tag a host with its rack, its cluster, or whether a compliance agent is installed.

import logging
import pathlib
 
log = logging.getLogger(__name__)
 
RACK_FILE = pathlib.Path("/etc/acme/rack")   # dropped here by the provisioner
 
 
def rack_info():
    """Tag the minion with its rack and slot. Never raise: this runs at startup."""
    try:
        raw = RACK_FILE.read_text(encoding="utf-8").strip()
    except OSError as exc:            # missing file, bad permissions, anything
        log.debug("rack grain unavailable: %s", exc)
        return {"rack": "unknown", "rack_slot": "unknown"}
    rack, _, slot = raw.partition("/")
    return {"rack": rack, "rack_slot": slot or "unknown"}

sudo salt 'web1' saltutil.sync_grains
sudo salt 'web1' grains.item rack rack_slot

web1:
    - grains.rack
web1:
    ----------
    rack:
        r14
    rack_slot:
        07

Two rules for grain modules. Keep them fast and keep them quiet, because they run on every grain load including minion startup, and an HTTP call to a service that has gone away will stall the daemon before it ever registers with the master. And remember that grains load before pillar exists, so you cannot read pillar data inside a grain module. On cloud instances you often need no module at all: set metadata_server_grains: True in the minion config (it defaults to False) and Salt pulls the instance metadata service at http://169.254.169.254/latest into grains for you, at the cost of a network call every time grains load.

Targeting and branching

On the command line, -G matches a single grain with shell-style wildcards. -P treats the value as a PCRE (Perl Compatible Regular Expression, the same pattern dialect grep -P speaks). -C builds compound expressions where G@ marks a grain clause and you get and, or and not. Compound targeting is where grains earn their keep, because you can slice a fleet by two or three facts at once without maintaining a list anywhere.

# -G matches one grain with wildcards
sudo salt -G 'roles:web' test.ping
 
# -P treats the grain value as a regular expression
sudo salt -P 'osfinger:Ubuntu-(22|24)\.04' grains.get osfinger
 
# -C compounds them; G@ marks a grain clause
sudo salt -C 'G@os_family:Debian and G@roles:web and not G@virtual:physical' test.ping

web1:
    True
web1:
    Ubuntu-24.04
web1:
    True

The top file spells the same match differently: the target string on its own line with - match: grain underneath, or grain_pcre if you want the regular-expression flavour. That is how a highstate (the run that applies every state assigned to a host) hands out states by role instead of by brittle hostname patterns. This is a perfectly good use of grains, because the worst case is a machine that installs a web server it did not need.

base:
  'roles:web':
    - match: grain
    - webserver
 
  'os_family:Debian':
    - match: grain
    - baseline.debian

Inside a state, the grains dictionary is available to Jinja (the templating language Salt renders SLS files through) before the YAML is ever parsed, which is what lets one file work on two distributions. You can branch with a plain if, but grains.filter_by is the cleaner tool. Hand it a lookup table keyed by grain value, tell it which grain to key on, and give it a default for anything it does not recognise. This is the seed of the map.jinja pattern you meet in the formulas lesson.

{%- set web = salt['grains.filter_by']({
      'Debian': {'pkg': 'apache2', 'svc': 'apache2'},
      'RedHat': {'pkg': 'httpd',   'svc': 'httpd'},
    }, grain='os_family', default='Debian') %}
 
webserver_pkg:
  pkg.installed:
    - name: {{ web.pkg }}
 
webserver_svc:
  service.running:
    - name: {{ web.svc }}
    - enable: True
    - require:
      - pkg: webserver_pkg

sudo salt -G 'roles:web' state.apply webserver

web1:
----------
          ID: webserver_pkg
    Function: pkg.installed
        Name: apache2
      Result: True
     Comment: The following packages were installed/updated: apache2
     Started: 14:02:11.884213
    Duration: 9182.402 ms
     Changes:
              ----------
              apache2:
                  ----------
                  new:
                      2.4.58-1ubuntu8.6
                  old:
----------
          ID: webserver_svc
    Function: service.running
        Name: apache2
      Result: True
     Comment: Service apache2 has been enabled, and is running
     Started: 14:02:21.101443
    Duration: 512.339 ms
     Changes:
              ----------
              apache2:
                  True
 
Summary for web1
------------
Succeeded: 2 (changed=2)
Failed:    0
------------
Total states run:     2
Total run time:   9.695 s

Add test=True to that command and Salt reports what it would change without changing anything, which is how you find out your grain branch picked the wrong package name before it installs the wrong package. The other failure mode is a grain that is absent on some minion. Subscript it directly and the render blows up for that host, and it takes the whole highstate down with it.

# webserver/init.sls contains: {% for r in grains['roles'] %}
# db1 has never had a roles grain set
sudo salt 'db1' state.apply webserver

db1:
    Data failed to compile:
----------
    Rendering SLS 'base:webserver' failed: Jinja variable 'dict object' has no attribute 'roles'
ERROR: Minions returned with non-zero exit code

The fix is a habit rather than a patch: read grains defensively with grains.get('roles', []) so a missing key renders as an empty list instead of an exception. Save the bare grains['os_family'] form for the handful of core grains that genuinely exist on every host you own.

The trust boundary

Now the part that gets people breached. A minion writes its own grains. Anyone with root on a compromised web server can run grains.setval roles prod-db, wait for the next highstate, and receive every state, every managed file, and, if you targeted pillar by grain, every secret you meant for that role. No exploit is needed anywhere in the chain. The attacker is using the feature exactly as designed, because the minion is the authority on what the minion is.

So the rule is short. Grains route work. They never gate secrets. Target pillar by something the minion cannot write: its minion id, which is bound to a public key you accepted by hand, or a nodegroup, which is a named list of minion ids defined in the master config. The difference is the difference between a badge the guest filled in themselves and a guest list the door staff keep on their own clipboard.

# a nodegroup lives here, on the master, in a file no minion can touch
nodegroups:
  prod_db: '[email protected],db2.prod.acme.internal'
 
# key acceptance stays a human decision
auto_accept: False
 
# bind the master to an address only your own network can reach
interface: 10.0.12.1

base:
  # WRONG: root on any minion can write this grain and collect the password
  # 'roles:db':
  #   - match: grain
  #   - mysql.replica
 
  # RIGHT: membership is decided on the master, by minion id, backed by a key you accepted
  'prod_db':
    - match: nodegroup
    - mysql.replica

That trust chain is only as good as the key ceremony at the start of it. Before you accept a new minion, compare the fingerprint the master shows you against the fingerprint printed on the minion's own console, over a channel that is not the Salt transport you are still trying to establish. Salt hashes keys with SHA-256 by default, so you are checking thirty-two byte pairs; read them out loud if that is what it takes.

# on the master, before you accept anything
sudo salt-key -f db1
 
# on db1's own console, out of band
sudo salt-call --local key.finger

Unaccepted Keys:
db1:  ea:0f:8d:4b:1c:77:9a:3e:52:c6:b8:11:d0:6f:a4:29:7b:e3:15:cc:48:92:0a:d7:36:be:f1:5d:83:2c:60:97
local:
    ea:0f:8d:4b:1c:77:9a:3e:52:c6:b8:11:d0:6f:a4:29:7b:e3:15:cc:48:92:0a:d7:36:be:f1:5d:83:2c:60:97

Be honest about detection too. The master caches each minion's reported grains under /var/cache/salt/master/minions/<id>/data.p, readable with salt-run cache.grains 'web1'. That cache is what the master uses to work out which minions should have answered a job, and what external auth rules consult when they match on grains. Comparing it against your own source of truth catches a machine that has started claiming a new role. What it will not catch is the moment of change, because an attacker with root runs salt-call --local grains.setval or edits /etc/salt/grains directly, and neither of those touches the master. File integrity monitoring (a watcher that alerts when a specific file changes on disk) on /etc/salt/grains is the cheap complement.

Freshness, cost, and hosts with no daemon

Grains are deliberately static, and that is their sharpest limitation. The badge is printed once, at the door. Hot-add memory, change an IP address, or edit /etc/salt/grains behind the daemon's back, and targeting keeps seeing the old values until something reloads them. saltutil.refresh_grains does it on demand and returns True. The minion config can do it on a timer, or before every single job.

# static custom grains; the loader merges this block last, so it wins a name clash
grains:
  environment: prod
  datacenter: ams1
 
# re-gather every 60 minutes. 0, the default, means never
grains_refresh_every: 60
 
# 3005 and later: re-gather right before every job. Accurate, and slower every time
grains_refresh_pre_exec: False
 
# the fqdns grain reverse-resolves every local IP at load time, and an address
# with no reverse record burns about five seconds waiting for the socket timeout
enable_fqdns_grains: False

That last option is the one that turns up as "the minion takes forever to start". The fqdns grain (FQDN is a fully qualified domain name, the long form like web1.prod.acme.internal) asks DNS to resolve every address the host holds back into a name. Each address with no reverse record costs roughly five seconds before the lookup gives up. Four addresses on a sulky network and your minion is twenty seconds late to work. Turn it off unless you actually read fqdns somewhere.

Grains are not tied to a resident daemon either. salt-ssh ships a small Python payload (the "thin" bundle, a tarball of Salt's own code) over an ordinary SSH connection, runs it on the far end, and collects grains fresh on every run, so grain-based Jinja inside your states works unchanged on hosts with nothing installed. The target does need a Python 3 interpreter for that payload to run on. The real asymmetry is targeting: salt-ssh picks hosts out of the roster file, and only glob and regular-expression matching on the target name are supported there. You cannot slice a salt-ssh run with -G. Select by name, then read grains once you are on the box.

sudo salt-ssh 'edge1' grains.item os osfinger virtual

edge1:
    ----------
    os:
        Debian
    osfinger:
        Debian-12
    virtual:
        physical

One habit is worth building on top of all this. After any grain change, prove that targeting moved: run salt -G 'roles:web' test.ping and check that the responders are exactly the machines you expected, no more and no fewer. If a host answers that you never put in that group, something wrote a grain you did not authorise, and finding out what is worth the next hour of your day.

Try this

Work through “Freshness, cost, and hosts with no daemon” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: grains.delval usually leaves a hole rather than nothing. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related