Running & reading results

Passed, failed, skipped.

A restaurant health inspector does not leave one sticker on the door that says "fine." They leave a sheet with a line per item. The walk-in cooler held temperature. The hand-wash sink had no soap. The basement prep room was padlocked, so nobody looked inside. Three outcomes, not two. InSpec reports the same way, and the mistake almost everyone makes on their first real scan is reading the padlocked-room line as though it said clean.

Writing the controls themselves is a separate lesson. This one is about running them and reading what comes back, in two languages at once: the coloured marks a human skims, and the exit code (the single small number a program hands back to whatever started it) that your build server actually acts on. Everything below is Chef InSpec 6 at the command line. Every flag shown behaves the same way on 5.18 and later.

Read the Scoreboard Before the Details

Start with a run against a real machine. -t is short for --target and points InSpec at a host over SSH (Secure Shell, an encrypted remote login). --sudo runs the checks through sudo (superuser do, the normal way to borrow root's powers for one command), which is what lets a test read a root-only file such as /etc/shadow without you logging in as root.

inspec exec ./linux-baseline -t ssh://[email protected] --sudo

Profile:   Linux Baseline (linux-baseline)
Version:   1.4.0
Target:    ssh://[email protected]:22
Target ID: 6f0d1c4e-5c9a-4b7e-9c3a-1f2b8d0a77e1
 
  ✔  sshd-01: SSH daemon configuration
     ✔  SSHD Configuration PermitRootLogin is expected to cmp == "no"
     ✔  SSHD Configuration PasswordAuthentication is expected to cmp == "no"
  ✔  sshd-04: SSH ciphers and key exchange
     ✔  SSHD Configuration Ciphers is expected to cmp == "[email protected],aes256-ctr"
  ×  os-05: No empty password fields (1 failed)
     ✔  File /etc/shadow is expected to be owned by "root"
     ×  File /etc/shadow content is expected not to match /^[^:]+::/
     expected "root:$y$j9T$Kk1x9Yv0Qm0Zt2Rr4B:20140:0:99999:7:::\ndaemon:*:20140:0:99999:7:::\nsvc_backup::20180:0:99999:7:::\n" not to match /^[^:]+::/
  ↺  pkg-02: auditd installed
     ↺  Skipped control due to only_if condition: not a Red Hat family host
  ↺  audit-07: Audit rules reviewed
     ↺  reviewed by hand each quarter, no automated check yet
 
 
Profile Summary: 2 successful controls, 1 control failure, 2 controls skipped
Test Summary: 4 successful, 1 failure, 2 skipped

Three parts to that. A header naming the profile, its version and the target, plus a Target ID, which is a stable fingerprint of the machine itself, so you can line up runs of the same box over weeks even if its address changes. Then one block per control. Then two summary lines at the bottom. Those two lines are the scoreboard, and you should read them before you scroll back up to anything else.

The summaries count different things, which is why they rarely agree. Profile Summary counts whole controls. Test Summary counts the individual assertions inside them. A single control can hold five describe blocks; if four hold and one breaks, Test Summary records four successes and one failure, while Profile Summary records exactly one control failure and no successes. Controls are graded like exam questions with no partial credit. One wrong line sinks the whole control. That is deliberate: a control stands for a compliance requirement, and a requirement is either met or it is not.

Now read the failure block instead of the failure count. InSpec prints the assertion that broke and the value it actually saw. Look at what it pulled out of /etc/shadow. The line svc_backup::20180 has two colons in a row where the password hash belongs, which means that service account has no password at all. Anybody who can reach a login prompt on that host can be svc_backup. The count told you something is wrong. Only the block tells you what.

That printed value carries a cost you should know about. InSpec dumps the entire observed value into the report, so matching against the content of /etc/shadow puts every password hash on the machine into your build log and into your JSON artifact, where a lot more people can read them. Match against a narrower resource instead. Writing describe shadow.where(password: '') do its('users') { should eq [] } end prints only the offending usernames when it fails, and nothing at all when it passes.

Real profiles get noisy fast. A CIS (Center for Internet Security) Linux benchmark profile carries several hundred controls, and scrolling past three hundred green ticks to reach the one you are debugging burns an afternoon. Narrow the run.

# run two controls by exact id
inspec exec ./linux-baseline -t ssh://[email protected] --sudo --controls sshd-01 sshd-04
 
# or select ids with a regular expression, a pattern language for matching
# text. Note the slashes inside the quotes: that is what tells InSpec to
# treat the string as a pattern rather than a literal id.
inspec exec ./linux-baseline -t ssh://[email protected] --sudo --controls '/^sshd-/'

Profile:   Linux Baseline (linux-baseline)
Version:   1.4.0
Target:    ssh://[email protected]:22
Target ID: 6f0d1c4e-5c9a-4b7e-9c3a-1f2b8d0a77e1
 
  ✔  sshd-01: SSH daemon configuration
     ✔  SSHD Configuration PermitRootLogin is expected to cmp == "no"
     ✔  SSHD Configuration PasswordAuthentication is expected to cmp == "no"
  ✔  sshd-04: SSH ciphers and key exchange
     ✔  SSHD Configuration Ciphers is expected to cmp == "[email protected],aes256-ctr"
 
 
Profile Summary: 2 successful controls, 0 control failures, 0 controls skipped
Test Summary: 3 successful, 0 failures, 0 skipped

That filter is sharper than it looks, and it cuts both ways. Get one character wrong in a control id and InSpec does not complain. It runs the empty set.

# one character wrong: sshd-1 instead of sshd-01
inspec exec ./linux-baseline -t ssh://[email protected] --sudo --controls sshd-1
echo "exit code: $?"

Profile:   Linux Baseline (linux-baseline)
Version:   1.4.0
Target:    ssh://[email protected]:22
Target ID: 6f0d1c4e-5c9a-4b7e-9c3a-1f2b8d0a77e1
 
 
Test Summary: 0 successful, 0 failures, 0 skipped
exit code: 0

inspec check gives you the number to guard against. It reads the profile and reports on its structure without touching a target host, so it is cheap enough to run on every commit, and it tells you how many controls are in there.

inspec check ./linux-baseline

Location:    ./linux-baseline
Profile:     linux-baseline
Controls:    5
Timestamp:   2026-07-22T09:12:44+00:00
Valid:       true
 
No errors or warnings

Five controls today. If tonight's run reports four, something dropped a control and nobody was told. That number is the whole gate, and it costs you one line of shell.

Three Statuses, and Only One of Them Is Evidence

Passed means the assertion was evaluated against this target, at this moment, and it held. That is evidence with a timestamp on it, not a promise about tomorrow morning. Failed means the assertion was evaluated and did not hold, and you get the expectation and the observed value printed side by side. Skipped means the assertion was never evaluated at all. Nothing was measured, nothing was observed. The line appears in the report so you know a check was meant to happen there, and that is the entire content of the message.

InSpec skips for four reasons, and in the default output all four look identical. An only_if guard on the control evaluated false. The resource does not exist on this platform, which is what happens when a Windows registry check lands on a Linux box. The profile source calls skip outright. Or a waiver file (a signed-off list of controls you have agreed not to enforce yet, loaded with --waiver-file) marks the control as approved and not run. Same yellow arrow, four very different meanings.

control 'pkg-02' do
  impact 0.5
  title 'auditd installed'
  desc  'The audit daemon records kernel-level events, so we can reconstruct who did what.'
 
  # Guard: Red Hat family only. Debian and Ubuntu ship the same daemon under
  # a different package name, so this exact check would report a false
  # failure there. On a Debian target the control is SKIPPED, not PASSED.
  only_if('not a Red Hat family host') { os.redhat? }
 
  describe package('audit') do
    it { should be_installed }
  end
end
 
control 'audit-07' do
  impact 0.7
  title 'Audit rules reviewed'
  desc  'Placeholder. Nobody has automated this one yet.'
 
  describe file('/etc/audit/rules.d/hardening.rules') do
    # skip() called inside the example records the reason in the report.
    # Honest, and it still produces exactly zero evidence.
    it 'matches the approved rule set' do
      skip 'reviewed by hand each quarter, no automated check yet'
    end
  end
end

Both controls land in the yellow column, and both are defensible on their own terms. pkg-02 really does not apply to a Debian host, though not for the reason people assume. Debian and Ubuntu do ship the audit daemon; they call the package auditd while Red Hat calls it audit, so running this exact check there would fail a machine that is perfectly well configured. audit-07 is a placeholder somebody left behind after a review meeting. The report cannot tell you which one is a sensible exclusion and which one is a hole, because the report only knows that neither of them ran. You have to read the reason string to find out.

The Skip Is the Status That Lies by Omission

A failure is honest. It names the broken thing and points at the line. A skip is quiet, and quiet is the part to distrust.

Here is the shape of the accident. Your team writes a Windows hardening profile: BitLocker (Microsoft's full-disk encryption) switched on, SMBv1 (an obsolete Windows file-sharing protocol) switched off, the built-in Administrator account renamed. Six months later a base image flips from Windows Server to Ubuntu, or somebody wires the wrong profile into the wrong pipeline stage. Every control skips on platform mismatch. The summary reads zero failures. The dashboard turns green and stays green. That machine is not hardened. It was never looked at, and the report says so if you read the middle column instead of the right one.

only_if guards rot the same way. only_if { os.redhat? } is correct the day it is written on a pure Red Hat Enterprise Linux fleet, and invisible the day somebody adds Debian nodes. The control does not fail on Debian. It evaporates. So watch the ratio, not the failure count. A profile reporting 40 successful, 0 failures, 12 skipped produced no evidence for roughly a quarter of your requirements, and you cannot tell an auditor which quarter without opening the skip reasons one at a time.

Exit Codes, the Only Part a Pipeline Reads

A courier does not read your handwriting back to dispatch. They punch one number into a handset: delivered, refused, address not found. Your build server works the same way. It ignores the coloured marks entirely and reads the exit code, where zero conventionally means success and anything else means some flavour of not-success. InSpec calls its behaviour here distinct exit codes, which means a wrapper script can tell the run outcomes apart without parsing a single line of text.

inspec exec ./linux-baseline -t ssh://[email protected] --sudo
echo "exit code: $?"

...
Profile Summary: 2 successful controls, 1 control failure, 2 controls skipped
Test Summary: 4 successful, 1 failure, 2 skipped
exit code: 100

The codes worth memorising. 0 means every test ran and passed, with nothing skipped. 100 means at least one test failed. 101 means at least one test was skipped and nothing failed. 1 means InSpec itself fell over. 2 is a failure inside the plugin system. 3 is a deprecation warning promoted to fatal. 172 means the Chef licence was not accepted. Failure outranks skip, so a run with three failures and forty skips returns 100 and never 101, which is precisely what makes a bare 101 a useful signal on its own.

Keep 1 firmly apart from 100 in your head, because they send you to different places. Exit 100 means InSpec formed an opinion and the opinion is bad news. Exit 1 means InSpec never got as far as an opinion: the host refused the connection, the credentials were wrong, the profile does not parse. Treating a transport error as "failed the audit" sends an engineer hunting for a broken control when the real problem is a firewall rule or an expired key.

# same profile, wrong host: 10.20.0.99 is not listening
inspec exec ./linux-baseline -t ssh://[email protected] --sudo
echo "exit code: $?"

[2026-07-22T09:41:17+00:00] ERROR: Train::Transports::SSHFailed: SSH session could not be established
exit code: 1

Train, in that error, is the connection library InSpec uses to reach targets. Treat it as the tool's plumbing. Anything carrying Train::Transports in its name is a pipe problem, not a finding about the machine, and no amount of reading your controls will fix it.

Code 172 catches teams moving up to InSpec 6. That release ships under a Progress commercial licence, and a missing or unaccepted licence stops the run before a single control executes, so a fleet-wide scan can return 172 on every host and look, to a badly written wrapper, like a mass failure. If those terms do not fit your environment, CINC Auditor (CINC stands for CINC Is Not Chef) is the community rebuild of InSpec from its open-source sources, with the Chef branding stripped out. The cinc-auditor command takes the same arguments, prints the same reports, and returns the same exit codes, minus the licence gate that produces 172.

Splitting the Skip Bucket With Enhanced Outcomes

Back to the padlocked room for a moment. There is a real difference between a room the inspector could not get into and a room the restaurant does not have. InSpec 5.18 and later can draw that line. Add --enhanced-outcomes and the single skipped bucket breaks into three. N/A (Not Applicable) covers controls switched off by an only_if guard or carrying impact 0.0. N/R (Not Reviewed) covers everything else that failed to run, such as an explicit skip or a resource the platform does not support. ERROR covers controls that raised an exception while executing, which previously hid among the skips and is usually a bug in your control rather than a finding about the host.

inspec exec ./linux-baseline -t ssh://[email protected] --sudo --enhanced-outcomes
echo "exit code: $?"

Profile:   Linux Baseline (linux-baseline)
Version:   1.4.0
Target:    ssh://[email protected]:22
Target ID: 6f0d1c4e-5c9a-4b7e-9c3a-1f2b8d0a77e1
 
  ✔  sshd-01: SSH daemon configuration
     ✔  SSHD Configuration PermitRootLogin is expected to cmp == "no"
     ✔  SSHD Configuration PasswordAuthentication is expected to cmp == "no"
  ✔  sshd-04: SSH ciphers and key exchange
     ✔  SSHD Configuration Ciphers is expected to cmp == "[email protected],aes256-ctr"
  ×  os-05: No empty password fields (1 failed)
     ✔  File /etc/shadow is expected to be owned by "root"
     ×  File /etc/shadow content is expected not to match /^[^:]+::/
  N/A  pkg-02: auditd installed
     ↺  Skipped control due to only_if condition: not a Red Hat family host
  N/R  audit-07: Audit rules reviewed
     ↺  reviewed by hand each quarter, no automated check yet
 
 
Profile Summary: 2 successful controls, 1 control failure, 1 control not reviewed, 1 control not applicable, 0 controls have error
Test Summary: 4 successful, 1 failure, 2 skipped
exit code: 100

That distinction is the difference between two very different conversations with an auditor. Not Applicable says this requirement does not apply to this machine, and a reviewer can accept it once they have checked the guard. Not Reviewed says this requirement applies here and nobody checked, which is a finding with your name on it. The default reporter paints both the same colour and calls them skipped.

Count the Statuses Yourself

Reading by eye works for one host. It falls apart across a nightly run of four hundred. Ask for a machine-readable report alongside the human one: --reporter takes a list, and any reporter can be redirected to a file using name:path. The counting below uses jq, a small command-line tool for pulling values out of JSON (JavaScript Object Notation, structured text that programs read easily). For very large suites, --reporter progress collapses the run to a stream of compact pass and fail marks plus the summary lines, which is far easier to watch scroll past in a build log than three thousand green ticks.

# human output on screen AND a JSON file on disk, from one run
inspec exec ./linux-baseline -t ssh://[email protected] --sudo \
  --reporter cli json:results.json
 
# count every individual test result by status
jq '[.profiles[].controls[].results[].status] | group_by(.) | map({(.[0]): length}) | add' results.json

{
  "failed": 1,
  "passed": 4,
  "skipped": 2
}

The JSON carries what the terminal trims. Every result object holds status, code_desc (the assertion written out in English), run_time, and, for a skip, a skip_message with the reason. A waived control also carries a waiver_data object with the justification text, an expiration_date, and a skipped_due_to_waiver boolean. That object is how you separate an exception somebody signed off on from an accidental platform mismatch, because in the terminal the two are the same yellow arrow.

# list every skipped control next to its reason, so nothing hides inside the count.
# any() picks each control once, even when several of its tests skipped.
jq -r '.profiles[].controls[]
       | select(any(.results[]?; .status == "skipped"))
       | "\(.id)\t\(.results[0].skip_message // .waiver_data.justification)"' results.json

pkg-02	Skipped control due to only_if condition: not a Red Hat family host
audit-07	reviewed by hand each quarter, no automated check yet

#!/usr/bin/env bash
set -uo pipefail
 
inspec exec ./linux-baseline -t "ssh://deploy@${HOST}" --sudo \
  --reporter cli json:results.json
code=$?
 
# 1, 2, 3 and 172 mean InSpec never reached an opinion. That is not a verdict.
case "$code" in
  0|100|101) ;;
  *) echo "inspec did not run cleanly (exit $code)"; exit 2 ;;
esac
 
total=$(jq '[.profiles[].controls[]] | length' results.json)
skipped=$(jq '[.profiles[].controls[].results[] | select(.status=="skipped")] | length' results.json)
 
# inspec check says this profile holds 5 controls. Fewer than that means a bad
# filter, or a control an upstream profile renamed under us. Raise the number
# as the profile grows.
[ "$total" -ge 5 ] || { echo "only $total controls ran; check the profile or the filter"; exit 2; }
 
# coverage holes fail loudly instead of passing quietly
[ "$skipped" -le 2 ] || { echo "$skipped skipped tests; read the reasons before shipping"; exit 4; }
 
exit "$code"

That script is deliberately dull. It refuses to read a transport error as a verdict, refuses to accept a run that tested almost nothing, and refuses to let a pile of skips slide through as a pass. Where the JSON gets stored, how long you keep it, and how it turns into evidence an auditor will sign are the subject of the reporting lesson.

Run your own profile twice this afternoon. Once as it stands, once with --enhanced-outcomes. If the second run surfaces even one N/R control you did not know about, that is a requirement nobody has ever checked on that machine, and you now have its id.

Try this

Run inspec exec ./linux-baseline -t ssh://[email protected] --sudo on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a filter that matches nothing still exits 0. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related