The compliance-scanning landscape

OpenSCAP, OSQuery, Terratest.

A restaurant kitchen gets checked three different ways. An inspector walks the room with a printed checklist and leaves a signed certificate for the wall. A probe in the walk-in fridge logs the temperature every fifteen minutes whether anyone is watching or not. And before a new dish reaches the menu, someone cooks it once in a test kitchen to find out whether the recipe is safe at all. Compliance tooling splits the same three ways. InSpec is the inspector with the readable checklist. OpenSCAP is a second inspector, the one who fills out the government's own forms. OSQuery is the fridge probe. Terratest is the test kitchen. Reach for the wrong one and you spend a month bending InSpec into a shape it was never built to hold.

These three turn up in the same audits as InSpec, over and over. You do not have to pick a winner. You do have to know which question each one answers, because the evidence they hand you is not interchangeable, and an assessor will spot the mismatch before you do.

OpenSCAP: The Auditor's Own Paperwork

Go back to the health inspector and look at what is actually in their hands. Two separate things. There is the form, which lists the questions, marks how serious each one is, and says what to do when the answer comes back wrong. And there is the thermometer, the instrument that produces the reading the form asks for. SCAP (Security Content Automation Protocol, a family of standards from NIST, the US National Institute of Standards and Technology, for writing security checks in a form that different vendors' tools can trade back and forth) splits security content exactly that way. XCCDF (Extensible Configuration Checklist Description Format) is the form: rule titles, severities, remediation text, and which rules belong to which profile. OVAL (Open Vulnerability and Assessment Language) is the thermometer: does this file contain this line, is this package older than that version. A datastream is the envelope holding both, plus a dictionary of which platforms the content applies to, as one shippable file.

OpenSCAP is the open-source scanner that reads all of that. It carries NIST certification as a SCAP scanner, which is the part an assessor actually cares about. Compare the reading experience: an InSpec control is Ruby you can follow over a colleague's shoulder, while SCAP content is XML that no human writes by hand. So do not start. The SCAP Security Guide (SSG) is a maintained open-source collection of datastreams for RHEL (Red Hat Enterprise Linux), Ubuntu, SUSE and others, each carrying official profiles: CIS benchmarks (from the Center for Internet Security, a non-profit that publishes consensus hardening guides), DISA STIGs (the US Defense Information Systems Agency's Security Technical Implementation Guides), ANSSI (the French national cyber-security agency's baselines), and PCI-DSS (Payment Card Industry Data Security Standard, the rules that bind you if you touch card data). On a RHEL 9 box you install the scanner and the content, then look inside the datastream before you scan anything.

sudo dnf install -y openscap-scanner scap-security-guide
 
# what profiles does this datastream actually ship?
oscap info /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml

Document type: Source Data Stream
Imported: 2026-03-11T09:22:14
 
Stream: scap_org.open-scap_datastream_from_xccdf_ssg-rhel9-xccdf.xml
Generated: (null)
Version: 1.3
Checklists:
	Ref-Id: scap_org.open-scap_cref_ssg-rhel9-xccdf.xml
		Status: draft
		Generated: 2026-03-11
		Resolved: true
		Profiles:
			Title: ANSSI-BP-028 (enhanced)
				Id: xccdf_org.ssgproject.content_profile_anssi_bp28_enhanced
			Title: CIS Red Hat Enterprise Linux 9 Benchmark for Level 2 - Server
				Id: xccdf_org.ssgproject.content_profile_cis
			Title: CIS Red Hat Enterprise Linux 9 Benchmark for Level 1 - Server
				Id: xccdf_org.ssgproject.content_profile_cis_server_l1
			Title: DISA STIG for Red Hat Enterprise Linux 9
				Id: xccdf_org.ssgproject.content_profile_stig
		Referenced check files:
			ssg-rhel9-oval.xml
				system: http://oval.mitre.org/XMLSchema/oval-definitions-5
			ssg-rhel9-ocil.xml
				system: http://scap.nist.gov/schema/ocil/2
Checks:
	Ref-Id: scap_org.open-scap_cref_ssg-rhel9-oval.xml
	Ref-Id: scap_org.open-scap_cref_ssg-rhel9-ocil.xml
	Ref-Id: scap_org.open-scap_cref_ssg-rhel9-cpe-oval.xml
Dictionaries:
	Ref-Id: scap_org.open-scap_cref_ssg-rhel9-cpe-dictionary.xml

Read that list slowly, because "CIS" on its own is ambiguous and the ambiguity is expensive. The bare _cis id is Level 2 Server, which is a great deal stricter than _cis_server_l1. Pick the wrong one and you meet two hundred failures on day one, the team decides compliance is unreasonable, and the whole exercise quietly dies. The rest of the output is worth knowing too. The OVAL file does the machine testing. The OCIL file (Open Checklist Interactive Language) holds the checks no machine can answer, the ones where a human has to be asked a question. And the dictionary at the bottom is CPE (Common Platform Enumeration, a naming scheme for operating systems and products), which the scanner uses to decide whether a given rule applies to this box at all. Now run the scan and ask the shell what the scanner made of it.

sudo oscap xccdf eval \
  --profile xccdf_org.ssgproject.content_profile_cis_server_l1 \
  --results scan-results.xml \
  --results-arf scan-arf.xml \
  --report report.html \
  /usr/share/xml/scap/ssg/content/ssg-rhel9-ds.xml
echo "exit=$?"

Title   Install sudo Package
Rule    xccdf_org.ssgproject.content_rule_package_sudo_installed
Ident   CCE-83523-1
Result  pass
 
Title   Set SSH Client Alive Interval
Rule    xccdf_org.ssgproject.content_rule_sshd_set_idle_timeout
Ident   CCE-90811-1
Result  fail
 
Title   Disable SSH Root Login
Rule    xccdf_org.ssgproject.content_rule_sshd_disable_root_login
Ident   CCE-90800-4
Result  fail
 
Title   Ensure gpgcheck Enabled In Main dnf Configuration
Rule    xccdf_org.ssgproject.content_rule_ensure_gpgcheck_globally_activated
Ident   CCE-83457-2
Result  pass
 
exit=2

That Ident line matters more than it looks. A CCE number (Common Configuration Enumeration) is a stable global identifier for one configuration setting, the way a barcode identifies one product no matter which shop stocks it. When your scanner says CCE-90800-4 and the assessor's tool says CCE-90800-4, you are provably talking about the same setting rather than two similarly worded rules. Three artifacts also came out of that single run, and each has a different audience. --results writes an XCCDF results file: the same checklist, now with a TestResult section recording what happened, which is what a GRC platform (Governance, Risk and Compliance, the system of record where an organization tracks its obligations) ingests without argument. --results-arf writes an ARF bundle (Asset Reporting Format), which wraps the results together with the content that produced them, so a reviewer six months later can prove which version of which rule was evaluated. --report writes a self-contained HTML page a human can actually read. That bundle is why OpenSCAP survives in regulated shops. The paperwork already matches the form the assessor asked for.

OpenSCAP also does something InSpec deliberately refuses to do: it will write the fix for you. Feed the results file back in and it generates remediation for the rules that failed, as a shell script or an Ansible playbook. The flag people get wrong here is the selector. Against a datastream you pass --profile and get fixes for every rule in the profile. Against a results file you pass --result-id, which is the id of the TestResult section written during the scan, and you get fixes only for what actually failed.

# the result id is inside the results file; "oscap info scan-results.xml" prints it
oscap xccdf generate fix \
  --fix-type ansible \
  --result-id xccdf_org.open-scap_testresult_xccdf_org.ssgproject.content_profile_cis_server_l1 \
  --output remediate.yml \
  scan-results.xml
 
head -29 remediate.yml

---
 
###############################################################################
# BEGIN fix (1 / 2) for 'xccdf_org.ssgproject.content_rule_sshd_set_idle_timeout'
###############################################################################
- hosts: all
  vars:
    sshd_idle_timeout_value: '900'
  tasks:
    - name: Gather the package facts
      ansible.builtin.package_facts:
        manager: auto
      tags:
        - sshd_set_idle_timeout
    - name: 'Set SSH Client Alive Interval: Set ClientAliveInterval'
      ansible.builtin.lineinfile:
        path: /etc/ssh/sshd_config
        create: true
        regexp: (?i)^\s*ClientAliveInterval\s+
        line: ClientAliveInterval {{ sshd_idle_timeout_value }}
        state: present
      when: '"openssh-server" in ansible_facts.packages'
      tags:
        - CCE-90811-1
        - low_complexity
        - low_disruption
        - medium_severity
        - restrict_strategy
        - sshd_set_idle_timeout

Read that playbook before you run it. Generated remediation is blunt on purpose. It rewrites config files in place, and for some rules it restarts the service afterwards, and it has no idea why anything on your box is the way it is. Notice that the SSH fix edits /etc/ssh/sshd_config with no syntax check and no backup. Get a related rule wrong on a box you reach only over SSH and you have locked yourself out of it. Treat the output as a first draft you review in a pull request, run against a throwaway host first, and never as something a pipeline applies unattended to production.

OSQuery: Turn The Fleet Into A Database

Instead of walking to every fridge with a clipboard, you ask one question and every fridge answers at once. That is OSQuery. It exposes the running operating system as a read-only database you query with SQL (Structured Query Language, the same language that pulls rows out of any ordinary database). Processes become rows in a processes table. Open network sockets become rows in listening_ports. Installed packages, loaded kernel modules, logged-in users, cron entries, browser extensions, file hashes: each is a table you can SELECT from and JOIN against the others. It began at Facebook, now Meta, and has lived under the Linux Foundation since 2019. It runs on Linux, macOS and Windows with mostly the same tables, so a single query travels across a mixed estate.

# which processes are listening on an address reachable from off-box?
osqueryi --json "
  SELECT p.name, p.pid, l.address, l.port, l.protocol
  FROM listening_ports l JOIN processes p ON l.pid = p.pid
  WHERE l.address NOT IN ('127.0.0.1', '::1', '');"

[
  {"address":"0.0.0.0","name":"sshd","pid":"1123","port":"22","protocol":"6"},
  {"address":"0.0.0.0","name":"nginx","pid":"1481","port":"443","protocol":"6"},
  {"address":"0.0.0.0","name":"redis-server","pid":"8802","port":"6379","protocol":"6"}
]

Two things to notice. Every value came back wrapped in quotes, including the port and the protocol number (6 is TCP), because osquery's JSON output carries no types at all. Whatever consumes it downstream has to cast the numbers back itself, and a comparison like port > 1024 done on strings in your log pipeline will give you nonsense. The second thing is that third row. A Redis server bound to every interface, on a host where nobody remembers installing Redis. An InSpec baseline would catch that only if somebody had already written a control naming Redis. The query found it without knowing to look, which is the difference between checking a list and asking what is there.

osqueryi is the interactive shell for one host, good for hunting. osqueryd is the daemon that runs a fixed schedule and writes results to a log that the whole fleet forwards into your SIEM (Security Information and Event Management platform, the place logs go to be searched, correlated and alerted on). A fleet manager such as Fleet, from fleetdm, hands the config out to every host and gathers the answers back.

{
  "options": {
    "config_refresh": 300,
    "logger_path": "/var/log/osquery",
    "schedule_splay_percent": 10,
    "watchdog_memory_limit": 350,
    "disable_events": false
  },
  "schedule": {
    "world_reachable_listeners": {
      "query": "SELECT p.name, p.pid, l.address, l.port FROM listening_ports l JOIN processes p ON l.pid = p.pid WHERE l.address NOT IN ('127.0.0.1', '::1', '');",
      "interval": 900,
      "description": "New or removed off-box listeners"
    },
    "sshd_config_integrity": {
      "query": "SELECT path, sha256 FROM hash WHERE path = '/etc/ssh/sshd_config';",
      "interval": 3600,
      "snapshot": true
    }
  },
  "decorators": {
    "always": [
      "SELECT hostname AS host, uuid AS host_uuid FROM system_info;"
    ]
  }
}

The behavior that trips everyone up on day one is differential logging. A night watchman who radios in every fifteen minutes to say "still fine" is a waste of a radio. One who calls only when a door opens is useful. Scheduled osquery queries work the second way. The first run logs every row as added, and after that you get a line only when a row appears or disappears. That is precisely what makes a fifteen-minute schedule affordable across ten thousand hosts, and it is also why your dashboard looks dead when nothing has changed. When you need the whole answer every single run, as with that file hash, set "snapshot": true on that query and accept the extra volume. The decorators block stamps hostname and UUID (universally unique identifier, the machine's permanent serial number) onto every result line, which is the difference between a searchable log and a pile of anonymous rows.

The daemon also polices itself, the way a circuit breaker protects a house from one bad appliance. If a query pushes the worker past watchdog_memory_limit (measured in megabytes, 200 by default, raised to 350 above) or burns too much CPU, the watchdog stops the worker, restarts it with an exponential backoff, and denylists the offending query for 24 hours. So your greedy SELECT * FROM file WHERE path LIKE '/%%'; will not take the fleet down. In osquery's path syntax a single % matches one directory level and %% walks the tree recursively, so that query means "stat every file on the disk". It will stop running instead, which is worse than crashing, because you will go on believing you have coverage you do not have. When a scheduled query goes quiet, read the osquery_schedule table and check its executions, wall_time_ms and denylisted columns before you conclude the hosts are clean.

Terratest: Catch It In The Pull Request

Terratest does not read your Terraform and reason about it. It runs it. A Go test calls terraform init and terraform apply against real cloud APIs, asserts against what actually came into existence, then destroys it. That is the test kitchen: you cook the dish once, taste it, and throw it away before it ever reaches the menu. Where InSpec inspects a server that already exists, Terratest kills a non-compliant module while it is still a diff a reviewer can reject.

package test
 
import (
	"strings"
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)
 
func TestS3BucketIsEncrypted(t *testing.T) {
	t.Parallel()
 
	opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../modules/s3",
		Vars: map[string]interface{}{
			// unique, lowercase name so parallel runs never collide
			"bucket_name": "acme-audit-logs-" + strings.ToLower(random.UniqueId()),
		},
	})
 
	// registered BEFORE apply, so a half-finished apply still gets torn down
	defer terraform.Destroy(t, opts)
	terraform.InitAndApply(t, opts)
 
	assert.Equal(t, "aws:kms", terraform.Output(t, opts, "sse_algorithm"))
	assert.Equal(t, "true", terraform.Output(t, opts, "block_public_acls"))
}

cd test && go test -v -timeout 30m ./...

=== RUN   TestS3BucketIsEncrypted
=== PAUSE TestS3BucketIsEncrypted
=== CONT  TestS3BucketIsEncrypted
    TestS3BucketIsEncrypted 2026-07-22T10:12:41Z logger.go:66: Running command terraform with args [init -upgrade=false]
    TestS3BucketIsEncrypted 2026-07-22T10:12:58Z logger.go:66: Running command terraform with args [apply -input=false -auto-approve -var bucket_name=acme-audit-logs-a7f3k1 -lock=false]
    TestS3BucketIsEncrypted 2026-07-22T10:13:29Z logger.go:66: aws_s3_bucket.this: Creation complete after 3s
    TestS3BucketIsEncrypted 2026-07-22T10:13:34Z logger.go:66: Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
    s3_encryption_test.go:29:
        	Error Trace:	/home/ci/infra/test/s3_encryption_test.go:29
        	Error:      	Not equal:
        	            	expected: "aws:kms"
        	            	actual  : "AES256"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-aws:kms
        	            	+AES256
        	Test:       	TestS3BucketIsEncrypted
    TestS3BucketIsEncrypted 2026-07-22T10:13:35Z logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -lock=false]
    TestS3BucketIsEncrypted 2026-07-22T10:13:52Z logger.go:66: Destroy complete! Resources: 4 destroyed.
--- FAIL: TestS3BucketIsEncrypted (71.09s)
FAIL
FAIL	github.com/acme/infra/test	71.284s
FAIL

The module produced AES256, which is S3-managed encryption, where the standard called for a customer-managed KMS key (Key Management Service, the AWS service that holds encryption keys you control and log). Both scramble the bytes at rest, so a checkbox audit would pass either. The difference shows up on the day you need it. With AES256 you cannot revoke the key, you cannot write a key policy that denies a compromised role, and you cannot look at a key-usage log to see who read the object. The test caught that in seventy-one seconds, in a pull request, on a bucket that no longer exists. The alternative is finding it eighteen months later during an audit of four hundred buckets that are already full of data.

The honest cost: you write Go, the runs take minutes rather than seconds, and every run creates billable resources. Your CI system (continuous integration, the service that builds and tests every change) needs credentials that can create and destroy real infrastructure, which is a security decision in its own right and deserves an argument before it deserves a ticket. Point those tests at a dedicated sandbox account with a spending cap, never at anything sharing a blast radius with production.

Where The Tools Meet

InSpec is the piece that translates between these worlds, because its output is pluggable. The --reporter flag takes several formats in one go, so a single run prints a human summary to the terminal and writes machine-readable JSON to disk at the same time. From there, MITRE's SAF CLI (Security Automation Framework) converts between shapes. saf convert xccdf_results2hdf -i scan-results.xml -o openscap.json pulls an OpenSCAP results file into HDF (Heimdall Data Format, a common shape for security findings regardless of which scanner produced them), which is the same structure InSpec's JSON reporter emits. Then saf convert hdf2ckl -i openscap.json -o findings.ckl turns either one into a STIG Viewer checklist your assessor already knows how to open. That is how a team runs two scanners and still hands over one consolidated pile of evidence instead of two arguments.

# xinetd removal is scheduled, but this box ships with it today
package-01:
  expiration_date: 2026-09-30
  run: false
  justification: "Legacy appliance host, removal tracked in SEC-4412"
 
# this one still runs and still reports, it only stops failing the build
os-04:
  expiration_date: 2026-08-15
  run: true
  justification: "Build agent needs . in PATH for vendor toolchain, SEC-4390"

inspec exec https://github.com/dev-sec/linux-baseline \
  --target ssh://ops@web-01 \
  --waiver-file waivers.yml \
  --reporter cli json:baseline.json junit2:baseline-junit.xml
echo "exit=$?"

Profile:   DevSec Linux Security Baseline (linux-baseline)
Version:   2.10.0
Target:    ssh://ops@web-01:22
Target ID: 6b1a4f2c-0d33-5a71-9c4e-2f8e11d0a7bd
 
  ✔  os-01: Trusted hosts login
     ✔  File /etc/hosts.equiv is expected not to exist
  ✔  os-02: Check owner and permissions for /etc/shadow
     ✔  File /etc/shadow is expected to be owned by "root"
     ✔  File /etc/shadow is expected not to be executable
  ×  sysctl-01: IPv4 Forwarding (2 failed)
     ×  Kernel Parameter net.ipv4.ip_forward value is expected to eq 0
     expected: 0
          got: 1
     ×  Kernel Parameter net.ipv4.conf.all.forwarding value is expected to eq 0
     expected: 0
          got: 1
  ↺  package-01: Do not run deprecated inetd or xinetd
     ↺  Skipped control due to waiver condition: Legacy appliance host, removal tracked in SEC-4412
 
Profile Summary: 51 successful controls, 1 control failure, 1 control skipped
Test Summary: 118 successful, 2 failures, 1 skipped
exit=100

That listing is trimmed; the profile ships around fifty controls and prints them all. Look at what the waiver did to package-01. The justification you wrote is not filed away somewhere private, it is printed into the report and carried into the JSON, so the reason lands in front of whoever reads the evidence. The run exits 100 because sysctl-01 failed, and note that one control produced two failures, since it checks both net.ipv4.ip_forward and net.ipv4.conf.all.forwarding. Waive that control with run: false and the next run exits 101: nothing failed, something was skipped. Still non-zero, still red in a pipeline that breaks the build on any non-zero code. That is correct behavior and you should not paper over it. A waived control is a debt with a due date, and 101 is the pipeline reminding you the debt is still on the books. Handle the three codes deliberately and let expiration_date do its work. The morning after that date passes, the waiver stops applying and the control fails for real.

One selection factor changed recently and it is not a technical one. From version 6 onward, Chef InSpec wants a license key from Progress, and it will open an interactive licensing prompt the first time you run it. In a pipeline, where nothing can answer that prompt, the run dies with exit code 172, which is a confusing failure to debug if you have never seen it before. CINC Auditor is the open-source rebuild of the same codebase, put out by the CINC project (the name stands for "CINC Is Not Chef"). It runs the same profiles with the same control language and you invoke it as cinc-auditor instead of inspec. OpenSCAP, OSQuery and Terratest carry no such question, and in a procurement meeting that is occasionally the argument that settles things, long before anyone compares features.

Point-In-Time Versus Continuous

The trap when you combine these is treating all four kinds of evidence as the same kind of evidence. An InSpec or OpenSCAP run is a photograph. It records what was true at the instant the shutter opened, signed and timestamped, which is exactly what an assessor wants on file. A host can drift out of compliance four minutes later and the green report stays green until the next scan. OSQuery running every fifteen minutes will see that drift, but a raw results log is not an attestation, and no auditor will accept a SIEM query in place of a scan report. Photograph and fridge probe, and you need both for different reasons.

Work through what the gap between them costs. Your baseline scan passes on Monday at 02:00. On Tuesday afternoon someone debugging a cache starts Redis with its stock config, which binds 0.0.0.0:6379 and asks nobody for a password. Your next scheduled scan is Friday. That is roughly three days in which anyone who can reach the port reads and writes your cache at will, and the compliance report for that week still says the estate was clean, because it was, at 02:00 on Monday. The OSQuery schedule from earlier catches the new listener within fifteen minutes and emits exactly one line naming the host, the process and the port. Terratest would have stopped it earlier still, if the Redis had arrived through a Terraform module rather than someone's shell prompt.

So write that gap down as a number, next to each control objective, in whatever document your assessor actually reads. If the baseline scan runs weekly and the OSQuery schedule runs every fifteen minutes, then your worst-case exposure window is fifteen minutes for anything a query watches and seven days for anything only the scan covers. Two numbers, one line each. The moment somebody has to type "7 days" into a box with their name on it, the argument for shortening the scan interval or writing one more query stops needing to be made by you.

Try this

Run sudo dnf install -y openscap-scanner scap-security-guide on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the same exit code number means opposite things in these two tools. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related