Policy testing & coverage

Trust your policies before shipping.

A green test suite tells you one thing: for the inputs you thought of, your policies behaved. Coverage tells you something more humbling. It shows which lines of Rego (the policy language used by OPA, the Open Policy Agent) your tests actually made the engine run. A building inspector who signs off after walking three rooms of a twelve-room house has not lied to you. He also has not inspected the house.

Add --coverage (short flag -c) to opa test and OPA writes down every Rego line the evaluator touched. The not_covered ranges are the rooms nobody opened. Add --threshold and your continuous integration (CI) build goes red when the number slips. A deny rule that never fired in any test compiled fine and reads fine, and it gives you nothing: you have zero evidence about what it does when a real violation shows up.

Turn the report on

Coverage falls out of opa test for free. Pass --coverage (or -c) and the evaluator records which lines it ran while your tests ran. Ask for --format json and you get a JSON (JavaScript Object Notation) report with, per file, the covered and not_covered line ranges plus a percentage. There is nothing to instrument, no build flag, no wrapper script. If opa test runs, coverage runs with it.

opa test --coverage --format json ./policy/ > coverage.json

# coverage.json written

opa test -c ./policy/

PASS: 12/12
Coverage: 87.5%

Find the branches nobody tested

The headline percentage is the least useful number in the report. The signal sits in not_covered: line ranges your tests never reached. Nine times in ten it is a deny condition no fixture satisfies, a helper nobody calls any more, or an else branch. Every gap resolves one of two ways. Write the missing test, or delete the dead Rego.

opa test -c -f json ./policy/ | jq '.files | to_entries[] | select(.value.coverage < 100) | {file: .key, pct: .value.coverage}'

{
  "file": "policy/k8s.rego",
  "pct": 72.2
}

opa test -c -f json ./policy/ | jq '.files["policy/k8s.rego"].not_covered'

[{"start": {"row": 24}, "end": {"row": 28}}]

Fail the build below a bar

--threshold takes a percentage. If total coverage lands under it, opa test exits non-zero and the pipeline stops, even when every test passed. Run it in the same CI job as the tests so an untested deny rule cannot slip into main. Set the bar where you actually are today (80 is a fine place to start) and ratchet it up a point at a time as you backfill cases.

opa test --coverage --threshold 85 ./policy/

PASS: 12/12
Coverage: 87.5%

# GitHub Actions step
opa test policy/ --coverage --threshold 85

##[error] Process completed with exit code 2.
# when coverage drops below 85%

Lint and format in the same job

Coverage on its own leaves holes that formatting and compile checks catch. opa fmt --list prints the files that are not in canonical form, which lets you fail CI on style drift. opa check --strict refuses code that compiles in loose mode while hiding a mistake. Regal, a linter built specifically for Rego, flags known bug patterns, including the undefined-field traps that make a rule quietly never fire. Policy is code. Give it the same guardrails you give the application.

opa fmt --list policy/ && test -z "$(opa fmt --list policy/)"
opa check --strict policy/
regal lint policy/

# exit 0 when formatted, compiles, and passes lint

Per-file coverage beats the headline

The aggregate number counts your _test.rego files too, and those run almost end to end every time, so they drag the total upward. A suite proudly reporting 92% can be hiding authz.rego sitting at 60%. Pull the per-file numbers out of the JSON report with jq (the command line tool for slicing JSON) and gate on the policy files themselves, rather than trusting --threshold on the blended figure.

Touching a line once with a weak input still leaves the compound condition on the next line unproven. After you close the line gaps, walk the branches: both ways a deny can trigger, empty input, empty data. Then run the same policies through Conftest (the tool that tests config files against Rego) using fixtures shaped like the files you really ship.

One number, then a backlog

opa test -c -f json policy/ | jq '.coverage'

87.5

Inherited a policy repo nobody has ever tested? Run coverage before you write a single new test. The not_covered list is your backlog, already assembled by the tool. Work it in risk order: deny rules guarding security critical paths first, formatting helpers last.

Break it on purpose

Here is the move strong teams use. Flip an operator inside a rule, change == to !=, run the suite, and check that something turns red. Coverage would happily report that line as executed either way, so it cannot tell you the test asserted anything about the result. You do not need mutation testing tooling for this. One deliberately broken commit on a throwaway branch shows whether the suite guards behavior or only visits it.

Regal can also enforce complexity limits on rules, which pairs well with coverage: one watches the shape of your policy, the other watches its reach. And opa check --strict catches a typo in a rule head that loose mode accepts happily, then never matches anything at runtime.

When the aggregate threshold papers over one thin file, write the gate yourself in the CI script. Loop over policy/*.rego, skip anything ending in _test.rego, pull each file's percentage with jq, and compare it against 85 with a shell test. It is ugly, and it fails on the file that actually matters.

Read coverage diffs in a pull request (PR) the way you read code diffs. A new deny rule arriving with no new test should fail review even when the global number still clears the bar, because the test files themselves are propping that number up.

Publish the trend once a week. Coverage sliding down while the policy line count climbs is a clean tech debt signal, and it shows up on a chart long before anyone feels it during an incident.

Put this in the on-call runbook: when a deployment is blocked by a new policy, the alert should link to the exact conftest command that reproduces the failure inside the policy repo. Whoever gets woken at 3am should not have to hunt for the one engineer who remembers writing the rule.

Teach newer engineers that a green coverage number with no Conftest fixture behind it is hollow. Walk them through one pull request that landed four things together: the deny rule, its unit test, a bad.yaml fixture that trips it, and the coverage bump proving the new lines ran. That is the shape every policy change should have.

Post the coverage diff as a PR comment by running jq over the head report and the artifact saved from the base branch. Reviewers then see which deny branches the change touched without checking the branch out and running anything locally.

If you have static application security testing (SAST) tooling that understands Rego, point it at the policy repo. Most teams do not, and Regal covers a good share of the same ground: undefined references, deprecated syntax, expressions that compile but can never match.

One slide a quarter is enough for leadership: policy count, test count, coverage trend, how many Conftest failures blocked a merge, how many denials Gatekeeper (OPA running as a Kubernetes admission controller) issued in the cluster. Those tie the policy program to outcomes anyone can count, without pretending a line of Rego equals a unit of risk removed.

Have the nightly run on main push the coverage JSON into object storage (an S3 bucket, say) under a date prefix. A trend graph built from six months of those files moves a team in a way a single number in a single pull request never does.

Clean the shelf every quarter

Rego that has stayed uncovered for two quarters running is almost certainly dead. The coverage report names those orphans for you. Deleting them buys fewer untested paths, a smaller bundle, and faster evaluation on every decision.

A team goal worth setting: no policy pull request merges if any .rego file it touched sits below 85%. Enforce it with a jq comparison of the coverage report between base and head, not with a promise written on a wiki page.

Progress stalls when coverage only ever measures the happy path. Force fixtures for the awkward inputs: a missing field, an empty list, an object of the wrong kind. Those are exactly the cases where a Rego expression goes undefined and the rule quietly declines to add anything to the deny set.

Store the coverage report next to the bundle you publish, tagged with the same version. Six months later, when somebody asks why a rule exists, the report plus the fixture that fails without it is your answer, and it beats anybody's memory.

Twice a year, cross check the report against your decision logs and drop the rules that nothing has hit in either place. Re-baseline the threshold after a large refactor instead of leaving a number that no longer describes anything. And decline any change that lowers coverage without replacing the assertions it removed.

Neither Regal nor opa check needs coverage to do its work, and coverage will never find what they find: unused variables, a not written where it flips the meaning of a rule, rule names nothing ever queries. Run all three in one CI job so a change has to clear style, compilation, and reach in a single pass.

Try this

Raise the threshold, archive the JSON report, and run fmt and check in the same job. When the number dips under the bar, go find the branch nobody tested. Do not quietly edit the bar downward.

opa test . -v --coverage --threshold 85
opa test . --coverage --format json > coverage.json
opa fmt --list .
opa check .

PASS: 12/12
Coverage: 91.4%
# coverage.json archived for the PR
# opa fmt --list prints files that need formatting
# opa check reports compile issues

Takeaway

Coverage is a flashlight, not a trophy. Per-file numbers beat a green headline that hides one deny path nobody ever exercised, and running lint and fmt in the same job stops style drift from covering for logic drift.

Keep the mutation habit. Flip one condition, confirm a test turns red. It is the fastest way to spot rules that are covered on paper and asserted nowhere.

Related