Testing detections

True-positive/negative tests and Atomic Red Team.

A smoke detector you never press the test button on is not a safety device. It is a plastic disc on the ceiling that makes you feel safe. Detection rules go the same way. Until you have watched a rule light up on a real attack, and stay silent while ordinary traffic walks past, we have a rule for that is a claim rather than a control. Detection testing is the habit of proving both halves of that claim on purpose, over and over, inside CI (continuous integration, the automated build that runs on every commit). Do that and the coverage number you report to leadership is a number you can stand behind at 2am with an incident open.

Every rule owes you two tests

An airport metal detector has two ways to be useless. It can stay quiet when someone walks through carrying a knife. It can also shriek at every belt buckle until the guards stop looking up. Detection rules fail in exactly those two directions, so every rule owes you two tests. A true-positive (TP) test feeds the rule telemetry from the real technique and asserts that it *fires*. Skip it and a silent gap sits on your dashboard wearing the costume of coverage. A true-negative (TN) test feeds it representative benign activity and asserts that it *stays quiet*. Skip it and you have built a noise machine that buries responders in false alerts. Here is the part cert exams like to probe: a rule that never matches anything and a rule that matches everything both survive a naive does the rule exist check. Only running both assertions together tells a working control apart from either failure mode. These are unit tests for detections, meaning small, fast, repeatable checks of rule *logic* against fixed sample events, with no live infrastructure in the loop. Wire them into the build so that loosening a rule until it misses the attack, or tightening it until it screams at normal traffic, breaks CI instead of shipping to production.

import json, pytest
from sigma_eval import matches   # thin project helper: (rule, event) -> bool
 
CASES = [
    # rule file,                          sample event,               should_fire
    ("rules/console_login_no_mfa.yml", "samples/login-no-mfa.json",   True),   # TP
    ("rules/console_login_no_mfa.yml", "samples/login-with-mfa.json", False),  # TN
]
 
@pytest.mark.parametrize("rule,event,should_fire", CASES)
def test_detection(rule, event, should_fire):
    assert matches(rule, json.load(open(event))) is should_fire
 
# ---- run it in CI ----
# $ pytest -q tests/test_detections.py
# .F
# FAILED test_detection[console_login_no_mfa.yml-login-with-mfa.json-False]
#   assert True is False        # rule ALSO fires on MFA logins -> too broad
# 1 failed, 1 passed in 0.14s   ->  CI blocks the merge

Convert the rule the way production will run it

A recipe written in grams is no use to a kitchen that only owns measuring cups. Somebody has to convert it, and the conversion is where the mistakes creep in. Sample tests prove your logic on paper. Your SIEM (security information and event management platform, the system that stores the logs and runs searches over them) does not execute YAML (a plain-text format for structured data). It runs a query in its own dialect. Sigma is a vendor-neutral YAML format for detection rules, pySigma is the Python library that parses it, and sigma-cli is the command-line wrapper around pySigma. Conversion takes two inputs. A backend is the target query language: Splunk SPL (Search Processing Language), Elastic ES|QL, or Microsoft kusto (KQL, Kusto Query Language). A pipeline is the field-mapping logic that translates Sigma's generic field names into the schema your logs actually use, so Sigma's Image becomes process.executable under an Elastic ECS (Elastic Common Schema) pipeline, while a Sysmon pipeline instead maps the process_creation category onto EventID=1. Convert the rule the same way CI will deploy it, then *read the query that comes out*. Silent failures live right here. A rule that is logically perfect but points at a field your pipeline never populates converts cleanly, deploys without error, and matches nothing, for as long as you leave it running.

$ pip install sigma-cli pysigma-backend-splunk pysigma-pipeline-sysmon
 
$ sigma list targets            # confirm the backend is installed
splunk   Splunk SPL & tstats queries
esql     Elasticsearch ES|QL
kusto    Microsoft Sentinel / Defender XDR (KQL)
 
$ sigma convert -t splunk -p sysmon rules/schtasks_persistence.yml
EventID=1 Image="*\\schtasks.exe" CommandLine="*/create*"
 
# Read that output before trusting it: the -p sysmon pipeline mapped Sigma's
# generic fields onto Sysmon EventID 1. Point it at the WRONG pipeline and the
# query compiles fine but queries fields your logs never fill in.

from sigma.collection import SigmaCollection
from sigma.backends.splunk import SplunkBackend
from sigma.pipelines.sysmon import sysmon_pipeline
 
rule = SigmaCollection.from_yaml(open("rules/schtasks_persistence.yml").read())
query = SplunkBackend(sysmon_pipeline()).convert(rule)[0]
print(query)
# -> EventID=1 Image="*\\schtasks.exe" CommandLine="*/create*"
 
# Now you can diff `query` against a golden file in CI: if a rule edit changes
# the compiled SPL unexpectedly, the build fails and a human reviews it.

Detonate the real technique

Reading the evacuation plan taped to the wall is not a fire drill. You learn different things when people actually walk down the stairs: which door sticks, which floor never hears the alarm. Adversary emulation is the fire drill. You run the real technique in a controlled lab and confirm the telemetry gets collected, parsed, and detected end to end, the whole path rather than the rule on its own. A converted query that matches your hand-written sample can still be blind in the wild, because you wrote that sample yourself. Atomic Red Team is the standard toolkit: a library of small tests mapped to MITRE ATT&CK (Adversarial Tactics, Techniques, and Common Knowledge, the public catalogue of how real attackers behave), each tied to a technique ID such as T1053.005, *Scheduled Task*. You drive them with the Invoke-AtomicTest cmdlet from the invoke-atomicredteam PowerShell module. The sequence never changes: look at what the test does, stage its prerequisites with -GetPrereqs, detonate, then run the built-in -Cleanup so the next run starts from a known-clean box.

PS> Install-Module -Name invoke-atomicredteam,powershell-yaml -Scope CurrentUser
PS> Import-Module invoke-atomicredteam
 
PS> Invoke-AtomicTest T1053.005 -ShowDetailsBrief     # what am I about to run?
T1053.005-1 Scheduled Task Startup Script
T1053.005-2 Scheduled task Local
 
PS> Invoke-AtomicTest T1053.005 -TestNumbers 1 -GetPrereqs   # stage inputs
Prerequisites met: T1053.005-1 Scheduled Task Startup Script
 
PS> Invoke-AtomicTest T1053.005 -TestNumbers 1               # detonate
Executing test: T1053.005-1 Scheduled Task Startup Script
SUCCESS: The scheduled task "T1053_005_OnLogon" has successfully been created.
SUCCESS: The scheduled task "T1053_005_OnStartup" has successfully been created.
Done executing test: T1053.005-1
 
PS> Invoke-AtomicTest T1053.005 -TestNumbers 1 -Cleanup     # remove artifacts

Prove it fired: query the telemetry

Pressing the test button is half the drill. The other half is walking to the panel in the lobby and checking the alarm was recorded. After a detonation, go and ask the platform directly. In Microsoft Sentinel and Defender XDR you ask in KQL. In Splunk you ask in SPL. Run the detection's logic as an ad-hoc search scoped to the few minutes around your detonation and confirm a row comes back. An empty result is a finding in its own right. It means the event died somewhere upstream: a stopped agent, a parser that broke on a format change, a field renamed by an ingest transform. That is exactly the class of failure a sample-only unit test can never see, because the sample never travelled anywhere.

// Microsoft Sentinel / Defender XDR (KQL)
DeviceProcessEvents
| where Timestamp > ago(15m)
| where FileName == "schtasks.exe" and ProcessCommandLine has "/create"
| project Timestamp, DeviceName, AccountName, ProcessCommandLine
 
// Timestamp             DeviceName    AccountName  ProcessCommandLine
// 2026-07-13 14:02:11   lab-win11-03  atrt-runner  schtasks /create /tn T1053_005_OnStartup /tr ...
 
 
# Splunk (SPL) — same event via Security channel EventCode 4698
index=win sourcetype="WinEventLog:Security" EventCode=4698 earliest=-15m
| table _time, host, Account_Name, Task_Name
 
# _time                 host          Account_Name  Task_Name
# 2026-07-13 14:02:11   lab-win11-03  atrt-runner   \T1053_005_OnStartup

From correct rules to real coverage

Per-rule testing proves individual detections are *correct*. Purple-team exercises, where the attack side and the defence side work the same range together, prove your *coverage* is wide. Map every technique you detonate to ATT&CK and score it detected, alerted, or missed. The heat map that falls out tells you where the next month of work belongs. Know what the green cells do not mean before you show them to anyone: a passing test means you detected one implementation of a technique, and emulation only ever covers behaviors somebody bothered to script. The trade-off is money. Full detonation needs an isolated lab whose telemetry mirrors production, and that lab costs real budget and real maintenance. So mature teams unit-test every rule on every commit, which is cheap and deterministic, and save end-to-end detonation for scheduled purple-team sprints and their highest-priority techniques. Every test on this page quietly assumes one thing: that the log arrived intact and on time. That assumption is what the next lesson, Log pipelines, takes apart, following an event from endpoint to the query you ran, through the parsing, routing, and enrichment stages where a detection you tested and trusted can still go blind without saying a word.

Unit tests and detonation tests fail in different ways, and your team needs to react differently to each. A failing pytest on a true-negative sample means the rule is too broad. Fix it before merge, even if the purple-team demo is tomorrow. A silent SIEM after Invoke-AtomicTest means something upstream broke: the agent, the parser, the pipeline mapping, or the compiled query. Write that miss down as a finding, with the exact timestamp window you searched. Do not shrug and re-run the atomic until something finally shows up. That habit trains a team to ignore real blind spots.

Keep a small matrix for every high-value technique: sample TP, sample TN, the converted query's golden file, the date of the last successful detonation, and the ATT&CK cell status (detected, alerted, or missed). Review it in the same meeting where you triage false positives. Teams that run "detection engineering" and "purple team" as separate efforts with no shared artifact end up with two dashboards that disagree, and an argument nobody wins. The matrix is the shared artifact.

When Atomic Red Team is too literal for your environment, write a thin internal harness that exercises the same behavior through the interfaces your operators actually use, for example creating a scheduled task through COM, or the cloud equivalent of whatever persistence worries you. Keep those harnesses in the detections repo, next to the rule they exercise. Treat cleanup as part of the test: a leftover artifact is a safety problem today and a false positive next quarter. Schedule weekly cleanup jobs in the lab subscription the same way you schedule the detonations.

Try this

In an isolated Windows lab with Sysmon (System Monitor, the Windows agent that logs process and network events) or your EDR shipping to a test SIEM, detonate one scheduled-task atomic and prove the event landed before you trust the rule.

PS> Import-Module invoke-atomicredteam
PS> Invoke-AtomicTest T1053.005 -TestNumbers 1 -GetPrereqs
Prerequisites met: T1053.005-1 Scheduled Task Startup Script
PS> Invoke-AtomicTest T1053.005 -TestNumbers 1
SUCCESS: The scheduled task "T1053_005_OnStartup" has successfully been created.
PS> Invoke-AtomicTest T1053.005 -TestNumbers 1 -Cleanup
 
# Then in Sentinel / Defender XDR (KQL), within ~15 minutes:
DeviceProcessEvents
| where Timestamp > ago(15m)
| where FileName == "schtasks.exe" and ProcessCommandLine has "/create"
| project Timestamp, DeviceName, ProcessCommandLine
# Expect one lab row. Empty result = collection/parsing gap, not a green cell.

Takeaway

Remember: every rule owes you a true-positive and a true-negative, plus a periodic end-to-end detonation. Sample tests prove the logic, atomics prove the pipe, and neither one on its own proves you cover the technique.

Next step: add pytest cases for one production rule, convert it the exact way CI deploys it, then put a monthly Atomic Red Team run for that technique on the calendar, with cleanup and a written log of whatever it missed.

Related