Testing detections

True-positive/negative tests and Atomic Red Team.

A detection you have not tested is a hypothesis, not a control. Detection testing — proving a rule fires on the malicious case and stays quiet on the benign one — is what separates coverage you can rely on from a false sense of security.

True positives and true negatives

Every detection needs two kinds of test. A true-positive test feeds it telemetry from the actual technique and asserts the rule fires — otherwise you have a silent gap that looks like coverage. A true-negative test feeds it representative benign activity and asserts the rule stays quiet — otherwise you have a noise machine that will drown responders. A rule that never fires and one that always fires both pass a naive "it exists" check; only both tests together prove it works. Wire these into CI so a change that breaks either is a failed build, not a production surprise.

# true-positive: malicious sample → the rule MUST fire
- input: cloudtrail-console-login-no-mfa.json
  expect: match
 
# true-negative: normal MFA login → the rule MUST NOT fire
- input: cloudtrail-console-login-with-mfa.json
  expect: no-match
 
# CI runs the rule against both; either wrong assertion fails the build.
# Regression: someone loosens the rule → the negative test catches the new noise.

Detonate the real technique

Sample-based tests prove logic; adversary emulation proves end-to-end coverage. Tools like Atomic Red Team execute small, ATT&CK-mapped techniques safely in a test environment, so you can confirm the telemetry is actually collected, the pipeline parses it, and the detection fires — the full path, not just the rule in isolation. Purple-team exercises (attackers and defenders working together) do this at scale, mapping which techniques you detect and which you miss. The findings feed straight back into the detection-as-code loop: a missed technique becomes a new, tested rule.