Metrics & the loop

Coverage, MTTD, precision — measure and improve.

Take the stopwatch, the scale and the training log away from an athlete and what is left is exercise. Effort, with no way to tell whether it is building strength or only sweat. A detection program without numbers works the same way: rules get written, alerts get closed, dashboards glow green, and nobody can say whether your organization would catch a real intrusion faster this quarter than last. This closing lesson bolts on the instruments. Three families of numbers, coverage, precision and speed, turn everything you have built across this course into something you can steer. Measure, diagnose, improve, measure again.

The three numbers worth tracking

Coverage asks: of the attacker behaviors that could realistically hit *your* environment, how many can you spot? The measuring stick is MITRE ATT&CK (Adversarial Tactics, Techniques and Common Knowledge), the public catalog of attacker moves you met in the Pyramid of Pain lesson, trimmed down to the techniques that apply to your stack. Precision asks: when an alert fires, how often is something actually wrong? True positives divided by total alerts. Its flip side, the share of alerts that turn out to be nothing (statisticians call that the false discovery rate; a SOC, or security operations center, will call it the false-positive rate), is what buries analysts. Speed asks two things: how long does an intrusion run before you notice (mean time to detect, MTTD), and how long from noticing to shutting it down (mean time to respond, MTTR)?

Look at what is missing from that list. Rule count. Alerts per day. Events ingested. Tools bought. Those are vanity metrics: they measure activity, not outcome. A SIEM (security information and event management platform, the system that swallows your logs and runs your rules) holding 900 rules and firing 5,000 alerts a day can have thinner real coverage and far worse precision than a tuned program running 200. Every number in this lesson comes out of data you already hold: your rule repository, and your SIEM's incident records.

Coverage: your rules against your actual threat model

Start with *claimed* coverage. If you followed the detection-as-code lesson, every Sigma rule in your repo carries attack.tXXXX tags naming the techniques it goes after. Those tags are your numerator. The denominator is your threat profile: the set of techniques used by the crews that target your industry and your stack, usually exported from a CTI (cyber threat intelligence) platform or an ATT&CK Navigator layer. From there it is set arithmetic.

# Claimed coverage: unique ATT&CK technique IDs tagged across your Sigma repo
grep -rhoE 'attack\.t[0-9]{4}(\.[0-9]{3})?' rules/ \
  | sed 's/^attack\.//' | tr '[:lower:]' '[:upper:]' | sort -u > covered.txt
wc -l < covered.txt
# 143
 
# Denominator: techniques in your threat profile (ATT&CK Navigator export)
jq -r '.techniques[] | select(.score > 0) | .techniqueID' threat-profile.json \
  | sort -u > relevant.txt
wc -l < relevant.txt
# 212
 
# Intersection = coverage; difference = next quarter's backlog
comm -12 covered.txt relevant.txt | wc -l
# 121                      -> 121 / 212 = 57% claimed coverage
comm -13 covered.txt relevant.txt | head -3
# T1055.001
# T1098.003
# T1567.002
# ...each gap becomes a hunting hypothesis or a new-rule ticket

Claimed coverage flatters you, always. A rule tagged t1059.001 might catch one flavor of encoded PowerShell while nine bypasses stroll straight past it. The tag records where you aimed, not what you hit. Validated coverage is the honest version, and it comes from the testing lesson: run the Atomic Red Team tests mapped to each technique and count the detections that actually fire. Then check the floor underneath both with DeTT&CT, which scores your *data source* coverage. No rule can catch credential dumping if the endpoint telemetry never reaches your pipeline. Publish the result as a Navigator heatmap so the holes are visible to everyone:

import json
techs = [{"techniqueID": t.strip(), "score": 1, "color": "#31a354"}
         for t in open("covered.txt")]
layer = {"name": "Detection coverage 2026-07", "domain": "enterprise-attack",
         "versions": {"layer": "4.5", "navigator": "5.3"}, "techniques": techs}
json.dump(layer, open("coverage-layer.json", "w"), indent=2)
 
# Open https://mitre-attack.github.io/attack-navigator/
#   -> Open Existing Layer -> upload coverage-layer.json
# Green cells = covered techniques; blank cells inside your threat profile = gaps.

Precision: read it off your closed incidents

You already own the precision data. Analysts write it every time they close an incident. In Microsoft Sentinel, the SecurityIncident table appends a fresh row on *every* update, and a Classification field gets set at close time: *TruePositive*, *FalsePositive*, *BenignPositive* (real activity, no threat, such as a pentest or an admin script), or *Undetermined*. Here is the detail that trips people up. Because each update is its own row, you have to collapse down to the latest state per incident with arg_max() before you count anything. Skip that step and an incident that was edited after close, or reopened and closed again, shows up as several "Closed" rows and gets counted several times.

SecurityIncident
| where TimeGenerated > ago(90d)
| summarize arg_max(TimeGenerated, *) by IncidentNumber   // latest state per incident
| where Status == "Closed"
| summarize total  = count(),
            tp     = countif(Classification == "TruePositive"),
            fp     = countif(Classification == "FalsePositive"),
            benign = countif(Classification == "BenignPositive")
            by Title
| extend precision = round(todouble(tp) / total, 2)
| order by total desc
 
// Title                                    total   tp   fp  benign  precision
// Suspicious service installed              412     9  361      42       0.02
// Impossible travel to atypical location    118    31   64      23       0.26
// Honeytoken AWS key used                     6     6    0       0       1.00

Read that output like a portfolio review. The service-installation rule fired 412 times to find nine real hits, 2% precision, which makes it a candidate for the tuning workflow from the alert-quality lesson or for the bin. The honeytoken rule is tiny and perfect: deception buys a level of precision that behavioral logic never reaches. Remember the base-rate problem from the UEBA (user and entity behavior analytics) lesson. When real intrusions are rare, even an accurate detector produces mostly false alarms, so a high-volume rule sitting below roughly 30% precision is a running cost, not a rounding error. Every alert spends analyst minutes. Precision per rule tells you which rules are worth the spend.

Speed: MTTD and MTTR, reported as percentiles

MTTD is the gap between the attacker's first move you can see (FirstActivityTime, the earliest event time across the alerts correlated into the incident) and the moment the incident was created. MTTR runs from creation to containment or close. For scale: Mandiant's M-Trends reports have put the global median *dwell time*, first compromise to detection, at around ten days in recent years, while ransomware crews routinely go from foothold to encryption inside 24 hours. Squeeze MTTD and you squeeze the damage. One rule about how you report it. Use medians and 90th percentiles, never means. A single intrusion found after 60 days drags an average past any target you set, and tells you nothing about a normal week.

SecurityIncident
| where TimeGenerated > ago(90d)
| summarize arg_max(TimeGenerated, *) by IncidentNumber
| where Status == "Closed" and isnotempty(FirstActivityTime)
| extend mttd_min = datetime_diff('minute', CreatedTime, FirstActivityTime),
         mttr_min = datetime_diff('minute', ClosedTime,  CreatedTime)
| summarize p50_mttd = percentile(mttd_min, 50), p90_mttd = percentile(mttd_min, 90),
            p50_mttr = percentile(mttr_min, 50), p90_mttr = percentile(mttr_min, 90)
            by Severity
 
// Severity   p50_mttd   p90_mttd   p50_mttr   p90_mttr      (minutes)
// High             22        240         95       1370
// Medium          133       2880        310       4820
//
// Medium's p90 MTTD of 2,880 min (2 days) is the number to attack first:
// usually a missing log source or a correlation rule that fires too late.

Watch for survivorship bias. MTTD is only ever computed over the intrusions you found. The ones you missed contribute nothing to the number; they arrive later as a breach notification. Purple-team exercises and post-incident reviews are the correction. Every red-team action and every step on a post-mortem timeline gets the same question put to it, "did a detection fire?", and every *no* becomes a coverage gap with an owner and a ticket.

The loop that ties this course together

Stand those three measurements in front of the toolchain from this course and detection engineering collapses into one feedback loop. Coverage gaps turn into hunting hypotheses, and a hunt that finds something ends its life as a new Sigma rule, converted with sigma convert, proven against Atomic Red Team tests in your CI (continuous integration) pipeline, and merged: detection as code. Low-precision rules go through the tuning workflow or get swapped for a sharper signal, a honeytoken in place of a noisy heuristic, aimed at behaviors near the top of the Pyramid of Pain where evasion costs the attacker real effort. A slow MTTD points at telemetry you never collected, which is a log-pipeline fix. A slow MTTR points at manual toil, which is a SOAR (security orchestration, automation and response) playbook. Incidents flow into the cloud DFIR (digital forensics and incident response) process, where you acquire evidence and build the timeline, and the retrospective sends every missed step back as a tested detection. UEBA baselines cover the ground that enumerable rules cannot.

Run the loop every quarter and the program compounds. A year in, you should be able to show coverage climbing against a named threat profile, median per-rule precision above 50%, and p50 MTTD measured in minutes for the techniques that matter. The numbers argue your budget for you, because "we detect 71% of our threat profile, up from 57%" is a sentence a CISO (chief information security officer) can take into a board meeting and defend. That is where this course lands: a measured, versioned, tested, automated system that gets better every time you exercise it, rather than a pile of rules.

Try this

Build yourself a one-page scoreboard: precision on your paging rules over the last seven days, sitting next to a count of ATT&CK tags on the rules you have enabled. Save it as a dashboard you will actually open on a Monday morning.

// Precision across paging incidents (adapt classifications to your SIEM)
SecurityIncident
| where TimeGenerated > ago(7d) and Severity in ("High","Critical")
| summarize total=count(), tp=countif(Classification == "TruePositive")
| extend precision = todouble(tp) / todouble(total)
# total  tp  precision
# 28     17  0.61
 
# Coverage sketch from your detections repo:
$ grep -R "attack.t" -h rules/ | sort | uniq | wc -l
47
# Pair: 47 tagged techniques enabled ≠ 47 techniques proven by detonation.
# Add a column for last successful test date before you call it coverage.

Takeaway

Measure outcomes: coverage of the techniques that threaten you, precision, MTTD, and whether your canaries are still alive. Rule count is not one of them. The loop is detect, respond, learn, then change the program on purpose.

Next step: publish a weekly precision and canary dashboard, and make one pull request to the detection repo a condition of closing any Sev-1 incident ticket.

Related