Alert quality & tuning

Precision over volume; beat alert fatigue.

A car alarm that howls at every passing truck stops protecting the car. It teaches the whole street to stop looking out of the window. Detection rules die the same death. Alert fatigue is what happens when a queue fills with alerts that turn out to be nothing: analysts learn that *nothing* is the safe bet, triage gets faster and shallower, and the one alert that is a real intrusion gets closed with the same reflexive click as the five hundred that were not. Tuning means deliberately removing the alerts nobody can act on, measured rather than guessed. That is not janitorial work. It decides whether anything you built in the previous thirteen lessons ever gets read.

Measure before you tune

Tuning by gut feel goes after the rule that annoys the loudest analyst, not the rule that eats the most attention. Start with ground truth, which your SIEM (security information and event management platform, the system that stores your logs and raises your alerts) already keeps for you. Every closed incident carries a triage verdict, the answer a human wrote down after actually looking. A rule firing is a claim. The verdict is the ruling on that claim, and the two are not the same thing. Microsoft Sentinel records three verdicts that matter here. TruePositive means real malicious activity. FalsePositive means the rule logic matched something it never should have. BenignPositive means the rule matched exactly the behavior it describes, but that activity was authorized: a pentest, an admin's script. From those verdicts you get per-rule precision, the fraction of everything a rule fired that deserved a human's time.

// Sentinel: per-rule precision over the last 30 days.
// SecurityIncident writes a new row on every update — arg_max keeps only
// the final state of each incident.
SecurityIncident
| where TimeGenerated > ago(30d) and Status == "Closed"
| summarize arg_max(TimeGenerated, Classification) by IncidentNumber, Title
| summarize TP = countif(Classification == "TruePositive"),
            FP = countif(Classification == "FalsePositive"),
            BP = countif(Classification == "BenignPositive") by Title
| extend Precision = round(1.0 * TP / (TP + FP + BP), 2)
| where TP + FP + BP >= 10   // don't judge a rule on two data points
| order by Precision asc
 
// Title                              TP    FP    BP    Precision
// ---------------------------------- ----- ----- ----- ---------
// Impossible travel sign-in          2     57    141   0.01
// PowerShell EncodedCommand          4     112   37    0.03
// Anonymous SMB share enumeration    6     41    12    0.10
// Kerberoasting: mass SPN query      9     3     1     0.69

Two lines do the real work: the guard and the sort. Requiring ten closed incidents stops you from "tuning" a rule on two data points. Sorting ascending puts your backlog in cost order. *Impossible travel* at 0.01 precision and 200 fires burns roughly 33 analyst-hours a month, at ten minutes per triage, to surface two real events. That is the number worth taking to a manager. The raw alert count is not, because volume on its own says nothing about health: a busy queue can be all noise, and a silent queue can be blind. Splunk Enterprise Security stores the same verdicts under a different name, dispositions, in its Incident Review queue. The vendor matters far less than the habit of measuring before you touch anything.

Write an exception, do not kill the rule

When a rule with real coverage turns noisy, the reflex is to switch it off. That swaps a visible annoyance for an invisible hole. The malicious case the rule existed to catch now happens unseen, and nothing reminds you of that every morning the way the noise did. The professional fix is a scoped exception, a filter that carves out one specific, verified benign pattern and leaves every other path to the alert intact. *Scoped* is the word carrying the weight here. NOT user="svc-*" is not an exception. It is an amnesty for every service account in the company. A good exception names the process, the parent process that launched it, the account, and ideally something an attacker cannot copy on a whim.

An exception is one lever among several. You can tighten the rule's selection so it demands more evidence at once (the encoded command *and* an outbound connection). You can raise an aggregation threshold, alerting on fifty failed logins instead of five. Or you can demote the rule from paging anyone to a context-only signal that enriches other alerts. Pick the narrowest lever that kills the noise you actually measured. Every lever costs you some recall (the share of real attacks you would still catch), and the narrow ones cost the least.

Exceptions as code: Sigma filters

If your rules live in Sigma, the vendor-neutral rule format from earlier in this course, resist the urge to edit exceptions into the rule file itself. An upstream rule is like a library you did not write: patching your local copy is how you end up stranded. Rules from SigmaHQ or a commercial feed get updated, and every local edit turns into a merge conflict on the next sync or, worse, coverage that disappears without a sound. Sigma v2 handles this with filter documents (the command-line tool calls them *meta filters*): standalone YAML files (a plain-text format for configuration) that name the rule IDs they apply to and carry nothing but the exception.

title: Sanctioned batch PowerShell via scheduler
description: svc-batch launches signed maintenance scripts every 15 min (CHG-4412)
logsource:
  category: process_creation
  product: windows
filter:
  rules:
    - fb843269-508c-4b76-8b8d-88679db22ce7   # Suspicious Execution of Powershell with Base64
  selection:
    ParentImage|endswith: '\scheduled-batch-runner.exe'
    User|endswith: '\svc-batch'
  condition: not selection

The mechanics are easy to follow. At conversion time, sigma-cli finds every rule whose id appears in the filter's rules list, checks that the logsource matches too, and ANDs the filter's condition into that rule's own condition. The query it emits for your SIEM carries your exception as a final NOT clause. The upstream rule file is never touched.

$ sigma convert -t splunk --filter filters/ \
    rules/proc_creation_win_powershell_encode.yml
 
Parsing Sigma rules  [####################################]  100%
Image IN ("*\\powershell.exe", "*\\pwsh.exe") CommandLine IN ("* -e *",
"* -en *", "* -enc *", "* -enco*", "* -ec *")
NOT (CommandLine="* -Encoding *" OR ParentImage IN
("*C:\\Packages\\Plugins\\Microsoft.GuestConfiguration.ConfigurationforWindows\\*",
"*\\gc_worker.exe*"))
NOT (ParentImage="*\\scheduled-batch-runner.exe" User="*\\svc-batch")

Everything above the last line is the upstream rule: its selection, plus the filters its author already built in. The final NOT is yours. Because your exception is a file, it travels through the same pull-request review and the same CI checks (continuous integration, the automated tests that run on every change) as a rule does, through the pipeline you built in *Testing detections*. That makes every exception reviewable, attributable to a person and a change ticket, and revertible in one commit the day that batch job is decommissioned.

Deduplicate and throttle

Some noise is not wrong, only repetitive. One misconfigured agent can emit the same alert every thirty seconds, and five hundred copies of a true positive are still four hundred ninety-nine too many. Throttling (Splunk's word) and alert suppression (Elastic's) collapse the repeats. After the first alert for a given key, further matches inside a time window raise nothing new: Splunk holds the alert action until the window expires, and Elastic additionally counts the suppressed matches on the original alert. The key you pick decides everything. Suppress on (user, dest) and a second compromised user still alerts. Suppress on the rule name alone and it does not.

# Splunk: one alert per unique (user, dest) pair per 4 hours.
# A different user or destination still alerts immediately.
[Suspicious PowerShell - Encoded Command]
counttype = number of events
relation = greater than
quantity = 0
alert.suppress = 1
alert.suppress.fields = user,dest
alert.suppress.period = 4h

Elastic Security says the same thing per detection rule with alert_suppression: { group_by: [...], duration: {...} }. The cost in both products is the window itself. Activity that begins *during* suppression inherits the silence, so keep windows short on high-severity rules, minutes to a few hours, and save the day-long windows for informational ones.

Enrich and prioritize

An alert that reads 4688: powershell.exe on WKS-0231 (event 4688 is Windows recording that a new process started) sends an analyst on five separate lookups before they can even decide whether to care. It is a smoke alarm that names the building and not the floor. Enrichment attaches the answers up front: how critical the asset is and who owns it, from the CMDB (configuration management database, the inventory of your machines and services), the user's privilege level and department, whether the host faces the internet, and threat-intel verdicts (what outside intelligence feeds already say about a file hash or an IP address, the numeric address of a machine on a network). Where you enrich is a money decision. Static, cheap joins like asset tags belong in the log pipeline you built in *Log pipelines*, so everything downstream inherits them. Volatile or per-lookup-priced data, such as commercial threat-intel APIs (application programming interfaces you call and pay for by the query), belongs at alert time, where you pay for hundreds of lookups a day instead of millions.

Risk-based alerting pushes prioritization one step further. Points on a driving licence work the same way: one small offence does not take your licence, and enough of them together do. Instead of every rule raising its own alert, each match adds a score to the entity involved, a user or a host, and only an accumulated score crossing a threshold pages a human. Splunk Enterprise Security ships this as Risk-Based Alerting, where detections become *risk rules* that add score to a risk object and a separate risk-incident rule fires on the running total. Elastic Security's entity risk score rests on the same principle. It is the strongest known answer to the low-and-slow attacker who stays under every individual rule's threshold. You pay for it twice: in detection latency, because the score needs time to build, and in triage effort, because "why did this fire" now has several answers to reconstruct.

Tuning never finishes, because the environment never stops drifting. New batch jobs appear, teams reorganize, and last quarter's exception becomes this quarter's blind spot. Run the precision query on a weekly schedule, put an expiry date on every filter, and treat a rule whose precision collapses the way you treat a broken build. Precision is only half the picture, though. A rule can be perfectly precise and still miss everything, and a program can run a quiet queue while an attacker walks in unseen. The other half, coverage, detection latency and the loop that turns every incident into new detections, is where this course closes, in *Metrics & the loop*.

Try this

Pull seven days of closures for one noisy rule out of your SIEM and work out a rough precision. Then decide: exclude, split, or rewrite. Whichever you pick, encode the choice in git.

// KQL sketch for precision on one rule
SecurityIncident
| where TimeGenerated > ago(7d)
| where Title has "Certutil Used as a Downloader"
| summarize
    total=count(),
    tp=countif(Classification == "TruePositive"),
    fp=countif(Classification == "FalsePositive")
  by Title
| extend precision = todouble(tp) / todouble(total)
# Title                                      total tp fp precision
# Certutil Used as a Downloader              40    2  38 0.05
# → write filter for SCCM parent, add negative sample, PR the Sigma change.

Takeaway

Tuning protects attention, the scarcest thing a security team owns. Measure precision before you touch a rule, write every exclusion down with a review date, and split a noisy rule that still carries value rather than silencing a whole technique.

Next step: pick the worst-precision rule that pages someone this week, add a tested filter with a ticket reference, and refuse any suppression that lives in a console and never reaches git.

Related