A Sentinel scheduled analytics rule runs every 5 minutes over the last 5 minutes of event time. In production it never fires, even though the account-takeover activity it looks for is happening, and no error shows up anywhere. What is the most likely cause?

Agent buffering, pipeline retries and vendor API polling delay the events, so they land in a time window the rule has already evaluated. This is the silent trap: the window closes on event time and assumes instant delivery. Fix it with an ingestion-delay overlap (earliest=-20m@m latest=-5m@m) or by filtering on ingestion_time().

You write the login-plus-new-key correlation but join on SourceIpAddress instead of UserIdentityArn. Several hundred staff sit behind one office NAT address. What happens when the rule runs?

Every login from that address pairs with every key creation from it, so candidate rows multiply and the rule produces meaningless matches. Joins hold candidate rows in memory and a shared NAT address is about as high-cardinality as a key gets. Always join on the most specific entity available, which here is the principal ARN.

Your monthly usage query returns ContainerLogV2 at 1942.7 GB, several times larger than anything else in the workspace, and no correlation rule reads that table. What is the sensible move?

Demote ContainerLogV2 to the Basic tier, accepting that it can no longer feed analytics rules but is still there to search during an investigation. Around 85% cheaper ingest for a table whose value is investigative rather than detective. The trade only works because nothing is correlating on it.

Courses Detection engineeringSIEM & correlation

SIEM & correlation

Ingest, correlate weak signals, manage cost.

Advanced30 min · lesson 5 of 15

A casino watches its floor through dozens of cameras, and one person upstairs watches all of them at once. Camera one shows a guest loitering near a high-stakes table. Unremarkable. Camera two shows the same guest chatting with a dealer. Still nothing. Camera three catches that dealer doing something odd with a shuffle. Any single frame is boring. The operator who recognizes the *same face* on all three screens is the one who calls security. A SIEM is that room for your infrastructure. Every log source is a camera, and correlation is the person upstairs putting the frames together.

A SIEM (Security Information and Event Management, the product category that includes Splunk, Elastic Security, Microsoft Sentinel and Google SecOps) is where your log pipeline delivers. It does three jobs. It ingests and indexes normalized events, so a search across billions of rows comes back in seconds. It runs detection rules, either on a schedule or against a near-real-time stream. And it retains history, so an investigation can look backward. Dashboards, case management, threat-intel enrichment: all of that hangs off those three.

Ingest, index, retain: what the box actually does

Parsing and normalization already happened upstream (that was the log pipelines lesson). The SIEM's own contribution starts with the index, which works like the index at the back of a textbook: a prebuilt map from words to page numbers, so you never have to read the whole book to find one term. Splunk builds time-bucketed inverted indexes, a journal of raw events plus a keyword-to-event map in its tsidx files. Elastic uses Lucene, the same idea with different plumbing. Sentinel sits on a columnar analytical store (Azure Data Explorer, the engine underneath Log Analytics). Google SecOps runs on Google's own planet-scale infrastructure with a BigQuery-backed data lake bolted on for analytics. The practical consequence is the same everywhere. A query filtered on indexed fields over a tight time range is fast and cheap. An unanchored regex (a text pattern with no fixed starting point) over 90 days of raw text is slow and expensive. Write your rules accordingly.

Detections run in one of two modes. Scheduled rules re-run a query every N minutes over a trailing window. They are the workhorse: cheap to execute, with the full query language available to you. Streaming rules, usually called NRT (near-real-time) rules, evaluate events as they arrive or on a very tight fixed cadence. Sentinel's NRT rules fire once a minute. You buy lower latency and you pay for it in expressiveness, because the logic has to stay simple and mostly stateless. Whichever mode you pick, map the rule's output to entities: the user, the host, the IP address. Entities are the keys that correlation and incident grouping join on, and a rule that emits bare rows with no entity attached is a rule nothing downstream can stitch together.

Three weak signals, one incident: correlation in KQL

Correlation is a join, the same move a bookkeeper makes when matching a paper receipt to a line on a bank statement. Two records from different places, matched on something they share, counted as a pair only if the timing lines up. In a SIEM you take events from different sources, match them on a shared entity, and require them to land inside a time window. Here is the classic cloud account-takeover chain written in KQL (Kusto Query Language, the language Sentinel searches with) against AWS CloudTrail, the audit log of every API (application programming interface) call made in an AWS account. A console login with no MFA (multi-factor authentication), then a new access key, then a burst of reads from S3 (Amazon's object storage). Same principal, fifteen minutes.

account-takeover.kql

// No-MFA login -> new access key -> bulk S3 reads, same principal, 15 min
let window = 15m;
let Logins = AWSCloudTrail
    | where EventName == "ConsoleLogin"
    | where tostring(parse_json(AdditionalEventData).MFAUsed) == "No"
    | project LoginTime = TimeGenerated, UserIdentityArn, SourceIpAddress;
let NewKeys = AWSCloudTrail
    | where EventName == "CreateAccessKey"
    | project KeyTime = TimeGenerated, UserIdentityArn;
let BulkReads = AWSCloudTrail
    | where EventSource == "s3.amazonaws.com" and EventName == "GetObject"
    | summarize Reads = count(), FirstRead = min(TimeGenerated) by UserIdentityArn
    | where Reads > 100;
Logins
| join kind=inner (NewKeys) on UserIdentityArn
| where KeyTime between (LoginTime .. (LoginTime + window))
| join kind=inner (BulkReads) on UserIdentityArn
| where FirstRead between (LoginTime .. (LoginTime + window))
| project LoginTime, UserIdentityArn, SourceIpAddress, KeyTime, Reads
 
// LoginTime             UserIdentityArn                         SourceIpAddress  KeyTime               Reads
// 2026-07-09T14:02:11Z  arn:aws:iam::111122223333:user/dev-sam  198.51.100.23    2026-07-09T14:06:47Z  412
// 1 row. Three medium signals -> one high-confidence incident.

No single where clause in that query would page anyone. A login without MFA happens dozens of times a day in most orgs. CreateAccessKey is what a developer does on a Tuesday afternoon. Heavy S3 reads describe every backup job ever written. The join is the detection. Deployed as a Sentinel scheduled analytics rule, running every 5 minutes over a 15-minute look-back, it costs almost nothing to execute, and mapping UserIdentityArn and SourceIpAddress as incident entities lets Sentinel pull related alerts into one incident instead of four. Google SecOps says the same thing in its own dialect: a YARA-L rule whose events section declares three event variables tied together by a shared $user match variable, with match: $user over 15m setting the window.

Late-arriving logs make a healthy rule fire never

A rule that runs every 5 minutes over the last 5 minutes of event time assumes logs turn up the instant they are written. They don't. Agent buffering, pipeline retries and vendor API polling (your SIEM pulling logs from a cloud service on a timer) routinely hold events back by minutes, and those events then land in a window your rule has already finished evaluating. Nothing errors. No warning appears anywhere. The detection is quietly dead. Filter on ingestion_time() in KQL, or give Splunk searches a skew offset like earliest=-20m@m latest=-5m@m, so a window only closes once the data has actually arrived.

The same hunt in SPL

Splunk's SPL (Search Processing Language) reaches the same incident by a different route. SPL does have a join command, but it is memory-hungry and capped at a row limit, so experienced Splunk engineers reach for something else. The idiomatic pattern is one search that pulls in all three event types, then stats grouped by the entity. You collapse everything down per principal, then filter on the *shape* of what collapsed.

account-takeover.spl

index=aws sourcetype=aws:cloudtrail earliest=-15m
    ((eventName=ConsoleLogin "additionalEventData.MFAUsed"=No)
     OR eventName=CreateAccessKey
     OR (eventSource=s3.amazonaws.com eventName=GetObject))
| stats earliest(_time) AS first latest(_time) AS last
        values(eventName) AS steps
        count(eval(eventName=="GetObject")) AS s3_reads
        by userIdentity.arn
| where mvcount(steps) >= 3 AND s3_reads > 100 AND (last - first) <= 900
| convert ctime(first) ctime(last)
 
userIdentity.arn                        first                last                 steps                                  s3_reads
arn:aws:iam::111122223333:user/dev-sam  07/09/2026 14:02:11  07/09/2026 14:13:58  ConsoleLogin CreateAccessKey GetObject  412

Read the where line slowly. mvcount(steps) >= 3 demands that all three distinct event names showed up. s3_reads > 100 sets the volume threshold. (last - first) <= 900 enforces the fifteen-minute window, expressed in epoch seconds. In Splunk Enterprise Security you would file this as a correlation search feeding RBA (risk-based alerting): each signal adds points to that entity's risk score, and a separate rule pages a human only once the running total crosses a line. That splits *interesting* from *pageable*, which is where mature SOCs (security operations centers) end up. Worth being precise here: a rule firing means a pattern matched. It does not mean an incident happened. A human still confirms that, which is why the number of alerts you generated is a measure of noise, not of how much you caught.

Inside the correlation engine, and where it breaks

Under the hood, a scheduled correlation rule is your query re-executed by a cron-like scheduler, with the SIEM keeping a watermark: a bookmark showing how far through event time it has already read. Every limitation falls out of that design. Joins hold candidate rows in memory, so high-cardinality entities explode combinatorially. Join on source IP when four hundred people share one office NAT (network address translation, which makes a whole building look like a single public address) and every login pairs with every key creation. Always join on the most specific entity available. Windows are rigid. An attacker who waits sixteen minutes walks straight past a fifteen-minute rule, so size windows from measured attacker tempo, meaning your purple-team timings and your own incident reports, rather than round numbers that feel tidy. And clocks lie. Source timestamps skew between systems, so ask for 'all N events inside the window' rather than a strict order, unless the order itself is the thing you are detecting.

From event to incident inside the SIEM

1Normalized events

handed off by the log pipeline

2Hot index

inverted/columnar store, fast search

3Correlation rules

joins on entity + time window

4Incident

entity-mapped, grouped, scored

5Response

handed to SOAR playbooks

Each stage narrows the funnel: millions of events, hundreds of rule matches, a handful of incidents worth a human's time.

Cost: ask your SIEM what it is eating

SIEM pricing is mostly ingest pricing. Sentinel's analytics tier runs roughly $4–5 per GB ingested at pay-as-you-go list price, depending on region. Splunk licenses by daily ingest volume or by workload. Either way, a chatty debug log costs exactly as much to index as a crown-jewel audit trail. That puts cost control in your hands rather than finance's, because you are the person who knows which tables actually feed rules. Start by asking the SIEM what it is eating:

ingest-cost-audit

// Sentinel: top billable tables over 30 days (Usage.Quantity is in MB)
Usage
| where TimeGenerated > ago(30d) and IsBillable == true
| summarize IngestedGB = round(sum(Quantity) / 1024, 1) by DataType
| top 5 by IngestedGB
 
// DataType          IngestedGB
// ContainerLogV2    1942.7
// SecurityEvent     812.4
// Syslog            623.9
// AWSCloudTrail     214.0
// SigninLogs        88.2
 
# Demote a high-volume, low-detection-value table to the cheap Basic tier
az monitor log-analytics workspace table update \
    --resource-group soc-rg --workspace-name soc-prod \
    --name ContainerLogV2 --plan Basic
# "plan": "Basic"   (~85% cheaper ingest; single-table KQL only, can't feed analytics rules)
 
# Keep CloudTrail interactive 90 days, archived to 1 year for compliance
az monitor log-analytics workspace table update \
    --resource-group soc-rg --workspace-name soc-prod \
    --name AWSCloudTrail --retention-time 90 --total-retention-time 365

The tiering logic is short. Tables that feed correlation rules (identity logs, CloudTrail, EDR, meaning endpoint detection and response, the agent running on laptops and servers) stay in the full analytics tier where rules can reach them. High-volume telemetry you only ever grep during an investigation, such as container stdout and verbose network flows, drops to a basic tier, and you accept the trade: it can no longer power an analytics rule. Anything kept purely because an auditor will ask for it goes to long-term retention, reachable only through an explicit search job or a restore. Splunk's equivalent levers are index-time filtering at the forwarder, and routing bulk data to cheaper S3-backed storage (SmartStore, or federated search over data you never index at all).

The habits worth keeping: map every rule to entities; join on the most specific entity you have; size windows from measured behavior; schedule with an ingestion-delay overlap so late events still correlate; tier tables by detection value and re-run that usage query monthly, because volume drifts and last quarter's answer goes stale. One thing a SIEM will *not* do is act. When the account-takeover incident above fires at 3 a.m., something still has to disable dev-sam's access keys before the copying finishes, and waiting for a human to wake up hands the attacker exactly the window they wanted. Wiring detections to automatic action is the job of SOAR (security orchestration, automation and response), and that is where we go next.

Try this

In a lab SIEM, run a small correlation-shaped query that joins identity logs and cloud audit logs on the same user inside a short window. The join key is the whole point of the exercise. Confirm the same person is identifiable on both sides before you trust a word the rule tells you.

terminal

// Example KQL shape — adapt table names to your workspace
let fail = SigninLogs
| where ResultType != 0 and TimeGenerated > ago(1h)
| project TimeGenerated, UserPrincipalName, IPAddress;
let keys = AWSCloudTrail
| where EventName == "CreateAccessKey" and TimeGenerated > ago(1h)
| project TimeGenerated, UserIdentityUserName, SourceIPAddress;
fail
| join kind=inner keys on $left.UserPrincipalName == $right.UserIdentityUserName
| where abs(datetime_diff('minute', TimeGenerated, TimeGenerated1)) <= 15
| project UserPrincipalName, IPAddress, SourceIPAddress, TimeGenerated, TimeGenerated1
# Empty join with known lab events ⇒ field mapping or identity normalization gap.

Takeaway

A SIEM indexes, detects and retains. Correlation is how three forgettable events turn into one case worth waking somebody for. Join on entities that stay stable, bound the time window tightly, and keep one eye on what the ingest is costing you while you do it.

Next step: pick one multi-stage attacker path you genuinely fear, write a correlation for it with an explicit entity key and an explicit window, and replay a controlled lab sequence against it before you let it page anybody.

Log pipelinesSOAR & response

eBPF for runtime securityRuntime & eBPF security