Activity Log & immutable trails

Diagnostic settings to a locked, central store.

The front-desk register and the room cameras

Every secure building keeps two kinds of records. The front desk logs who unlocked which door and when. Cameras inside each room record what people actually did once they were in. Lose the register and you cannot say who came through the door. Lose the cameras and you know someone opened the vault, but not what they took out of it.

Azure splits its audit trail the same way. The Activity Log is the front-desk register. It records control-plane operations (management actions on resources): who created, changed, or deleted something, from which address, and when, across the whole subscription. Azure writes it for you and keeps roughly 90 days of it at no charge. Diagnostic settings are the room cameras. They capture the data plane (what happens inside a resource): a secret read from Key Vault, an object pulled from storage, a network firewall's allow-or-deny decision.

Here is the asymmetry that catches people. The register runs by default. The cameras are off. Almost nothing keeps data-plane logs until you attach a diagnostic setting to it. So a Key Vault an attacker empties, or a storage account they quietly copy, leaves you nothing to reconstruct afterward, because nobody turned the camera on. A diagnostic setting is the switch that turns an un-auditable resource into an investigable one. You cannot respond to what you never recorded.

Wire both planes into one workspace

Correlation needs one review room where every feed lands next to the register, so you can line events up in time. A Log Analytics workspace (Azure Monitor's queryable log store) is that room, and it is what Microsoft Sentinel, Azure's cloud SIEM (security information and event management, the system that gathers logs and runs detections on them), reads from. Detection only works if both planes flow into the same workspace.

You wire the Activity Log in once, at the subscription scope, with az monitor diagnostic-settings subscription create. This is the step people forget in a fresh subscription, because it is not a per-resource setting and nothing nags you for it. The security-relevant categories are Administrative (who changed what), Security (which carries Microsoft Defender for Cloud alerts, the service formerly named Azure Security Center), and Policy. Each resource's data-plane categories go through that resource's own setting, and the category names differ by type. Key Vault emits AuditEvent. A storage account emits StorageRead, StorageWrite, and StorageDelete. A network security group (NSG, the allow/deny rules wrapped around a subnet or a network interface card) emits NetworkSecurityGroupEvent and NetworkSecurityGroupRuleCounter, the record of which rules actually fired. Packet-level flow logs are a separate Network Watcher feature that writes to a storage account rather than riding this diagnostic setting, and the older NSG flow logs are being retired in favor of virtual network (VNet) flow logs. List the categories before guessing.

# Before wiring a resource, ask what it can even emit. Categories differ per type.
kvid=$(az keyvault show -g app -n app-kv --query id -o tsv)
 
az monitor diagnostic-settings categories list --resource "$kvid" \
  --query "value[].{name:name, type:categoryType}" -o table

Name                          Type
----------------------------  --------
AuditEvent                    Logs
AzurePolicyEvaluationDetails  Logs
AllMetrics                    Metrics

# Control plane: the Activity Log -> central workspace. Do this ONCE per subscription.
sub=$(az account show --query id -o tsv)
wsid=$(az monitor log-analytics workspace show -g sec-logging -n central-la --query id -o tsv)
 
az monitor diagnostic-settings subscription create \
  --name activity-to-la --location global --workspace "$wsid" \
  --logs '[{"category":"Administrative","enabled":true},
           {"category":"Security","enabled":true},
           {"category":"Policy","enabled":true}]'

{
  "location": "global",
  "name": "activity-to-la",
  "logs": [
    { "category": "Administrative", "enabled": true },
    { "category": "Security", "enabled": true },
    { "category": "Policy", "enabled": true }
  ],
  "type": "Microsoft.Insights/diagnosticSettings",
  "workspaceId": "/subscriptions/8f2c1e9a-.../resourceGroups/sec-logging/providers/microsoft.operationalinsights/workspaces/central-la"
}

# Data plane: this Key Vault's AuditEvent -> the same workspace.
az monitor diagnostic-settings create --name kv-to-la \
  --resource "$kvid" --workspace "$wsid" \
  --logs '[{"category":"AuditEvent","enabled":true}]'
 
# Confirm the vault is no longer invisible.
az monitor diagnostic-settings list --resource "$kvid" \
  --query "value[].{name:name, workspace:workspaceId}" -o table

Name      Workspace
--------  ----------------------------------------------------------------------
kv-to-la  /subscriptions/8f2c1e9a-.../resourceGroups/sec-logging/.../central-la

One production fact shapes every choice here: ingestion is where a Log Analytics bill comes from. You pay per gigabyte pulled in and per gigabyte kept. A chatty category like StorageRead on a hot account can become the biggest line on your Azure invoice. Route noisy tables to a cheaper Basic or Auxiliary table plan, or straight to archive, and keep the interactive Analytics tier for what your detections actually query. Then list the settings back every time. A create that silently pointed at the wrong workspace looks fine right up to the incident where you go looking and the data was never there.

A workspace is not evidence yet

The tapes in that review room can be edited by anyone holding a key to the room. A workspace is not tamper-proof. Anyone with enough rights, or an attacker who has taken those rights, can shorten its retention, drop a table, or purge rows. For a trail you can put in front of an auditor or a court, keep a second copy that no one can rewrite.

That copy lives in immutable blob storage (write-once-read-many, or WORM: once data lands, it cannot be edited or deleted for a fixed period). Think of a safe-deposit box that seals the instant you close it and will not reopen until a set date, not even for the bank. A time-based retention policy fixes that minimum retention in days. The flag --allow-protected-append-writes lets a streaming export keep adding new blocks (fresh log data) while forbidding any change to what is already written.

While the policy is Unlocked you can still weaken it, including dropping the period to zero. Locking makes it real: afterward the policy can only be extended, never shortened or deleted, and no role, subscription Owner included, can step around it until the period runs out. Turn on blob versioning first, so an overwrite creates a new version instead of replacing the old one. Two scopes exist: container-level immutability protects the container as a whole, while version-level immutability (which needs versioning on) pins each individual blob version, a good fit for a purge-resistant append stream. A separate legal hold (az storage container legal-hold set --account-name auditsa --container-name activity --tags case-4471) adds an open-ended hold with no end date that you lift by hand when litigation is over.

# Versioning first, so an overwrite keeps the prior version instead of replacing it.
az storage account blob-service-properties update \
  --account-name auditsa -g sec-logging --enable-versioning true
 
# Time-based retention: 400 days, write-once. protected-append lets the export keep
# streaming (new blocks) while edits and deletes of existing data stay blocked.
az storage container immutability-policy create \
  --account-name auditsa --container-name activity \
  --period 400 --allow-protected-append-writes true

{
  "allowProtectedAppendWrites": true,
  "etag": "\"0x8DDA1F0A3C7B9E2\"",
  "immutabilityPeriodSinceCreationInDays": 400,
  "state": "Unlocked"
}

# Lock it. The etag from the create guards against a racing change. After this,
# not even a subscription Owner can shorten or delete the policy.
az storage container immutability-policy lock \
  --account-name auditsa --container-name activity \
  --if-match '"0x8DDA1F0A3C7B9E2"'

{
  "allowProtectedAppendWrites": true,
  "etag": "\"0x8DDA1F0B5522A10\"",
  "immutabilityPeriodSinceCreationInDays": 400,
  "state": "Locked"
}

Attackers blind the trail first

A careful burglar tapes over the lens before cracking the safe, not after. An attacker who understands your logging deletes a diagnostic setting, removes the old subscription log profile, or weakens the archive policy first, then does the loud work with nobody watching. The useful part: each of those blinding moves is itself a control-plane operation, so it lands in the Activity Log you are already collecting. The register records its own tampering, including the Caller (the Microsoft Entra ID identity, the current name for what used to be called Azure AD, that made the call) and the source address.

So you watch for it two ways. An activity log alert fires the instant anyone deletes a diagnostic setting, wired to an action group (the list of who gets paged). A Sentinel analytics rule runs the same logic in KQL (Kusto Query Language, the query language for Azure logs) for correlation and hunting. Watch the specific operations attackers use to go dark: Microsoft.Insights/diagnosticSettings/delete, Microsoft.Insights/logprofiles/delete (the legacy subscription-wide log profile), and immutability changes on the archive account.

ag=$(az monitor action-group show -g sec-logging -n soc-oncall --query id -o tsv)
 
# Page the SOC the moment anyone deletes a diagnostic setting in this subscription.
az monitor activity-log alert create --name diag-setting-removed \
  -g sec-logging --scope "/subscriptions/$sub" --action-group "$ag" \
  --condition category=Administrative and \
              operationName=Microsoft.Insights/diagnosticSettings/delete

{
  "condition": {
    "allOf": [
      { "equals": "Administrative", "field": "category" },
      { "equals": "Microsoft.Insights/diagnosticSettings/delete", "field": "operationName" }
    ]
  },
  "enabled": true,
  "location": "global",
  "name": "diag-setting-removed",
  "scopes": [ "/subscriptions/8f2c1e9a-..." ]
}

# The same meta-event as a Sentinel analytics rule / hunt, in KQL.
wsguid=$(az monitor log-analytics workspace show \
  -g sec-logging -n central-la --query customerId -o tsv)
 
az monitor log-analytics query --workspace "$wsguid" --timespan P7D \
  --analytics-query "AzureActivity
    | where OperationNameValue endswith 'DIAGNOSTICSETTINGS/DELETE'
    | where ActivityStatusValue == 'Success'
    | project TimeGenerated, Caller, CallerIpAddress, _ResourceId" -o table

TimeGenerated         Caller                CallerIpAddress  _ResourceId
--------------------  --------------------  ---------------  -----------------------------------
2026-07-14T09:41:12Z  [email protected]  34.221.14.9      /subscriptions/8f.../vaults/app-kv

Expect benign noise. Pipelines and Terraform runs delete and recreate diagnostic settings all the time. Tune on the Caller (your deployment service principal at 2 a.m., or a human you did not expect?) and on whether a delete was followed within a minute by a matching create. Page your security operations center (SOC) on-call for the pattern that looks like blinding, and let the routine churn feed a dashboard instead of a pager. The locked archive still matters even with alerts firing: an alert tells you the trail was cut, the immutable copy is the trail the attacker could not cut.

Nothing should be born blind

You want a checklist taped to the door that fills itself in. Manual diagnostic settings rot. Someone spins up a Key Vault on a Friday and it is invisible by default, and now your coverage has a hole nobody noticed. Azure Policy closes that gap with the deployIfNotExists effect: a rule that, when it finds a resource missing the required setting, deploys the setting for you. Because it has to create a resource, the assignment needs a managed identity (an Azure-run service account the policy acts as) with rights to do the deploy, and a one-time remediation task backfills everything that already existed before you assigned the policy.

Assign it at a management group (a container that sits above subscriptions) so the rule covers every current and future subscription under it, not one at a time. Then keep the workspace and the archive in a dedicated logging subscription with almost no human access. Compromising a workload then never hands the attacker their own audit record to edit.

One catch the portal hides for you: this built-in declares two roles its identity needs, Monitoring Contributor and Log Analytics Contributor. The --role flag on the assignment grants only one, so you add the second yourself, or remediation deployments fail later with a permissions error that looks nothing like a role problem.

# Find the built-in by display name; never hard-code the GUID.
def=$(az policy definition list --query \
  "[?policyType=='BuiltIn' && displayName=='Deploy Diagnostic Settings for Key Vault to Log Analytics workspace'].name" -o tsv)
echo "$def"   # bef3f64c-5290-43b7-85b0-9b254eef4c47
 
mg="/providers/Microsoft.Management/managementGroups/contoso-root"
 
# Assign at the management group. deployIfNotExists must create a setting,
# so it needs an identity with rights across every child subscription.
az policy assignment create --name kv-diag-enforce --policy "$def" \
  --scope "$mg" --location eastus \
  --mi-system-assigned --role "Monitoring Contributor" --identity-scope "$mg" \
  --params "{\"logAnalytics\":{\"value\":\"$wsid\"}}"
 
# --role granted one role. Add the second the definition also requires.
pid=$(az policy assignment show --name kv-diag-enforce --scope "$mg" \
  --query identity.principalId -o tsv)
az role assignment create --assignee-object-id "$pid" \
  --assignee-principal-type ServicePrincipal \
  --role "Log Analytics Contributor" --scope "$mg"

{
  "identity": {
    "principalId": "b1c2d3e4-5f6a-7b8c-9d0e-1a2b3c4d5e6f",
    "tenantId": "9a8b7c6d-...",
    "type": "SystemAssigned"
  },
  "name": "kv-diag-enforce",
  "scope": "/providers/Microsoft.Management/managementGroups/contoso-root"
}

# Backfill vaults that already exist non-compliant, at the same MG scope.
az policy remediation create --name kv-diag-backfill \
  --management-group contoso-root --policy-assignment kv-diag-enforce
 
# Remediation runs async, in bounded batches. Check how it finished:
az policy remediation show --name kv-diag-backfill \
  --management-group contoso-root \
  --query "{state:provisioningState, deployed:deploymentStatus}"

{
  "name": "kv-diag-backfill",
  "provisioningState": "Accepted",
  "resourceDiscoveryMode": "ExistingNonCompliant"
}
 
{
  "deployed": {
    "failedDeployments": 0,
    "successfulDeployments": 7,
    "totalDeployments": 7
  },
  "state": "Succeeded"
}

You leaned on one Azure Policy effect, deployIfNotExists, to keep logging from rotting across the estate. There are others (audit, deny, modify, and append), each with its own evaluation timing, remediation model, and failure modes. The next lesson, Azure Policy & effects, takes them apart so you can enforce far more than diagnostic settings, from allowed regions to required tags to blocked public IP addresses, with the same guardrail muscle you just built.

Try this

Run echo "$def" # bef3f64c-5290-43b7-85b0-9b254eef4c47 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: locking is a one-way door. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related