Centralized audit logging & immutable trails

Org trails, a separate logging account, and tamper-evidence.

A courtroom throws out evidence the moment the chain of custody breaks. The sealed bag, the signed intake sheet, the locked evidence room the investigating officer cannot quietly walk into alone: all of it exists so nobody can later claim the evidence was swapped or edited. Your cloud audit logs are exactly that kind of evidence. If the attacker who breaks into an account can also reach in and rewrite that account's own record of what happened, you do not have a record. You have whatever story the attacker decided to leave behind. Centralized, immutable audit logging is the plumbing that keeps your logs admissible: captured in every account, shipped somewhere almost nobody can touch, and sealed so that even a root-level administrator cannot go back and edit history.

What a trustworthy trail has to guarantee

An audit log answers a few questions about every action on your cloud: who called which API (Application Programming Interface, the control channel your tools and the attacker both talk through), on what resource, from where, and whether it was allowed. AWS (Amazon Web Services) splits these into management events (someone changed a setting) and data events (someone read or wrote an object). GCP (Google Cloud Platform) splits them into Admin Activity, Data Access, System Event and Policy Denied logs. Azure exposes Administrative, Security and Policy categories in its Activity Log, plus per-resource diagnostic logs. Different names, same recording.

A trail you can lean on during an incident has four separate properties, and you need all four. Completeness: every account, project and subscription, in every region, streamed to one place, with no way to opt out. Isolation: that place is a separate security boundary the workload accounts cannot administer, so owning a workload does not mean owning its logs. Immutability: the storage is WORM (Write-Once-Read-Many, meaning once an object lands it cannot be overwritten or deleted until its retention clock runs out). Tamper-evidence: cryptographic digests (short mathematical fingerprints of the data) let you prove later that nothing was altered or quietly dropped. Miss one and a patient attacker walks through the gap. Retention without immutability is a delete waiting to happen. Immutability inside the same account the attacker owns is a padlock hanging on a hook right next to its own key.

Completeness: one sink the accounts cannot switch off

A big office building runs every door camera into one guarded room in the basement, and no single tenant can unplug the feed on their own floor. Your logs want that same shape. The first move is a single collection point per cloud, wired so no individual account gets to decide whether it is watched. On AWS you create an organization trail from the management account. One trail, and it captures every member account across every region and delivers to a single S3 (Simple Storage Service) bucket that lives in a dedicated log-archive account. On GCP you create an aggregated log sink at the organization node with the --include-children flag, which funnels every child project's audit logs to one destination. GCP hands you an advantage here: Admin Activity logs are always on and cannot be turned off. Azure has no single tenant-wide switch, so you push the Activity Log out of each subscription into a diagnostic setting, and you make that mandatory by applying it through Azure Policy at the management-group root instead of trusting each team to remember. Same idea three times. Only the spelling changes.

# One org-wide, multi-region trail, delivering into the log-archive account.
$ aws cloudtrail create-trail --name org-trail \
    --s3-bucket-name acme-org-audit \
    --is-organization-trail --is-multi-region-trail \
    --enable-log-file-validation \
    --kms-key-id alias/cloudtrail-archive

{
    "Name": "org-trail",
    "S3BucketName": "acme-org-audit",
    "IncludeGlobalServiceEvents": true,
    "IsMultiRegionTrail": true,
    "TrailARN": "arn:aws:cloudtrail:us-east-1:911122223333:trail/org-trail",
    "LogFileValidationEnabled": true,
    "KmsKeyId": "arn:aws:kms:us-east-1:911122223333:key/2f1a8b0c-4d3e-9f77-a1b2-c3d4e5f60718",
    "IsOrganizationTrail": true
}

# A created trail is not yet a logging trail. Turn it on, then confirm delivery.
$ aws cloudtrail start-logging --name org-trail
$ aws cloudtrail get-trail-status --name org-trail

{
    "IsLogging": true,
    "LatestDeliveryTime": "2026-07-22T10:15:03.482000+00:00",
    "StartLoggingTime": "2026-07-20T09:12:44.198000+00:00",
    "LatestDeliveryAttemptTime": "2026-07-22T10:15:03Z",
    "LatestDeliveryAttemptSucceeded": "2026-07-22T10:15:03Z"
}

# GCP: route every child project's audit logs to a bucket in the log-archive project.
$ gcloud logging sinks create org-audit-sink \
    storage.googleapis.com/acme-org-audit \
    --organization=123456789012 \
    --include-children \
    --log-filter='logName:"cloudaudit.googleapis.com"'

Created [https://logging.googleapis.com/v2/organizations/123456789012/sinks/org-audit-sink].
Please remember to grant `serviceAccount:[email protected]`
the Storage Object Creator role on the bucket.

# Azure: ship the subscription's Activity Log to the archive storage account.
# Note the subcommand: 'subscription create', not the resource-level 'create'.
$ az monitor diagnostic-settings subscription create \
    --name activity-to-archive \
    --location eastus \
    --storage-account "/subscriptions/00000000-0000-0000-0000-000000000000/resourceGroups/logging/providers/Microsoft.Storage/storageAccounts/acmeorgaudit" \
    --logs '[{"category":"Administrative","enabled":true},
             {"category":"Security","enabled":true},
             {"category":"Policy","enabled":true}]'

{
  "id": "/subscriptions/00000000-0000-0000-0000-000000000000/providers/microsoft.insights/diagnosticSettings/activity-to-archive",
  "location": "eastus",
  "logs": [
    {"category": "Administrative", "enabled": true},
    {"category": "Security", "enabled": true},
    {"category": "Policy", "enabled": true}
  ],
  "name": "activity-to-archive",
  "storageAccountId": "/subscriptions/00000000-.../resourceGroups/logging/providers/Microsoft.Storage/storageAccounts/acmeorgaudit",
  "type": "Microsoft.Insights/diagnosticSettings"
}

Now decide how much detail to record, because completeness does not mean recording every byte at any price. Management events are cheap and low-volume, so capture them everywhere, always. Data events (the S3 GetObject and PutObject calls, Lambda invokes, GCP Data Access logs, Azure per-resource diagnostics) are a different animal: they can multiply your log volume ten- to a hundred-fold, and on AWS they are billed per event once you pass the free management tier. Switch them on across a large organization and a modest bill turns into a five-figure monthly one and terabytes a day. The pattern that survives both an audit and a budget review is selective: turn data events on only for the stores that matter (the crown-jewel dataset, the secrets bucket), send the loud high-volume streams to a cheaper queryable store like CloudTrail Lake, BigQuery or Log Analytics, and keep the raw sealed copy in cold locked object storage. One operational catch to remember: a GCP sink writes as its own service account, and until you grant that identity the roles/storage.objectCreator role gcloud just reminded you about, it delivers nothing, silently.

Isolation: a logging account almost nobody can log into

Think of the log-archive account as the bank's vault room, not the teller's cash drawer. Tellers move money all day, but the vault is a different room on a different key system, and the people handling day-to-day cash cannot walk into it alone. The workload accounts are the tellers. The log-archive account is the vault. It runs no applications, has no human daily-driver logins, and its whole job is to hold the audit bucket. Because it is a separate account, compromising production does not hand the attacker the logs: different trust boundary, different keys. But isolation only holds if the attacker cannot also reach up to the management account and switch the trail off from above. That is one thing a Service Control Policy is for.

An SCP (Service Control Policy) is an org-wide guardrail that caps what any principal in a member account may do, and it applies even to that account's root user and its administrators. Attach one at the organization root and the log-archive account, which is itself a member account, cannot delete its own audit bucket, rewrite the bucket policy, or weaken the Object Lock configuration, even from root. A compromised admin there can still type the command. It just returns AccessDenied, because nothing inside the account can grant back what the SCP took away.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyTamperingWithTrails",
      "Effect": "Deny",
      "Action": [
        "cloudtrail:StopLogging",
        "cloudtrail:DeleteTrail",
        "cloudtrail:UpdateTrail",
        "cloudtrail:PutEventSelectors"
      ],
      "Resource": "arn:aws:cloudtrail:*:*:trail/org-trail"
    },
    {
      "Sid": "ProtectTheArchiveBucket",
      "Effect": "Deny",
      "Action": [
        "s3:DeleteBucket",
        "s3:PutBucketPolicy",
        "s3:PutObjectLockConfiguration"
      ],
      "Resource": "arn:aws:s3:::acme-org-audit"
    }
  ]
}

Know the one gap this does not close: an SCP does not restrict the management account itself, and the org trail is managed from there. So keep the management account nearly empty, with almost nobody able to sign in, and let immutability be the backstop for the case where it is compromised anyway. GCP and Azure reach the same place by different routes. On GCP the aggregated sink lives on the organization node, so a child project cannot alter or redirect it, because the project does not own the object; you harden that by pinning logging.sinks.update and logging.sinks.delete to a small org-level break-glass group instead of any project role. On Azure you put a management lock on the storage account so an owner cannot delete it, on purpose or by accident, while the lock is on.

# Azure: a CanNotDelete lock on the archive storage account.
$ az lock create --name protect-audit-archive \
    --lock-type CanNotDelete \
    --resource-group logging \
    --resource acmeorgaudit \
    --resource-type Microsoft.Storage/storageAccounts

{
  "id": ".../storageAccounts/acmeorgaudit/providers/Microsoft.Authorization/locks/protect-audit-archive",
  "level": "CanNotDelete",
  "name": "protect-audit-archive",
  "notes": null,
  "type": "Microsoft.Authorization/locks"
}

Be honest about what a management lock buys you: it is a speed bump. A privileged attacker who holds the permission to remove the lock can remove it first, then delete the account. It stops mistakes and casual tampering, which is most of what goes wrong. For the attacker with full control, you need the storage itself to say no, and that is the next property.

Immutability: WORM even against your own root

Isolation keeps the workload accounts out. Immutability handles the harder case: the log-archive account itself is compromised, or the org administrator has gone rogue, and you need the stored objects to refuse deletion anyway. Wet concrete takes any shape you press into it. Once it cures, it holds that shape whether you still want it or not. Cured concrete is what you want your log storage to be. Each cloud ships a true WORM primitive that answers to the storage service, not to IAM (Identity and Access Management), which means no policy edit and no privileged principal can talk it out of enforcing retention. On AWS it is S3 Object Lock in COMPLIANCE mode: once set, an object cannot be deleted, nor its retention shortened, by anyone including the account root, until the clock expires. There is a weaker sibling, GOVERNANCE mode, that looks identical until you notice any principal holding s3:BypassGovernanceRetention can override it, so a compromised admin who can grant themselves that permission walks right through. COMPLIANCE is the one that survives a full takeover. On GCP it is a bucket retention policy that you then lock, after which it can never be shortened or removed. On Azure it is a time-based immutability policy on the container that, once locked, guarantees write-once semantics for the whole window.

# AWS: Object Lock must be enabled AT bucket creation (it also forces versioning on).
$ aws s3api create-bucket --bucket acme-org-audit \
    --region us-east-1 --object-lock-enabled-for-bucket
 
# Default every new object to 7 years of COMPLIANCE-mode retention.
$ aws s3api put-object-lock-configuration --bucket acme-org-audit \
    --object-lock-configuration \
    '{"ObjectLockEnabled":"Enabled",
      "Rule":{"DefaultRetention":{"Mode":"COMPLIANCE","Years":7}}}'
 
# Verify. This is the only way to know it actually took.
$ aws s3api get-object-lock-configuration --bucket acme-org-audit

{
    "Location": "/acme-org-audit"
}
 
{
    "ObjectLockConfiguration": {
        "ObjectLockEnabled": "Enabled",
        "Rule": {
            "DefaultRetention": {
                "Mode": "COMPLIANCE",
                "Years": 7
            }
        }
    }
}

# GCP: set a 7-year (2555-day) retention, then LOCK it. The lock is one-way.
$ gcloud storage buckets update gs://acme-org-audit --retention-period=P2555D
$ gcloud storage buckets update gs://acme-org-audit --lock-retention-period
 
# Confirm the lock actually latched.
$ gcloud storage buckets describe gs://acme-org-audit \
    --format="yaml(retention_policy)"

retention_policy:
  effectiveTime: '2026-07-22T10:26:14.512000+00:00'
  isLocked: true
  retentionPeriod: '220752000'

# Azure: create a time-based immutability policy on the container, then LOCK it.
# --if-match takes the etag the 'create' step returned.
$ az storage container immutability-policy create \
    --account-name acmeorgaudit \
    --container-name insights-activity-logs \
    --period 2555
 
$ az storage container immutability-policy lock \
    --account-name acmeorgaudit \
    --container-name insights-activity-logs \
    --if-match '"0x8DCE3A1B2C3D4E5"'

{
  "allowProtectedAppendWrites": null,
  "etag": "\"0x8DCE3A1B2C3D4E5\"",
  "immutabilityPeriodSinceCreationInDays": 2555,
  "name": "default",
  "state": "Unlocked"
}
 
{
  "allowProtectedAppendWrites": null,
  "etag": "\"0x8DCE3A1F9B21C40\"",
  "immutabilityPeriodSinceCreationInDays": 2555,
  "name": "default",
  "state": "Locked"
}

Tamper-evidence: proving nothing was dropped

Immutability protects the objects that made it into the vault. Tamper-evidence answers a subtler question: how do you know the set is complete, that no file was quietly removed or edited before the lock took hold, or in a window where someone had write access to the bucket? The foil seal under a medicine bottle cap cannot stop someone prying it off. What it does is make a tampered bottle impossible to pass off as untouched. AWS gives you that same foil seal for logs, called log-file validation. With it enabled, CloudTrail writes an extra digest file into the same bucket every hour. Each digest lists the log files delivered for that period together with their SHA-256 hashes (a hash is a fixed-length fingerprint where changing a single byte of the input changes the output completely), and the digest itself is signed with SHA-256-with-RSA using a private key AWS holds. Delete a log file, edit one event, or drop an hour, and the arithmetic stops matching. You run one command to check.

$ aws cloudtrail validate-logs \
    --trail-arn arn:aws:cloudtrail:us-east-1:911122223333:trail/org-trail \
    --start-time 2026-07-21T00:00:00Z

Validating log files for trail arn:aws:cloudtrail:us-east-1:911122223333:trail/org-trail between 2026-07-21T00:00:00Z and 2026-07-22T10:30:00Z
 
Results requested for 2026-07-21T00:00:00Z to 2026-07-22T10:30:00Z
Results found for 2026-07-21T00:04:12Z to 2026-07-22T10:04:55Z:
 
34/34 digest files valid
1982/1982 log files valid

Tamper with the bucket and the same command turns loud. An edited object prints a line like Log file s3://acme-org-audit/AWSLogs/.../….json.gz INVALID: hash value doesn't match, and a touched digest prints Digest file … INVALID: signature verification failed, with the valid counts dropping to match. GCP and Azure do not hand you an equivalent per-file signature, so you lean on structure instead. On GCP the integrity comes from Google-managed storage plus the fact we already used: the org-level aggregated sink is owned by no project, so a compromised project cannot reroute or thin it. On Azure the immutable-blob guarantee is itself the evidence, because a locked policy means the blobs you are reading could not have been altered in place. In all three, the raw copy stays in cold locked object storage while anything you want to search fast is shipped to a separate analytical store, so a query-engine compromise never touches the sealed original.

Watch the watchers

There is one move a competent attacker makes before almost anything else: they blind the camera. So the highest-value, lowest-noise detection you can build is an alarm on anyone touching the logging setup at all. Real changes to a trail or a sink are rare and planned, which means nearly every unplanned one is either a mistake or an intrusion. Alert on CloudTrail StopLogging and DeleteTrail. Alert on S3 PutBucketPolicy and any change to the Object Lock configuration. Alert on GCP DeleteSink and on setIamPolicy against the logging project. Alert on Azure diagnostic-setting deletion and edits to an immutability policy. And route those alarms out of the account they fire in, so the same compromise cannot swallow the warning it triggers. These sealed trails are also the raw feed for the managed detectors you wire up next: GuardDuty on AWS, Security Command Center on GCP, and Defender on Azure all read exactly this stream, and a finding is only ever as trustworthy as the log integrity sitting underneath it.

Try this

Run aws cloudtrail start-logging --name org-trail on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a locked retention is a bill you cannot cancel, and a lost key is a shredder. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related