Audit logs & immutable trails

Aggregated sinks to a locked logging project.

The first thing a capable attacker does after breaking into your cloud usually isn't stealing data. They go looking for the cameras. If they can quietly stop the recording, or edit the tape after the fact, then everything they do next leaves no trace you can trust. So before any detection tooling, before Security Command Center, you need one thing in place: a complete record of who did what, kept somewhere the attacker can't reach and nobody can rewrite.

Google Cloud writes most of that record for you automatically. It's called Cloud Audit Logs, and the easiest way to hold it in your head is a logbook at a building's front desk. Every time someone badges through a door, a line gets written: which card, which door, what time. In GCP the doors are API calls, the individual requests your tools make to Google's services, and the badge is your identity. Four separate logbooks get kept, and two of them are the ones you actually have to make decisions about.

Four log streams, two you decide on

Admin Activity logs are always on. You can't switch them off and they cost nothing. They record every change to configuration or permissions: someone created a firewall rule, deleted a bucket, granted a teammate a new role. Think of it as the log of anyone changing the locks or handing out keys. This is the stream that catches most attacker moves, because to do real damage they nearly always have to change something first.

Data Access logs are the other one, and they're off by default (BigQuery is the exception; it records data access on its own). These capture reads and writes to the data itself: who opened the object, who ran the query, who downloaded the file. Admin Activity tells you someone was handed a key to the filing cabinet. Data Access tells you they actually opened it and read a folder. It's off by default for a good reason. On a busy project it's a firehose, and you pay per gigabyte ingested, so you turn it on deliberately where the sensitive data lives rather than everywhere at once.

The last two matter less day to day. System Event records changes Google itself makes, like live-migrating your virtual machine (a running server) off a host that needs maintenance. Policy Denied records a request that a security control turned away, such as a call blocked by a VPC Service Controls perimeter, the invisible fence around your projects you'll meet later in this course. Turning Data Access logging on is a permissions change, done through IAM (Identity and Access Management), Google's system for deciding who can do what. You set it once at the organization node, the top of the resource tree, and every project below inherits it.

# Pull the current org policy first, add the auditConfigs block, then re-apply.
$ gcloud organizations get-iam-policy 123456789012 --format=yaml > org-policy.yaml
$ cat org-policy.yaml
auditConfigs:
- service: allServices
  auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_READ
  - logType: DATA_WRITE
bindings:
- members:
  - user:[email protected]
  role: roles/resourcemanager.organizationAdmin
etag: BwYh2n0aQ1s=
version: 1
 
$ gcloud organizations set-iam-policy 123456789012 org-policy.yaml
Updated IAM policy for organization [123456789012].
auditConfigs:
- auditLogConfigs:
  - logType: ADMIN_READ
  - logType: DATA_READ
  - logType: DATA_WRITE
  service: allServices
bindings:
- members:
  - user:[email protected]
  role: roles/resourcemanager.organizationAdmin
etag: BwYh2n0bR4k=
version: 1

That auditConfigs block is what flips Data Access recording from off to on across every service, org-wide, for both reads and writes. One caution before you run it: set-iam-policy replaces the entire policy, so always pull the current one with get-iam-policy first, edit it, and keep the etag. Hand-write a partial file and you'll wipe every existing role binding on the organization in one command.

Get the footage out of the building

By default, every project's logs sit inside that same project, in a log bucket named _Default. That's a problem the moment the project is compromised. It's the CCTV recorder sitting on a desk in the room that's being robbed. Whoever owns the project owns the recording and can delete it. The fix is to stream a copy somewhere else the instant each log is written.

A sink is the rule that does the streaming. Think of a mail-forwarding rule: any log matching this filter, send a copy to this address, forever. Create the sink at the organization node with --include-children and it applies to every project underneath, the ones you have now and the ones you'll make next year, as one aggregated pipe. Point it at a storage bucket that lives in a separate logging project, one that maybe five people can even see. Now a workload compromise in production can't reach its own audit trail, because the trail already left the building.

One detail trips almost everyone up. The sink writes using its own dedicated identity, a service account created just for it, like a courier who only holds a key to the drop box. If you don't grant that identity permission to write to the destination bucket, the sink is created 'successfully' and then silently drops every log. You find out during an incident, when the bucket you were counting on is empty.

$ gcloud storage buckets create gs://acme-audit-trail-2026 \
    --location=us --uniform-bucket-level-access --public-access-prevention
Creating gs://acme-audit-trail-2026/...
 
$ gcloud logging sinks create org-audit \
    storage.googleapis.com/acme-audit-trail-2026 \
    --organization=123456789012 --include-children \
    --log-filter='logName:"cloudaudit.googleapis.com"'
Created [https://logging.googleapis.com/v2/organizations/123456789012/sinks/org-audit].
Please remember to grant `serviceAccount:[email protected]`
the Storage Object Creator role on the bucket.
 
$ gcloud storage buckets add-iam-policy-binding gs://acme-audit-trail-2026 \
    --member='serviceAccount:[email protected]' \
    --role='roles/storage.objectCreator'
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/storage.objectCreator
etag: CAE=

A recording nobody can erase

Getting the logs off-site stops a project-level attacker from deleting them. It does nothing to stop someone with rights in the logging project itself. For that you make the storage refuse. This is Bucket Lock, and it gives you exactly the CCTV property you want: a recording nobody can edit or delete before its retention date, not the security team, not even Google support.

You set a retention period on the bucket, say seven years to match a compliance rule, and objects in it can't be deleted or overwritten until they reach that age. Then you lock the policy. Locking is the part that matters. An unlocked retention policy can be shortened by anyone who can edit the bucket, which is no protection at all. A locked one is permanent: you can raise the number later, never lower it, and never remove it. Write once, read many. That's the step that turns logs into evidence.

$ gsutil retention set 7y gs://acme-audit-trail-2026
Setting retention policy on gs://acme-audit-trail-2026/...
 
$ gsutil retention lock gs://acme-audit-trail-2026
This will PERMANENTLY set the Retention Policy on gs://acme-audit-trail-2026 to:
  Duration: 7 Year(s)
This setting cannot be reverted!  Continue? [y/N] y
Locking retention policy on gs://acme-audit-trail-2026/...
 
$ gsutil retention get gs://acme-audit-trail-2026
  Retention Policy (LOCKED):
    Duration: 7 Year(s)
    Effective Time: Thu, 16 Jul 2026 00:00:00 GMT

Watch the watchers

Here's the move that pays for itself. Deleting a sink, flipping Data Access logging back off, trying to shorten retention: every one of those is a configuration change, so every one lands in Admin Activity, the stream nobody can turn off. An attacker reaching for the light switch trips a sensor on the way. So build one alert on top of everything else and page a human the instant the logging setup is touched. 'Who changed the audit configuration' is about the highest-signal detection you can own, because almost nobody has a legitimate reason to.

You wire it up in two pieces. A log-based metric counts the specific events, a sink being deleted or an audit config being changed. An alerting policy watches that metric and pages on-call the moment the count goes above zero.

$ gcloud logging metrics create audit-tamper \
    --description='Sink deleted or Data Access audit config changed' \
    --log-filter='protoPayload.methodName="google.logging.v2.ConfigServiceV2.DeleteSink"
      OR (protoPayload.methodName="SetIamPolicy"
          AND protoPayload.serviceData.policyDelta.auditConfigDeltas:*)'
Created [https://logging.googleapis.com/v2/projects/acme-logging-prod/metrics/audit-tamper].
 
$ cat audit-tamper.yaml
displayName: "Audit trail tampering"
combiner: OR
conditions:
- displayName: "sink deleted or Data Access logging disabled"
  conditionThreshold:
    filter: 'metric.type="logging.googleapis.com/user/audit-tamper" resource.type="global"'
    comparison: COMPARISON_GT
    thresholdValue: 0
    duration: 0s
    aggregations:
    - alignmentPeriod: 60s
      perSeriesAligner: ALIGN_COUNT
notificationChannels:
- projects/acme-logging-prod/notificationChannels/4407512900112
alertStrategy:
  autoClose: 604800s
 
$ gcloud alpha monitoring policies create --policy-from-file=audit-tamper.yaml
Created alert policy [projects/acme-logging-prod/alertPolicies/1587340928465012].

This trail is raw material, not the detection itself. In the next lesson, Security Command Center sits on top of it. Its Event Threat Detection engine reads these audit logs continuously and turns patterns into findings: a service account key suddenly used from a new country, a sink deleted at 3 a.m., a role granted to an outside email address. None of that works if the logs are incomplete or forgeable. The centralized, locked trail you just built is what makes every detection after it worth trusting.

Admin Activity logs are on by default; Data Access logs are the ones people forget, and they are the ones that tell you who read a bucket object at 2 a.m. Enable them on the services that hold regulated data, accept the volume cost, and filter aggressively at the sink if you must — but do not skip the stream to save a line item you will regret during forensics.

Routing is the real control. An org-level sink with includeChildren ships every project’s audit trail into a dedicated logging project or bucket the workload Owners cannot edit. Pair that with a locked retention policy and CMEK you control. Watch the watchers by alerting on sink deletion, IAM changes on the log bucket, and anyone granting themselves logging.admin on the destination project.

Try this

Confirm data-access logs on a sensitive bucket, describe your org sink, and show that the destination bucket denies even Owners a delete.

gcloud projects get-iam-policy payments-prod --format="yaml(auditConfigs)" 
gcloud logging sinks describe org-audit-sink --organization=ORGANIZATION_ID
gsutil iam get gs://acme-org-audit-logs
gcloud logging read \
  'protoPayload.serviceName="storage.googleapis.com" AND protoPayload.methodName="storage.objects.get"' \
  --project=payments-prod --limit=2 --format="table(timestamp,protoPayload.authenticationInfo.principalEmail)"

auditConfigs:
- auditLogConfigs:
  - logType: DATA_READ
  - logType: DATA_WRITE
  service: storage.googleapis.com
 
name: org-audit-sink
destination: storage.googleapis.com/acme-org-audit-logs
filter: 'logName:"cloudaudit.googleapis.com"'
includeChildren: true
 
# destination bucket IAM (trimmed): no allUsers; no project Owner delete
# retention policy locked; object versioning on

Takeaway

Remember: turn on the log streams that matter (especially data access on sensitive services), sink them outside the projects that produce them, and lock retention so an attacker with Owner still cannot erase the tape.

Next you will hang Security Command Center on top of those logs so findings show up on one wall instead of twelve project consoles.

Related