>_SecOpsLog_
CoursesGCP securityDetection as code & response

Detection as code & response

Log metrics, Pub/Sub pipelines, auto-response.

Advanced35 min · lesson 12 of 15

A finding that nobody reads until Monday isn't a control. It's a to-do item nobody bothered to write down. Google Cloud (GCP for short) ships a built-in risk tracker called Security Command Center, or SCC, and it can flag the exact moment someone hands themselves the Owner role on your production project. Owner means full control, the keys to everything. But if that flag just sits in a browser tab an on-call engineer glances at twice a day, the attacker gets a head start measured in hours. Detection as code is how you close that gap.

Think about a smoke alarm. The cheap kind screams and hopes somebody's home to hear it. The wired kind is bolted straight to the sprinklers: smoke trips the sensor, water comes down, nobody has to be awake for it to work. This lesson is about wiring your findings to the sprinklers instead of to a guard who has to notice the noise first.

Detections you keep in git

Half of your detections you don't write at all. Google runs a service inside SCC called Event Threat Detection, or ETD. It works like a security guard whose only job is watching the building's door logs scroll past, checking every badge swipe against a list of known-bad moves. ETD reads your audit logs (the running record of who did what), plus your network and DNS traffic logs, the moment each line lands. It checks every entry against Google's threat intelligence and a library of rules, then files a finding when something matches. A service account (a login used by software, not a person) suddenly connecting through Tor, the anonymity network people use to hide where they're really coming from. A brute-force attacker who guessed passwords until an SSH login worked and got into a server, SSH being the normal way admins log in remotely. An account that just handed itself admin rights. You never write those rules and you never run a scanner, but ETD is not a free baseline either: it ships with SCC's Premium and Enterprise tiers, so on the Standard tier none of the findings the rest of this lesson reacts to will ever arrive. Once your organization is on a paid tier, ETD reads every project underneath it by default, and you can switch an individual detector off when one turns out to be noisy in your environment.

The other half is yours to write, kept in version control and reviewed like any other code. A log-based metric is just a counter: it ticks up by one every time a log entry matches a filter you define. The trick that actually catches attackers is counting a permitted action, something the system is happy to allow, and watching for it happening when it shouldn't. Granting the Owner role is allowed. No permission check will ever block it. But a counter that fires on the very first Owner grant turns a normal-looking action into an alarm.

terminal
$ gcloud logging metrics create owner-grants \
    --description="Owner role granted via SetIamPolicy" \
    --log-filter='protoPayload.methodName="SetIamPolicy"
      AND protoPayload.serviceName="cloudresourcemanager.googleapis.com"
      AND protoPayload.serviceData.policyDelta.bindingDeltas.role="roles/owner"
      AND protoPayload.serviceData.policyDelta.bindingDeltas.action="ADD"'
Created [owner-grants].

That filter is looser than it looks. bindingDeltas is a list, and each line of the filter is tested against the whole list rather than one entry at a time, so a policy change that strips Owner off one person while adding a lesser role to another also counts. Scoping it to cloudresourcemanager.googleapis.com keeps it to project-level policy edits, but the responder still has to open the log entry and read the delta before deciding anything.

A counter does nothing on its own. An alerting policy is the thing that watches it: you point the policy at the metric, set the threshold to anything above zero, and attach a notification channel so a real Owner grant pages a human instead of quietly decorating a dashboard. Match the condition on the metric type and nothing else. A log-based metric inherits the resource type of the log entry that fed it, so a condition pinned to a resource type you guessed wrong is a policy that never fires and never complains. Keep the policy in the same repo as the metric. Then the whole detection lives in one file that a teammate can review.

owner-grants-alert.yaml
displayName: Owner role granted
combiner: OR
conditions:
  - displayName: owner-grants > 0
    conditionThreshold:
      filter: >
        metric.type="logging.googleapis.com/user/owner-grants"
      comparison: COMPARISON_GT
      thresholdValue: 0
      duration: 0s
      aggregations:
        - alignmentPeriod: 300s
          perSeriesAligner: ALIGN_SUM
notificationChannels:
  - projects/prod-sec/notificationChannels/9876543210
terminal
$ gcloud alpha monitoring policies create \
    --policy-from-file=owner-grants-alert.yaml
Created alert policy [projects/prod-sec/alertPolicies/1357924680].

Wire the alarm to the sprinklers

So far a human still gets paged. To take the person out of the fast path, SCC can broadcast every finding into a Pub/Sub topic. Pub/Sub is a message queue, which you can think of as a building PA system: SCC makes the announcement once, and anything subscribed to that channel hears it instantly. A notification config is the standing order you leave at the front desk. It says, in effect, every finding that matches this filter, publish it to this topic. One catch before you create it. SCC publishes under its own service account, that same kind of software identity, and it needs permission to post to the topic. Grant that first, or the create call comes back with a permission error instead of a working config.

terminal
$ gcloud pubsub topics create scc-findings
Created topic [projects/prod-sec/topics/scc-findings].
 
$ gcloud pubsub topics add-iam-policy-binding scc-findings \
    --member="serviceAccount:service-org-849376543210@gcp-sa-scc-notification.iam.gserviceaccount.com" \
    --role="roles/pubsub.publisher"
Updated IAM policy for topic [scc-findings].
 
$ gcloud scc notifications create high-sev-findings \
    --organization=849376543210 \
    --description="Active HIGH-severity findings" \
    --pubsub-topic=projects/prod-sec/topics/scc-findings \
    --filter='state="ACTIVE" AND severity="HIGH"'
Created notification config: organizations/849376543210/notificationConfigs/high-sev-findings.

A single notification config at the organization level covers every project underneath it. That's exactly what you want, because attackers rarely land in the one project you happened to be watching. The reach is also the danger. Any function that reacts to this stream gets a license to change IAM, the system that decides who can do what, across the entire org. That makes its own service account one of the most powerful identities you run. Give it a custom role holding the precise permissions it needs and nothing spare, and keep its code in the same reviewed repo as your detections.

The function that pulls the trigger

Now something has to listen and act on what it hears. A Cloud Function subscribed to the topic wakes up for every finding that gets published, reads it, and runs the fix. Maybe it strips off the rogue Owner grant. Maybe it disables a service account key that leaked. Maybe it quarantines a virtual machine by swapping the network tags that decide what that machine is allowed to talk to. There's one design rule you can't skip. Pub/Sub delivers at least once, which is a polite way of saying the same finding will sometimes show up twice. So your code has to be idempotent, a fancy word for safe to run again. Read the current state before you change anything. Then disabling an already-disabled key is a quiet no-op, not an error that pages the very person you were trying to let sleep.

terminal
$ gcloud functions deploy scc-auto-remediate \
    --gen2 --region=europe-west1 --runtime=python312 \
    --trigger-topic=scc-findings \
    --entry-point=remediate --source=./remediate \
    --run-service-account=remediator@prod-sec.iam.gserviceaccount.com
Preparing function...done.
Deploying function...done.
state: ACTIVE
serviceConfig:
  serviceAccountEmail: [email protected]
eventTrigger:
  eventType: google.cloud.pubsub.topic.v1.messagePublished
  pubsubTopic: projects/prod-sec/topics/scc-findings

Prove it actually fires

You do not want the first end-to-end run of this chain to happen during an incident. A real SCC message is just a chunk of structured text (the format is called JSON) with the finding tucked inside it. So you can rehearse the whole path yourself: publish a fake message shaped the same way, then read the function's logs to see what it did. Do this in a throwaway project first. You're about to run code whose entire purpose is to delete access.

terminal
$ gcloud pubsub topics publish scc-findings \
    --message='{"finding":{"category":"Persistence: IAM Anomalous Grant","severity":"HIGH","resourceName":"//cloudresourcemanager.googleapis.com/projects/shop-prod","access":{"principalEmail":"[email protected]"}}}'
messageIds:
- '11527819923123456'
 
$ gcloud functions logs read scc-auto-remediate --gen2 --region=europe-west1 --limit=5
LEVEL  NAME                EXECUTION_ID  TIME_UTC                 LOG
I      scc-auto-remediate  9xk2a7f1      2026-07-16 09:14:02.331  parsed finding Persistence: IAM Anomalous Grant on projects/shop-prod
I      scc-auto-remediate  9xk2a7f1      2026-07-16 09:14:03.108  removed binding roles/owner for user:[email protected]
I      scc-auto-remediate  9xk2a7f1      2026-07-16 09:14:03.902  finding remediated in 1.6s

The function above runs as a service account that lives inside Google Cloud, so Google hands it fresh, short-lived credentials on its own and there's no long-lived key sitting around to leak. The moment you want that same fix running from somewhere outside Google, say a GitHub Action or a security tool in your own datacenter, you hit a wall. You can't just mail it a service account key without recreating the exact leak you built all this to catch. Workload Identity Federation is how an outside caller gets in without a key, and that's what's next.

Your remediator is now attack surface
Auto-remediation is a loaded gun pointed at your own environment. A filter that's too broad, plus an action that's too aggressive, is how one misfire takes down production. Say you build a function that strips every public IAM binding it finds. It will cheerfully break the static-site bucket your marketing team made public on purpose. So scope it hard. Act only on HIGH severity findings. Keep an allowlist of known-good exceptions the function must leave alone. Run it in report-only mode first, where it logs what it would have done without touching anything, and read those logs before you let it act for real. Give yourself a stop button too: a flag the function reads on every run, so you can switch it off from the console at three in the morning instead of waiting for a deploy to land.
Finding to fix, no human in the loop
1ETD + your metrics
threat rules and custom counters raise findings
2finding in SCC
ACTIVE, HIGH severity, matches the filter
3notification config
publishes the finding to a Pub/Sub topic
4Cloud Function (gen2)
triggered per message, parses the JSON
5remediate + record
remove binding, disable key, log every step
The audit log captures both the finding and the fix, so the whole response is reconstructable after the fact.

Detection-as-code means the notification filter, the Cloud Function or Cloud Run service that triages, and the paging policy all review like application code. A pull request changes what HIGH means for your org. That sounds bureaucratic until the night someone edits a filter in the console and silently drops persistence findings for a week.

Start with a tiny catalog: anomalous Owner grants, public buckets, keys created despite policy, and sink deletions. Each item needs a severity, a destination, an owner, and a test. Expand only when those four stay healthy. Noise trains people to ignore the wall; a short sharp list trains them to move.

Try this

Prove the path on the pieces you just built. Describe the notification config to see the filter it is really running, add a second subscription on the topic so you can read the stream with your own eyes, publish one test finding, then check that both the subscription and the function saw it. The deploy already gave the function its own subscription, so yours gets a separate copy of every message rather than stealing it.

terminal
gcloud scc notifications describe high-sev-findings \
  --organization=849376543210
gcloud pubsub subscriptions create scc-findings-sub \
  --topic=scc-findings --project=prod-sec
gcloud pubsub topics publish scc-findings --project=prod-sec \
  --message='{"finding":{"category":"Persistence: IAM Anomalous Grant","severity":"HIGH","state":"ACTIVE"}}'
gcloud pubsub subscriptions pull scc-findings-sub \
  --auto-ack --limit=1 --project=prod-sec
gcloud functions logs read scc-auto-remediate --gen2 \
  --region=europe-west1 --project=prod-sec --limit=20
output
name: organizations/849376543210/notificationConfigs/high-sev-findings
description: Active HIGH-severity findings
pubsubTopic: projects/prod-sec/topics/scc-findings
serviceAccount: service-org-849376543210@gcp-sa-scc-notification.iam.gserviceaccount.com
streamingConfig:
  filter: state="ACTIVE" AND severity="HIGH"
 
Created subscription [projects/prod-sec/subscriptions/scc-findings-sub].
 
messageIds:
- '11527819923123457'
 
# pulled message (trimmed)
{"finding":{"category":"Persistence: IAM Anomalous Grant","severity":"HIGH","state":"ACTIVE"}}
 
# function log (trimmed)
I  scc-auto-remediate  9xk2a7f2  parsed finding Persistence: IAM Anomalous Grant on projects/shop-prod

Takeaway

Remember: a detection that only lives in a console is a sticky note. Keep filters and response functions in git, wire SCC to Pub/Sub, and prove the path with a controlled test finding before you trust it in an incident.

Next you will remove long-lived CI keys entirely with Workload Identity Federation, shrinking the credential theft class your detections keep paging on.

Quick check
01Working through Try this, you run gcloud pubsub subscriptions create scc-findings-sub --topic=scc-findings, publish the test finding, then pull it with --auto-ack --limit=1 and get the JSON back. A teammate says you have just stolen that message from the remediator. What actually happens on the topic?
Incorrect — That is how several workers sharing a single subscription behave. Separate subscriptions on the same topic each receive the full stream, which is what makes the extra one safe to add.
Incorrect — An ack only settles the message on the subscription you pulled from. The function's subscription tracks its own delivery state, so your ack cannot clear its copy.
Correct — The deploy created a subscription for scc-auto-remediate, and yours is a second one. You get to read the stream with your own eyes while the remediator keeps reacting to it.
Incorrect — Duplicate invocations come from at-least-once redelivery on the function's own subscription, not from adding another reader. Extra subscriptions add watchers, not extra runs of the function.
02Your owner-grants metric is live, and owner-grants-alert.yaml sets thresholdValue: 0 with comparison COMPARISON_GT over a 300s alignment period. A teammate wants to raise thresholdValue to 5 so the policy pages less often. What does that change cost you?
Correct — One person handing themselves Owner on production is the whole incident, so the condition has to trip on the first count, which is exactly what greater than zero does.
Incorrect — A log-based metric matches whatever your filter describes, allowed or denied. This filter matches successful SetIamPolicy calls that add roles/owner, and those succeed every time.
Incorrect — The threshold sits in the conditionThreshold block of the alerting policy YAML. The metric is only a counter and has no opinion about when a count deserves a page.
Incorrect — The counter reads audit log entries directly, so nothing on the SCC to Pub/Sub path feeds it. Duplicate deliveries on the topic cannot move this count either way.
03You built a Cloud Function that strips every public IAM binding it finds, and you are about to let it act for real across the org. Which step would show you, before any damage, that it is about to remove the public-read binding from the marketing team's deliberately public bucket?
Incorrect — Worth doing, and it proves the wiring from topic to function. But it only exercises the one finding you invented, so it says nothing about the real bindings sitting in your projects.
Correct — It runs against your real findings without touching anything, so the marketing bucket shows up as a would-have-removed line in the log instead of as an outage.
Incorrect — That caps the blast radius and you should do it, but changing bucket IAM is precisely a permission this function needs, so a tight role still lets this misfire through.
Incorrect — Less volume is not the same as knowing what it would do. The function still acts blind the first time it fires on a resource that is public on purpose.
Security Command CenterWorkload Identity Federation

Related

IAM mental modelAWS security for DevOps engineersMulti-account & org identity architectureAdvanced cloud securityThe IAM evaluation modelAWS security engineeringEntra ID, Conditional Access & PIMAzure securityPython for DevSecOps cheat sheetCheat sheetDevSecOps & supply-chain security interview questionsInterview prep