Incident response & forensics

Contain, snapshot, revoke, hunt persistence.

A crypto-miner is pinning one of your web servers at 100% CPU, and it's 3 a.m. That server is a VM, a virtual machine: a computer that exists only as software running inside Google's data center. Someone got in. From this second on, every command you type either saves evidence or shreds it, so the order you work in matters more than how fast you move.

Cloud incident response is a footrace between the attacker and you. The awkward part: everything here happens through API calls, one program telling Google's systems what to do over the network, so the intruder moves at machine speed, not human speed. What wins is a runbook you've rehearsed cold, one that pins the box down without wiping the trail that tells you what happened. Treat it like a crime scene. Seal off the room, photograph everything before you touch it, then change the locks. Get that order wrong and you've destroyed your own case.

The finding that woke you probably didn't come from a human staring at dashboards. It came from Event Threat Detection (ETD), a service that constantly reads through your logs, your record of admin actions and your network traffic, watching for known-bad patterns: a machine phoning a known crypto-mining pool, a login from a Tor exit node, someone hammering an SSH port to guess the password. The 100% CPU is just the symptom you'd notice. ETD is what actually recognized the attack and filed a finding in Security Command Center, Google's console that gathers security alerts from across your whole account. That's your smoke alarm. The runbook below is what you do once it goes off.

Seal the room first

Your instinct will be to reach for delete. Don't. A deleted VM answers no questions. What you want instead is a VM that can't talk to anyone: nothing can reach it, and it can't reach out to the attacker's command server (the machine that sends orders to the malware, usually shortened to C2, for command-and-control). The clean way to do this in Google Cloud is a firewall rule inside your VPC (Virtual Private Cloud, your own private network in Google's data center) that denies all traffic and is aimed at a network tag. Then you stick that tag on the sick machine. Think of the tag as a quarantine wristband and the firewall as a bouncer who turns away anyone wearing it.

Block the outbound traffic especially. That's the leash on the C2 channel, the thing that stops the malware from calling home for orders. Block inbound too, so the attacker can't just reconnect. But leave the machine switched on. Power it off and you lose whatever's in memory, and memory is often where the most useful evidence lives. If you need a way in while it's sealed, don't reopen SSH to the whole internet. Poke one narrow door through Identity-Aware Proxy (IAP), Google's broker that lets you reach a private machine without ever giving it a public address. IAP always connects from one fixed block of Google addresses, 35.235.240.0/20, so you allow that range on port 22 and nothing else.

$ gcloud compute firewall-rules create quarantine-deny-egress \
    --network=prod-vpc --direction=EGRESS --action=DENY \
    --rules=all --destination-ranges=0.0.0.0/0 \
    --target-tags=quarantine --priority=0
Creating firewall...done.
Created [https://www.googleapis.com/compute/v1/projects/acme-prod/global/firewalls/quarantine-deny-egress].
 
$ gcloud compute firewall-rules create quarantine-deny-ingress \
    --network=prod-vpc --direction=INGRESS --action=DENY \
    --rules=all --source-ranges=0.0.0.0/0 \
    --target-tags=quarantine --priority=1
Creating firewall...done.
Created [https://www.googleapis.com/compute/v1/projects/acme-prod/global/firewalls/quarantine-deny-ingress].
 
$ gcloud compute instances add-tags web-01 \
    --zone=europe-west1-b --tags=quarantine
Updated [https://www.googleapis.com/compute/v1/projects/acme-prod/zones/europe-west1-b/instances/web-01].

Firewall rules run in priority order, and here's the counterintuitive bit: the lowest number wins, so priority 0 is the strongest rule you can write. It beats every allow rule already sitting on the network (those usually live at priority 1000 or higher). The egress block goes right at the front, at 0, because you never want a quarantined box starting an outbound connection, so nothing ever needs to sit above it. The ingress block sits one slot back, at 1, on purpose. That leaves the very front slot, priority 0, open for the one exception you might want: a single allow rule for the IAP range, checked before the deny and beating it, so your investigators can still get in. The outbound block won't cut off that session either, because Google's firewall tracks connections and always lets the return half of one it already allowed back out. The instant that last command returns and the tag lands, the box is a sealed room. Nothing in, nothing out, but still running and still holding evidence.

Photograph before you touch

A disk snapshot is your photograph. It's a frozen copy of the disk taken at one exact moment, which Google stores separately from the VM, so even if the instance gets destroyed later the snapshot survives. Take it before you do anything destructive, and label it with the incident number so nobody deletes it six months from now thinking it's junk.

$ gcloud compute disks snapshot web-01 \
    --zone=europe-west1-b \
    --snapshot-names=ir-8891-web01-boot \
    --storage-location=europe-west1 \
    --labels=incident=ir-8891,purpose=forensics
Creating snapshot(s) ir-8891-web01-boot...done.
Created [https://www.googleapis.com/compute/v1/projects/acme-prod/global/snapshots/ir-8891-web01-boot].
 
$ gcloud compute snapshots describe ir-8891-web01-boot \
    --format="value(status, diskSizeGb, sourceDisk.basename())"
READY   50   web-01

Snapshots are incremental and cheap, so cost is rarely a reason to skip one. The real upgrade is where it lands. If you can, snapshot into a separate forensics project that the compromised workload's identities can't touch. Then the damage stops at the project boundary, and your evidence sits somewhere the attacker never had a way to reach.

Change the locks

That VM was almost certainly running as a service account, a robot identity that software uses to authenticate instead of a human typing a password. If the attacker scraped its credentials off the box, they now hold that identity. Here's the trap that catches people. You disable the leaked key, breathe out, and the attacker keeps making API calls for another hour.

A service-account key is what mints access tokens, short-lived passcodes good for about an hour. Disabling the key stops it minting new ones. It does nothing to the tokens already out in the wild. To cut a live token right now, disable the service account itself. IAM, Google's Identity and Access Management system, the thing that decides who's allowed to do what, checks whether the account is still enabled on every single request, so a disabled account starts failing right away. Then strip its role bindings, the grants that spell out what it's allowed to do, so that even if some token slips through, it can do precisely nothing.

$ gcloud iam service-accounts disable [email protected]
Disabled service account [email protected].
 
$ gcloud iam service-accounts keys list \
    [email protected] \
    --format="value(name.basename(), keyType, disabled)"
9a1c4e2f7b0d31a8   USER_MANAGED   True
 
$ gcloud projects remove-iam-policy-binding acme-prod \
    --member="serviceAccount:[email protected]" \
    --role="roles/editor"
Updated IAM policy for project [acme-prod].
bindings:
- members:
  - serviceAccount:[email protected]
  role: roles/logging.logWriter
etag: BwYh2kQ9mZ0=
version: 1

Hunt for the second way in

Containment isn't the finish line. A competent attacker plants a spare way in the moment they land, because they assume you'll eventually shut the front door. So you reconstruct what the identity did, using Cloud Audit Logs, which record every administrative API call in the project. The trick that makes this reliable gets set up long before any incident: an aggregated log sink that automatically copies those logs into a locked bucket in a separate project. If the attacker took over the source project and wiped its logs, your clean copy is still sitting safely next door, untouched.

You're hunting for footholds. A new service-account key created somewhere else. A fresh permission grant. An org policy quietly loosened. A service account you don't recognize. Resources spun up in a region you never use. Logging switched off. Any one of those is a way back in. Look at the level above the project, too. Org policies (account-wide guardrails, like a rule that says no VM may get a public IP address) inherit downward: a constraint set on a parent folder cascades to every project underneath it. So an attacker who relaxes one high up weakens everything below it at once.

$ gcloud logging read \
    'protoPayload.authenticationInfo.principalEmail="[email protected]"
     AND protoPayload.methodName="google.iam.admin.v1.CreateServiceAccountKey"' \
    --project=acme-logs --freshness=7d \
    --format="table(timestamp, protoPayload.request.name)"
TIMESTAMP                    NAME
2026-07-16T02:58:11.402Z     projects/acme-prod/serviceAccounts/[email protected]

There it is. Three minutes before the CPU alarm, the compromised identity minted a new key on a different service account, deploy@. That's the spare set of keys. Kill it the same way, then rotate everything the box could see: secrets, keys, tokens. Rebuild the workload from a known-good image instead of scrubbing the infected one in place, because you'll never be completely sure you got every last thing the attacker left behind. Reconnect it last. One note for afterward, not for 3 a.m.: if a VPC Service Controls perimeter had been drawn around the project, the stolen token couldn't have carried data out to an address beyond that perimeter in the first place. Think of it as a moat around the whole project that data can't be dragged across, even by someone holding valid credentials. Run this entire sequence as a drill on a calm afternoon. The first time you do it for real should not be at 3 a.m.

Quarantine on GCP usually means: remove public exposure, apply a firewall tag that denies egress except to your forensic collector, disable or swap the service account if token theft is in play, and snapshot every disk before stop or delete. Cloud Logging and packet mirroring are your photographs. Do not run destructive cleaners on the live box because they feel productive.

Parallel tracks save time. One person preserves evidence. Another rotates keys, secrets, and WIF trust if CI was in the blast radius. A third watches SCC and audit logs for the attacker’s next move. The runbook should name those seats before anyone is tired. When you re-enter service, rebuild from known-good images; do not “clean” a compromised VM and hope.

Try this

In a lab VM you do not care about, practice the isolate commands: remove external IP / apply a quarantine tag, snapshot disks, and capture serial port or memory notes before you power anything off.

gcloud compute instances describe web-1 --zone=europe-west1-b \
  --format="yaml(status,networkInterfaces,serviceAccounts,disks)"
# quarantine: no external IP + restrictive tags (lab)
gcloud compute instances delete-access-config web-1 --zone=europe-west1-b --access-config-name="external-nat"
gcloud compute disks snapshot web-1 --zone=europe-west1-b --snapshot-names=web-1-ir-$(date +%Y%m%d%H%M)
gcloud compute instances stop web-1 --zone=europe-west1-b  # only AFTER snapshots

status: RUNNING
networkInterfaces:
- networkIP: 10.10.2.14
  accessConfigs:
  - name: external-nat
    natIP: 34.78.10.20
serviceAccounts:
- email: [email protected]
 
Updated [web-1].   # access config removed
 
Snapshot created [web-1-ir-202607160315].
# stop only once evidence disk snapshots exist

Takeaway

Remember: isolate first, image second, rotate credentials third, hunt for persistence fourth. Speed without order destroys the only evidence you will get.

You have walked the gcp-sec path from hierarchy to incident response. In a real event, pull the runbook, declare an incident commander, and resist the urge to reboot clean until snapshots and key disables are done.

Related