Cloud incident response & forensics

Isolation, snapshots, credential compromise, and runbooks.

Cloud incident response is like investigating a break-in at a building whose rooms rearrange themselves every few minutes. The intruder carries a key that quietly copies itself. Brush the wrong switch and the whole crime scene gets bulldozed and rebuilt spotless. There is no server to unplug and carry down to a lab. Everything is an API call (a request your tools send to the provider over the network), every credential expires on a timer, and the evidence, the memory and disks and logs, keeps changing under you and quietly deletes itself.

So cloud forensics is a race, run through the same control plane you administer with (the provider's management interface, the thing your dashboards and scripts talk to), to freeze the state of things faster than your own automation wipes it. Three plain words hold the whole job together. Containment shrinks the blast radius (how far the attacker can reach). Eradication pulls out the attacker's footholds. Forensics preserves and reconstructs what really happened. The order is fixed: contain first, preserve second, understand third. And containment must never destroy the record.

Isolate the box, keep it breathing

This is why the reflex to terminate a hacked machine is almost always the wrong first move. Terminating throws away volatile memory: secrets sitting decrypted in RAM (a machine's fast, temporary memory that empties the moment it loses power), the malware's command-and-control settings (the configuration it uses to phone home to the attacker), processes injected straight into memory that never touched the disk. On a throwaway instance, terminating usually deletes the root disk too, erasing the only account of what was taken and whether the attacker left themselves a way back in. Isolation makes a box harmless. A snapshot makes it answerable. A terminated box is just gone.

Isolation is one idea spelled three ways. On AWS you drop the instance into a quarantine security group (a virtual firewall wrapped around the instance) that permits nothing, and you detach it from its load balancer and Auto Scaling group (the service that keeps a fixed number of healthy instances running). On Google Cloud you re-tag the virtual machine so a deny-all firewall rule you staged earlier latches onto it, or you strip its external IP address. On Azure you bind a lock-down network security group (NSG, Azure's per-resource firewall) to the network card. Every time, the box keeps running so its memory stays alive, while the network goes dark so the attacker is locked out. Detaching matters as much as denying. Leave the instance in its Auto Scaling group and the platform may spot an 'unhealthy' box and terminate it for you, shredding the evidence you were trying to hold.

# modify-instance-attribute is silent on success; the quarantine SG replaces ALL current SGs
$ aws ec2 modify-instance-attribute --instance-id i-0ab12cd34ef567890 --groups sg-0quarantine
 
# then pull the box out of serving so the platform cannot recycle it
$ aws autoscaling detach-instances --instance-ids i-0ab12cd34ef567890 \
    --auto-scaling-group-name web-asg --should-decrement-desired-capacity

{
    "Activities": [
        {
            "ActivityId": "9c3e0f4a-1b2c-4d5e-8f90-a1b2c3d4e5f6",
            "AutoScalingGroupName": "web-asg",
            "Description": "Detaching EC2 instance: i-0ab12cd34ef567890",
            "Cause": "At 2026-07-14T09:41:52Z instance i-0ab12cd34ef567890 was detached in response to a user request, shrinking the capacity from 4 to 3.",
            "StartTime": "2026-07-14T09:41:52.001000+00:00",
            "StatusCode": "InProgress",
            "Progress": 50,
            "Details": "{\"Subnet ID\":\"subnet-0abc1234\",\"Availability Zone\":\"us-east-1a\"}"
        }
    ]
}

# re-tag so the pre-staged deny-all firewall rule captures the VM
$ gcloud compute instances add-tags web-01 --zone=us-central1-a --tags=quarantine

Updated [https://www.googleapis.com/compute/v1/projects/prod-app/zones/us-central1-a/instances/web-01].

# bind a deny-all NSG to the instance's network card
$ az network nic update -g rg-prod -n web-01-nic --network-security-group nsg-quarantine

{
  "name": "web-01-nic",
  "networkSecurityGroup": {
    "id": "/subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Network/networkSecurityGroups/nsg-quarantine",
    "resourceGroup": "rg-prod"
  },
  "provisioningState": "Succeeded",
  "resourceGroup": "rg-prod"
}

Freeze the disks and the memory

With the box sealed, copy the evidence before you touch anything destructive. A snapshot is a photocopy of a disk at one instant, taken through the control plane. It is crash-consistent, which means it captures the disk exactly as it sat at that moment, the way a disk looks after a power cut. That is fine for forensics, where you are reading the bytes, not booting a live database off them. Snapshot the boot volume and every data volume. Tag each one to the case number so your chain of custody (the paper trail proving nobody altered the evidence) holds up. Then copy the snapshot into a separate forensics account or project that the compromised principal has no route into. Evidence still sitting inside the blast radius is not evidence: an attacker holding the workload's role could delete or edit it. Snapshots cost real money and take minutes to hours at scale, so keep them scripted. A runbook that makes you stop and look up flag syntax mid-incident is not a runbook.

$ aws ec2 create-snapshot --volume-id vol-0abc123def4567890 --description "IR-8891 evidence" \
    --tag-specifications 'ResourceType=snapshot,Tags=[{Key=case,Value=IR-8891}]'

{
    "SnapshotId": "snap-0f1e2d3c4b5a6d7e8",
    "VolumeId": "vol-0abc123def4567890",
    "State": "pending",
    "StartTime": "2026-07-14T09:42:11.000000+00:00",
    "Progress": "",
    "OwnerId": "123456789012",
    "Description": "IR-8891 evidence",
    "VolumeSize": 30,
    "StorageTier": "standard",
    "Encrypted": true,
    "Tags": [
        {
            "Key": "case",
            "Value": "IR-8891"
        }
    ]
}

$ gcloud compute disks snapshot web-01 --zone=us-central1-a \
    --snapshot-names=ir-8891-evidence --storage-location=us

Creating snapshot(s) ir-8891-evidence...done.

# snapshot created straight into a dedicated forensics resource group
$ az snapshot create -g rg-forensics -n ir-8891-evidence \
    --source /subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Compute/disks/web-01-osdisk

{
  "creationData": {
    "createOption": "Copy",
    "sourceResourceId": "/subscriptions/xxxx/resourceGroups/rg-prod/providers/Microsoft.Compute/disks/web-01-osdisk"
  },
  "diskSizeGb": 30,
  "diskState": "Unattached",
  "location": "eastus",
  "name": "ir-8891-evidence",
  "provisioningState": "Succeeded",
  "resourceGroup": "rg-forensics"
}

The control plane cannot photograph RAM. A snapshot copies disks, and the memory of a running machine sits outside its reach. To grab it you run an agent inside the guest operating system, something like AVML (Acquire Volatile Memory for Linux) or LiME (Linux Memory Extractor), and have it write the memory image straight to your evidence bucket. That image holds the live network connections, unencrypted payloads, and injected code that termination would have destroyed, so it goes first, through the narrow rule you left open before the network went dark.

The credential-compromise runbook

Most cloud incidents are not malware humming on a box. They are a leaked credential: an access key committed to a public code repository, a token lifted from the instance metadata endpoint (a special internal address, 169.254.169.254, where a machine fetches its own short-lived credentials), a service-account key file left on a laptop that walked out of a cafe. The runbook has four beats: revoke, reconstruct, hunt, restore.

The part that keeps burning teams is that credentials do not all die the same way. A long-lived key, an IAM (Identity and Access Management, the cloud's system for who-can-do-what) user's access key or a Google service-account key, you switch off directly and it is finished. A temporary session is a different animal. AWS Security Token Service (STS) credentials from an AssumeRole call, a federated login token, a Microsoft Entra ID (Microsoft's cloud identity service, once called Azure Active Directory) access token: none of these can be pulled back one at a time, and they stay valid until their own timer runs out. An STS session can be handed out to run for up to twelve hours. Deactivating the key that assumed the role does nothing to the sessions already minted from it. To kill live AWS sessions you deny by the clock: attach a policy to the role that rejects every action carrying a token issued before your cutoff, using the aws:TokenIssueTime condition (the stamp AWS puts on every temporary credential recording when it was born). That is exactly what the console's 'Revoke active sessions' button does behind the glass. On Google Cloud you disable the service account and its keys. In Entra you revoke the account's refresh tokens (the long-lived tokens it uses to mint fresh short-lived ones) and switch the account off; the short access tokens already handed out expire on their own, usually within an hour, though Entra's default lets them run 60 to 90 minutes. A compromised app identity (a service principal, Entra's name for a workload's own login) is the awkward case: its tokens cannot be revoked one by one either, so you strip its client secrets and certificates so it cannot mint new ones, then disable it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "DateLessThan": { "aws:TokenIssueTime": "2026-07-14T09:40:00Z" }
      }
    }
  ]
}

# 1) stop the long-lived key minting new sessions (silent on success)
$ aws iam update-access-key --user-name app --access-key-id AKIAIOSFODNN7EXAMPLE --status Inactive
 
# 2) kill sessions already issued from it (silent on success)
$ aws iam put-role-policy --role-name app-role --policy-name AWSRevokeOlderSessions \
    --policy-document file://revoke-older-sessions.json
 
# 3) verify the deny actually landed
$ aws iam get-role-policy --role-name app-role --policy-name AWSRevokeOlderSessions \
    --query 'PolicyDocument.Statement[0].Condition'

{
    "DateLessThan": {
        "aws:TokenIssueTime": "2026-07-14T09:40:00Z"
    }
}

# disable the leaked key first, then the service account itself
$ gcloud iam service-accounts keys disable a1b2c3d4e5f60718293a4b5c6d7e8f9012345678 \
    [email protected]
$ gcloud iam service-accounts disable [email protected]

Disabled key [a1b2c3d4e5f60718293a4b5c6d7e8f9012345678] for service account [[email protected]].
Disabled service account [[email protected]].

# revoke every refresh token for the account, then switch it off
$ az rest --method POST \
    --url "https://graph.microsoft.com/v1.0/users/[email protected]/revokeSignInSessions"
$ az ad user update --id [email protected] --account-enabled false

{
  "@odata.context": "https://graph.microsoft.com/v1.0/$metadata#Edm.Boolean",
  "value": true
}

Reconstruct what the credential touched

Stopping the bleeding is not the same as knowing what happened. For that you open the audit trail, the cloud's own ledger of every API call: who did what, when, and from which address. On AWS that ledger is CloudTrail, and CloudTrail Lake lets you query it in plain SQL. On Google Cloud it is Cloud Audit Logs. On Azure it is the Activity Log, paired with Entra sign-in logs for the identity side. Scrolling a JSON event log by hand during an incident is hopeless; a query lets you ask a precise question and get a precise list. Filter by the compromised principal and the incident window, then read off which APIs ran, from which source IPs, in which regions. A single finding from GuardDuty, Security Command Center, or Microsoft Defender is a place to start looking, not the whole story. The audit trail is the ground truth.

$ aws cloudtrail start-query --query-statement \
  "SELECT eventTime, eventName, sourceIPAddress, awsRegion
   FROM a1b2c3d4-5678-90ab-cdef-1234567890ab
   WHERE userIdentity.accessKeyId = 'AKIAIOSFODNN7EXAMPLE'
     AND eventTime > '2026-07-14 08:00:00'
   ORDER BY eventTime"

{
    "QueryId": "8e4f0c1a-2b3c-4d5e-6f70-1a2b3c4d5e6f"
}

$ aws cloudtrail get-query-results --query-id 8e4f0c1a-2b3c-4d5e-6f70-1a2b3c4d5e6f \
    --query 'QueryResultRows[*]' --output text

2026-07-14T08:12:03Z    CreateAccessKey    203.0.113.7    us-east-1
2026-07-14T08:12:44Z    PutUserPolicy      203.0.113.7    us-east-1
2026-07-14T08:15:10Z    RunInstances       203.0.113.7    ap-south-1

$ gcloud logging read \
  'protoPayload.authenticationInfo.principalEmail="[email protected]"
   AND protoPayload.methodName:("CreateServiceAccountKey" OR "SetIamPolicy")' \
  --freshness=1d \
  --format="table(timestamp, protoPayload.methodName, protoPayload.requestMetadata.callerIp)"

TIMESTAMP                       METHOD_NAME                                  CALLER_IP
2026-07-14T08:19:22Z            google.iam.admin.v1.CreateServiceAccountKey  203.0.113.7
2026-07-14T08:20:05Z            SetIamPolicy                                 203.0.113.7

$ az monitor activity-log list --caller [email protected] \
    --start-time 2026-07-14T08:00:00Z \
    --query "[].{time:eventTimestamp, op:operationName.localizedValue, ip:httpRequest.clientIpAddress}" \
    -o table

Time                  Op                                Ip
--------------------  --------------------------------  -----------
2026-07-14T08:22:31Z  Create or Update Virtual Machine  203.0.113.7
2026-07-14T08:23:09Z  Create role assignment            203.0.113.7

That timeline answers three questions in order. What did the attacker read or copy out (the exfiltration question). What did they change (new policies, new resources, deleted logs). And the one that decides whether the incident is actually over: did they set up a way back in. The first two you can read off the calls in front of you. The third you have to go looking for, because the handful of calls that build persistence hide inside a day of ordinary admin noise.

So hunt persistence on purpose, because a competent attacker plants a second door before you find the first. The tells rhyme across all three clouds: new users or access keys, a role's trust policy quietly rewritten to trust an attacker's account, fresh service-account keys, new Entra app registrations or federated credentials, resources launched in regions you never use (RunInstances in ap-south-1 from a shop that only runs in us-east-1 is the classic mining tell), logging switched off or redirected, and new grants at the organization level: a service control policy here, an IAM binding on a folder or org node there, a management-group role assignment. Every foothold you miss is a re-entry an hour later.

Restore, or burn it down

Only after every foothold is pulled and everything the credential could reach is rotated do you restore. Rebuild clean, then reconnect. Do not un-isolate the original box and call it recovered; that box is evidence now, not a server. And there is a harder call hiding in here. If the attacker held admin over the account, or the audit logs themselves were switched off or edited, you cannot forensically trust a word that account says. The honest move is to stop trusting it and rebuild in a fresh account or project, treating the old one as contaminated ground. The same misconfigurations you chase at the tail of an incident, public keys, over-broad trust policies, logging that drifted off, are the ones a Cloud Security Posture Management tool (CSPM, software that scans your cloud for risky settings around the clock) flags every single day, which is where the next lesson picks up.

Try this

Work through “Restore, or burn it down” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: deny-all can lock out your own responders. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related