Incident response in AWS

Key compromise runbook, step by step.

Alarms are the easy part. A smoke detector that screams at 3 a.m. does you no good if nobody knows where the exits are, who calls the fire brigade, or how to shut the gas off. The last two lessons wired up the detectors: CloudTrail (the AWS audit log that writes down every API call like lines in a ledger) and GuardDuty (the service that reads that ledger, watches your network traffic, and flags the activity that looks like an attack). Incident response, IR, is everything that happens after the alarm sounds. It is the practiced sequence that turns 'we think a key leaked' into 'the attacker is out, the evidence is saved, and we know how they got in.'

An incident is confirmed unauthorized activity. A stolen access key being used from an address on another continent. A cryptominer chewing through pricey GPU (graphics-chip) instances you never launched. An IAM (Identity and Access Management, the service that decides who is allowed to do what) user that nobody on your team remembers creating. The word that matters is confirmed. Before that, you have an alert. After it, you have a clock running.

Why AWS incidents are credential incidents

In a datacenter, containment is something you can touch. You walk to the rack and pull the network cable, and the machine is off the network whether the attacker likes it or not. AWS has no cable to pull. The fight happens on the control plane, the management API (application programming interface, the set of remote calls a service accepts) layer where instances are launched, users are created, and disks are copied. Anyone with valid credentials can reach that API from a laptop in any coffee shop on earth. So nearly every real AWS incident is a credential incident: someone is holding keys they should not have.

The clock is the whole problem. The thing IR exists to shrink is dwell time, the stretch between the attacker getting in and you getting them out. A key pasted into a public GitHub repository is usually scraped and used within minutes, because bots sit and watch for exactly that. The common pattern: call RunInstances to spin up mining rigs on your bill, create a spare IAM user so they keep a way in after you rotate the leaked key, then start reading S3 (Simple Storage Service, the object store) to find data worth stealing. Every minute you spend fumbling is money burned and roots pushed deeper.

The same API that hurts you also arms you. You can freeze an identity, cut a machine off the network, and copy its disk in seconds, from a laptop, without walking anywhere. The standard response lifecycle (detect, contain, preserve, eradicate, learn) maps cleanly onto AWS API calls, which means you can script it, rehearse it, and hand the boring parts to automation.

Scope before you swing

When GuardDuty fires, the finding names the identity for you under resource.accessKeyDetails. The instinct is to start deleting things right away. Fight it for ninety seconds. What that identity already did decides your whole response, and you cannot tell the difference between a key that listed one bucket and a key that created three users by staring at the alert. CloudTrail can replay the last 90 days of management events, filtered to a single access key.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=AccessKeyId,AttributeValue=AKIAIOSFODNN7EXAMPLE \
  --start-time 2026-07-13T00:00:00Z \
  --max-results 5 \
  --query 'Events[].{time:EventTime,name:EventName,source:EventSource}' \
  --output table

-------------------------------------------------------------------------
|                             LookupEvents                              |
+----------------+---------------------+------------------------------+
|      name      |       source        |             time             |
+----------------+---------------------+------------------------------+
|  CreateUser    |  iam.amazonaws.com  |  2026-07-13T09:21:47+00:00   |
|  RunInstances  |  ec2.amazonaws.com  |  2026-07-13T09:16:03+00:00   |
|  RunInstances  |  ec2.amazonaws.com  |  2026-07-13T09:14:22+00:00   |
|  ListBuckets   |  s3.amazonaws.com   |  2026-07-13T09:11:58+00:00   |
+----------------+---------------------+------------------------------+

Read it from the bottom up, oldest first. The key listed buckets, launched two instances, then created a user. The two RunInstances calls are the mining. The CreateUser is persistence, a second identity so that killing the original key does not lock the attacker out. That one row changes your plan. This is not a quick rotate-and-move-on. You are prying out a foothold.

Contain the identity

Now cut them off, and watch the trap that catches almost every first-timer. Deactivating an access key kills that key and nothing else.

# Deactivate the leaked key so it can assume nothing new
aws iam update-access-key \
  --user-name ci-deployer \
  --access-key-id AKIAIOSFODNN7EXAMPLE \
  --status Inactive

# no output; the CLI is silent and exits 0 on success

If the attacker used that key to call sts:AssumeRole (Security Token Service, the part of AWS that hands out temporary credentials), they stopped needing the key the moment they got a token back. That STS token is good for up to 12 hours, and AWS cannot pull it back once it is issued. There is no cancel button for a live token. What you can do is add a rule that makes AWS reject it on its next call. The console has a button for this labelled 'Revoke active sessions.' Here is the policy it actually attaches.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyStaleSessions",
      "Effect": "Deny",
      "Action": "*",
      "Resource": "*",
      "Condition": {
        "DateLessThan": { "aws:TokenIssueTime": "2026-07-13T09:30:00Z" }
      }
    }
  ]
}

# Reject every STS session the key handed out before the cutoff
aws iam put-role-policy \
  --role-name app-runtime \
  --policy-name AWSRevokeOlderSessions \
  --policy-document file://revoke-older-sessions.json

# no output; the CLI is silent and exits 0 on success

Read the condition as a sentence: deny every action on every resource when the token was issued before 09:30 UTC today. This works because of the one rule in IAM evaluation you can always count on. An explicit Deny beats every Allow, no matter where the Allow comes from: the identity's own policy, a resource policy, a permission boundary, an Organizations service control policy or resource control policy, all of it. It does not matter that the attacker's token still carries the role's real permissions. The instant its issue time falls on the wrong side of your cutoff, every call it makes is denied.

Your own workloads heal themselves. A legitimate service running as app-runtime sees its calls start failing, re-assumes the role, and gets a fresh token stamped after the cutoff, which sails through the condition. You get a short blip, not an outage. Set the cutoff to now, never to a time in the future, or you lock out the recovery along with the attacker.

Isolate the host without killing it

One of those RunInstances calls is a live box the attacker controls. You want it off the network. You do not want it powered off. A running compromised instance is holding evidence that disappears the moment it shuts down: the malware sitting in memory, the keys it decrypted to run, the open command-and-control connection back to whoever is driving it. Reboot it and you have swept the crime scene for them. So the order is quarantine, lock, strip, image, and you do it in that order.

# Swap into a pre-made quarantine group that allows nothing
aws ec2 modify-instance-attribute \
  --instance-id i-0abc12345def67890 \
  --groups sg-0b7c8d9e0f1a2b3c4
 
# Stop the attacker (or a panicked teammate) from terminating it
aws ec2 modify-instance-attribute \
  --instance-id i-0abc12345def67890 \
  --disable-api-termination
 
# Find the association that binds the instance to its IAM role
aws ec2 describe-iam-instance-profile-associations \
  --filters Name=instance-id,Values=i-0abc12345def67890 \
  --query 'IamInstanceProfileAssociations[0].AssociationId' \
  --output text

iip-assoc-0a1b2c3d4e5f6a7b8

# Detach the role so the instance's temporary creds go dead
aws ec2 disassociate-iam-instance-profile \
  --association-id iip-assoc-0a1b2c3d4e5f6a7b8
 
# Snapshot every attached volume before anything else changes
aws ec2 create-snapshot \
  --volume-id vol-0f1e2d3c4b5a69788 \
  --description "IR-2041 web-tier forensic image" \
  --tag-specifications 'ResourceType=snapshot,Tags=[{Key=incident,Value=IR-2041}]'

{
    "IamInstanceProfileAssociation": {
        "AssociationId": "iip-assoc-0a1b2c3d4e5f6a7b8",
        "InstanceId": "i-0abc12345def67890",
        "IamInstanceProfile": {
            "Arn": "arn:aws:iam::111122223333:instance-profile/app-runtime",
            "Id": "AIPAEXAMPLE1234567890"
        },
        "State": "disassociating"
    }
}
{
    "Description": "IR-2041 web-tier forensic image",
    "Encrypted": true,
    "KmsKeyId": "arn:aws:kms:us-east-1:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab",
    "SnapshotId": "snap-0d9e8f7a6b5c43210",
    "StartTime": "2026-07-13T09:34:12.000Z",
    "Progress": "",
    "State": "pending",
    "VolumeId": "vol-0f1e2d3c4b5a69788",
    "VolumeSize": 100,
    "OwnerId": "111122223333",
    "Tags": [
        { "Key": "incident", "Value": "IR-2041" }
    ]
}

An EBS (Elastic Block Store) snapshot is a photograph of an EC2 (Elastic Compute Cloud, AWS's virtual servers) instance's virtual hard drive at a single instant. Taken before you poke anything, it is the disk evidence you hand to a forensics team. Detaching the IAM role matters just as much as the network move. It stops the metadata service, the internal address an instance queries to get its own temporary credentials, from handing the attacker anything fresh. The credentials they already scraped keep working until they expire, usually within a few hours, which is the other reason that session-time Deny you set on app-runtime earlier earns its keep: the instance runs as that same role, so your cutoff denies its stolen sessions too.

Eradicate by rebuilding

You cannot prove a compromised host is clean. No scan returns 'definitely no backdoors,' because a decent implant hides from the very tools you would use to look for it. So do not try to clean it. Terminate it and redeploy from infrastructure-as-code you trust (the Terraform or CloudFormation that built it in the first place), then rotate every secret that machine could have read: the Secrets Manager (AWS's password vault) entries its role could fetch, the database passwords, anything baked into the image.

When the mess is wider than one instance, stop hand-writing queries. Amazon Detective builds a behavior graph out of your CloudTrail, VPC Flow Logs, and GuardDuty data, so you can follow one identity across every call it made and every resource it touched without composing SQL in Athena (the service that runs queries over logs in S3) by hand.

Know where these tools stop, because the gaps bite during a real incident. lookup-events reaches back only 90 days, and only for management events, not data events like individual S3 object reads (you turn those on separately, and pay for them). A snapshot of an encrypted volume is a brick to a forensics account that cannot use the KMS (Key Management Service, AWS's encryption-key manager) key protecting it. Grant that account use of the key when you share the snapshot, and encrypt the volume with a customer-managed key in the first place, because the default AWS-managed key cannot be shared across accounts at all. And none of this captures memory. For a RAM image you run a capture tool through SSM (Systems Manager, AWS's remote-command service) while the instance is still alive, which is one more reason you froze the box instead of rebooting it.

Prepare before the pager

How this goes is mostly decided before the incident starts. You cannot calmly stand up a forensics account at 3 a.m. with an attacker in your logs. Build the pieces you will grab for ahead of time: an empty, tagged quarantine security group in every VPC, a separate forensics account that snapshots get shared into, and a break-glass responder role with real power, locked behind MFA (multi-factor authentication) and wired to alarm loudly every single time anyone assumes it.

Then make the first five minutes run without a human. An EventBridge (AWS's event router) rule can match high-severity GuardDuty findings and kick off a runbook that tags, isolates, and snapshots before anyone has even read the page.

aws events put-rule \
  --name ir-guardduty-high-severity \
  --event-pattern '{
    "source": ["aws.guardduty"],
    "detail-type": ["GuardDuty Finding"],
    "detail": { "severity": [ { "numeric": [">=", 7] } ] }
  }'

{
    "RuleArn": "arn:aws:events:us-east-1:111122223333:rule/ir-guardduty-high-severity"
}

Point that rule at an SSM Automation document or a Lambda function (a snippet of code AWS runs on demand, with no server for you to keep alive) with aws events put-targets, and containment starts itself while you are still finding your laptop.

The first time you revoke live sessions and quarantine a box should not be during a real breach. GuardDuty will hand you a harmless fire to practice on.

aws guardduty create-sample-findings \
  --detector-id 12abc34d567e8fa901bc2d34e56789f0 \
  --finding-types UnauthorizedAccess:EC2/SSHBruteForce

# no output; the finding lands in the console within a minute or two

Run that in a sandbox account, then walk the whole runbook end to end as a game day: scope the key, revoke the sessions, quarantine the instance, snapshot the disk, tear it down and rebuild. Do it on a quiet Tuesday with a coffee in hand, so the moves are already in your fingers on the day it is not a drill.

Related