Incident response & learning

Automated response and blameless post-mortems.

A high-rise fights fire in four layers. Smoke detectors notice trouble while it is still smoke. Sprinklers drown the ordinary, well-understood fire the second it starts. The fire brigade gets called only for the blaze that jumped the fire-break and needs a person deciding what to do next. Then, days later, somebody walks the building and bolts a detector onto the one ceiling where the smoke went unseen. Cloud incident response runs on those same four layers. Incidents happen to everybody, so what separates a mature DevOps practice from a shaky one is how fast it notices, how fast it responds, and whether it does both the same way every single time. Three clocks measure that. MTTD (mean time to detect) is how long the problem hides before your monitoring sees it. MTTA (mean time to acknowledge) is how long until a human picks up the page. MTTR (mean time to restore service) is how long your customers stay broken. Every technique below exists to shrink one of those three.

From signal to action: the detection pipeline

Fast response starts by wiring the thing that notices straight to the thing that acts. Three AWS services do the noticing. CloudWatch alarms watch a metric and trip when it crosses a line you drew (an error-rate spike). GuardDuty, the threat-detection service, raises a finding when something looks compromised (an instance quietly beaconing out to an unknown host). AWS Config rules inspect resource settings and fire when one drifts out of policy (a bucket that went public thirty seconds ago). Each of those emits an event. EventBridge is the bus in the middle, a serverless pipe that matches incoming events against JSON patterns and forwards the matches wherever you point them. Point it at an *actuator*, meaning something that actually performs the fix: a Lambda function, an SSM Automation runbook (Systems Manager's scripted-procedure engine), or a Step Functions workflow. The response runs in seconds. Nothing polls, nothing waits for the next cron tick. One cost note worth banking: events that AWS services publish to your account's default bus (GuardDuty, Config, alarm state changes) are free to match, and you pay roughly $1.00 per million only for *custom* events you publish yourself.

# Match only high-severity GuardDuty findings (severity >= 7):
$ aws events put-rule --name gd-critical-to-runbook \
    --event-pattern '{
      "source": ["aws.guardduty"],
      "detail-type": ["GuardDuty Finding"],
      "detail": { "severity": [ { "numeric": [ ">=", 7 ] } ] }
    }'
{
    "RuleArn": "arn:aws:events:us-east-1:123456789012:rule/gd-critical-to-runbook"
}
 
# Target an SSM Automation runbook (an EventBridge role is required to invoke it):
$ aws events put-targets --rule gd-critical-to-runbook --targets '[{
      "Id": "isolate",
      "Arn": "arn:aws:ssm:us-east-1:123456789012:automation-definition/Isolate-Ec2:$DEFAULT",
      "RoleArn": "arn:aws:iam::123456789012:role/EventBridge-SSM-Invoke"
    }]'
{
    "FailedEntryCount": 0,
    "FailedEntries": []
}

Two details decide whether this holds up during a real incident. EventBridge delivers at-least-once and keeps retrying a failing target for up to 24 hours, so hang a dead-letter queue (a holding pen for messages that could never be delivered) off every target. Without one, a transient failure swallows your response and nobody ever finds out. Quotas shape the design as well: 300 rules per event bus per Region by default, raisable through Service Quotas, and PutEvents accepts at most 10 entries per request. Then filter hard, on the severity carried inside the finding. The numeric pattern above wakes the isolation runbook only for GuardDuty findings at severity 7 or higher, so low-value noise never reaches an actuator and never pages anyone.

Codify the response as a runbook, not a reflex

Pilots who lose an engine do not improvise. They reach for a laminated checklist and run it in order, because a fix held in memory drifts a little every time it is used and a written one does not. A runbook is that checklist for your system, written as executable steps so the 3 a.m. version matches the drill version exactly. SSM Automation documents are the AWS-native form: ordered steps (aws:executeAwsApi to call an API, aws:invokeLambdaFunction to run your own code, aws:approve to stop and wait for a human) with typed parameters and an IAM role (Identity and Access Management, the AWS permissions system) that the document assumes, scoped to only what those steps need. When the response has branches, waits, or parallel fan-out, a Step Functions state machine reads better and lasts longer. A *Standard* workflow can run for a year, costs about $0.025 per 1,000 state transitions, and keeps an auditable execution history for free. Any runbook that touches a fleet needs rate control. --max-concurrency caps how many targets it works on at once, --max-errors aborts the whole run once failures pile up, and together they stop a wrong fix from sweeping every instance you own.

# Restart only tagged web instances, 10% at a time, stop after 5 failures:
$ aws ssm start-automation-execution \
    --document-name "AWS-RestartEC2Instance" \
    --targets "Key=tag:app,Values=web" \
    --target-parameter-name InstanceId \
    --max-concurrency "10%" --max-errors "5"
{
    "AutomationExecutionId": "c3f2a0d1-7b4e-4a9c-9e21-6d0f8b2a1c34"
}
 
$ aws ssm get-automation-execution \
    --automation-execution-id c3f2a0d1-7b4e-4a9c-9e21-6d0f8b2a1c34 \
    --query 'AutomationExecution.{Status:AutomationExecutionStatus,Ok:ExecutedBy}'
{
    "Status": "Success",
    "Ok": "arn:aws:sts::123456789012:assumed-role/SsmAutomationRole/..."
}

Systems Manager Incident Manager

AWS packages the human side of response as Systems Manager Incident Manager. (AWS has since closed it to new customers. Existing accounts keep using it as normal, and the pattern it models is the thing worth understanding, whichever paging tool you end up standardising on.) You define contacts and an escalation plan (page the primary on-call, then the secondary five minutes later if nobody answers), on-call rotation schedules so the phone book has a calendar attached, and a response plan that bundles a severity, a chat channel (AWS Chatbot pushes it into Slack or Teams), and the runbooks to launch automatically. Point a CloudWatch alarm's --alarm-actions straight at a response-plan ARN (Amazon Resource Name, the unique identifier of an AWS resource) and one breach opens an incident, engages on-call, starts the runbook, and begins a timeline, with no human in the loop yet. Incident creation is de-duplicated: while an incident for that alarm is still open, further ALARM/OK flapping folds into the same record instead of paging the team a dozen times. Two operational catches come with it. The service is Regional, so it needs a replication set across Regions or the incident record dies along with the Region that is on fire. And you are billed per incident and per contact engagement, with Service-Quota limits in the low hundreds on contacts, escalation plans and response plans, so rehearse against a low-severity plan before you wire anything to SEV-1.

$ aws ssm-incidents start-incident \
    --response-plan-arn arn:aws:ssm-incidents::123456789012:response-plan/prod-api-sev2 \
    --title "prod API 5xx spike"
{
    "incidentRecordArn": "arn:aws:ssm-incidents::123456789012:incident-record/prod-api-sev2/9d1f7c2e-..."
}
 
$ aws ssm-incidents list-incident-records \
    --query 'incidentRecordSummaries[].{Title:title,Status:status,Impact:impact}' --output table
----------------------------------------------------
|              ListIncidentRecords                 |
+--------------------+----------+------------------+
|       Title        |  Status  |     Impact       |
+--------------------+----------+------------------+
|  prod API 5xx spike|  OPEN    |  2               |
+--------------------+----------+------------------+

From that moment the service does the boring clerical work for you. Every state change and every engagement gets a timestamp on a timeline. Related items hang off the record: the dashboard that was screaming, the deploy that shipped forty minutes earlier, the runbook execution. The three clocks get their numbers without anybody holding a stopwatch. Responders talk in the chat channel while the audit trail assembles itself behind them, and that trail is exactly the raw material a post-mortem needs. It is also exactly the material nobody can honestly reconstruct from memory three days later.

The blameless post-mortem

The loop closes with learning, or it does not close at all. A *blameless* post-mortem (Amazon runs its own version internally under the name Correction of Errors, or COE) asks what the system and the conditions allowed to happen, never which person to hang it on. That is not politeness. Engineers who expect punishment quietly withhold the one detail that would have stopped the next occurrence. So walk the timeline. Ask "why" again and again until you land on something structural rather than "human error," because "human error" is where an investigation stops, not where it ends. Then write action items that are concrete, owned by a named person, and dated: an alarm for the signal you did not have, automation for the step somebody did by hand, a real fix for the root cause, a sharper runbook. Incident Manager ships post-incident-analysis templates built for this. Track every item to closure, because an untracked action item is a promise rather than a fix. And when a service keeps overspending its error budget (the amount of unreliability you agreed up front you could afford), stop shipping features and spend that time on resilience.

$ aws ssm-incidents list-timeline-events \
    --incident-record-arn arn:aws:ssm-incidents::123456789012:incident-record/prod-api-sev2/9d1f7c2e-... \
    --query 'eventSummaries[].{At:eventTime,Type:eventType}' --output table
---------------------------------------------------
|              ListTimelineEvents                 |
+------------------------+------------------------+
|          At            |         Type           |
+------------------------+------------------------+
|  2026-07-14T02:11:07Z  |  Custom Event          |  # alarm -> incident opened
|  2026-07-14T02:11:34Z  |  Custom Event          |  # runbook AWS-RestartEC2Instance ran
|  2026-07-14T02:19:52Z  |  Custom Event          |  # on-call acknowledged (MTTA ~9m)
|  2026-07-14T02:41:03Z  |  Custom Event          |  # service restored (MTTR ~30m)
+------------------------+------------------------+

Whatever you page with, Incident Manager or something else entirely, it needs a rotation, an escalation path, and a bridge link that lives somewhere other than one senior engineer's memory. A list of contacts with no plan attached is a phone book.

Write the fixes down where machines can read them. A ticket with an owner keeps the work visible while it is open, and folding the durable part back into an Automation document or a Config rule keeps it working long after everyone has forgotten which incident caused it.

Write your severity definitions on a calm Tuesday afternoon, never during the incident. They exist to end the 2 a.m. argument about how bad this really is. SEV-1 should mean customer-wide impact you can point at on a graph, not "the person on call is stressed."

Try this

Go and prove the wire actually exists. List your Incident Manager contacts and response plans if the service is switched on, and either way pull one CloudWatch alarm and read its action list to confirm it really points at an SNS topic (Simple Notification Service, the AWS fan-out messaging service that delivers the page).

aws ssm-contacts list-contacts --query 'Contacts[].{Alias:Alias,Type:Type}' --output table
aws ssm-incidents list-response-plans --query 'responsePlanSummaries[].name' --output table
aws cloudwatch describe-alarms --alarm-names app-sev1 \
  --query 'MetricAlarms[0].{Actions:AlarmActions,State:StateValue}' --output json

oncall-primary | PERSONAL
prod-sev1-plan
{
  "Actions": ["arn:aws:sns:us-east-1:111122223333:pager"],
  "State": "OK"
}

Takeaway

Remember: detect fast, automate the fixes you can undo, page a human for the ones you cannot, and review in the open without blame. MTTR falls when the response is written down as code instead of recalled under pressure.

Next: run a game day. Break something on purpose, work it through the real response plan, and file two action items that change automation rather than documentation.

Closing the loop, and closing the course

That feedback is the whole point, and it is what ties this course together. Every piece you built now feeds the loop. CodePipeline, CodeBuild and CodeDeploy give you delivery that is fast and reversible, which is what makes "roll it back" a real option at 3 a.m. CloudFormation, the CDK (Cloud Development Kit), SAM (Serverless Application Model) and StackSets keep your infrastructure as code somebody can review and re-apply. Systems Manager, Config, CloudWatch, X-Ray and EventBridge supply the signals and the guardrails. Progressive delivery shrinks the blast radius when something ships broken, and HA/DR automation (high availability and disaster recovery) keeps you alive when an entire Region has a bad day. The incident loop is what tunes all of it. Measure, automate, learn, repeat. Each incident should leave you detecting faster and leave one more failure mode handled without waking anybody, so the same surprise never gets a second turn.

Nobody gets a year without incidents. What you can build is a loop that tightens every time it turns: automate the response you already understand, save scarce human judgement for the failure nobody has seen before, and make every outage buy a permanent improvement instead of a repeat of the same 3 a.m. page. Fast reversible delivery, honest signals, rehearsed recovery and a blameless review afterwards are the working shape of the AWS DevOps Professional craft, which is why this lesson comes last.

Related