Courses AWS DevOps Engineer ProfessionalResilience & incident response

Incident response & learning

Automated response and blameless post-mortems.

Advanced25 min · lesson 15 of 15

However well you automate, incidents happen — and the mark of a mature DevOps practice is how fast and consistently it detects, responds, and learns. Automated incident response and blameless post-mortems close the operations loop.

Automated detection and response

Fast incident response comes from connecting observability to automated action: CloudWatch alarms, Config findings, and GuardDuty detections emit events, EventBridge routes them, and Lambda or SSM Automation runbooks execute the response — scale a fleet, restart a service, quarantine a compromised resource, page on-call — in seconds. For anything but the simplest cases, codify the response as a rehearsed runbook (or Step Functions workflow) so it runs the same way every time rather than being improvised under pressure. This is the self-healing, hands-off operations model: the system handles known problems automatically, and humans engage for the novel and high-judgment ones, arriving to a contained situation with context already gathered.

automated response, then human judgment

# Detect → route → respond, automatically:
#   CloudWatch alarm (error spike)  → EventBridge → Lambda: scale + notify
#   GuardDuty finding (compromise)  → EventBridge → SSM runbook: isolate + snapshot
#   Config non-compliance           → EventBridge → runbook: remediate
#
# Codify multi-step response as a runbook / Step Functions workflow — rehearsed,
# consistent, logged. Humans handle the novel + high-judgment cases.

Learn: the blameless post-mortem

The loop closes with learning. After an incident, a blameless post-mortem focuses on the systems and conditions that allowed it — not on blaming people — and produces concrete actions: new alarms for signals you missed, automation for steps that were manual, fixes for the root cause, and runbook improvements. Each incident should make recurrence less likely and detection faster, feeding directly back into the observability, automation, and resilience practices from the rest of the course. This continuous-improvement mindset — measure, automate, learn, repeat — is the heart of DevOps: fast, safe, automated delivery backed by observability, resilience, and a culture that turns every incident into a durable improvement rather than a repeated surprise.

The incident loop

1detect

alarms, findings, events

2respond automatically

runbooks / Lambda in seconds

3engage humans

for novel, high-judgment cases

4blameless post-mortem

concrete fixes → prevent recurrence

Detect, respond automatically, and learn. Every incident becomes new alarms, automation, and fixes — the DevOps improvement loop.

An incident without a post-mortem is a wasted lesson

Closing an incident without asking what the system allowed, what was detected too late, and what response was manual throws away its most valuable output. Run a blameless post-mortem that produces concrete actions — new detections, automation, and root-cause fixes — so the same incident cannot silently recur.

HA & DR automationFinish · self-test