Automation runbooks

Codified, event-triggered operational tasks.

A runbook is a recipe card the kitchen follows the same way every single time. Here the cook is the AWS control plane, so it never skips a step at 2 a.m. under pressure, and it staples a receipt to every action it takes. In AWS Systems Manager (SSM, the service you use to operate and administer your servers and cloud resources), that recipe is an Automation document: a versioned YAML or JSON file that writes down a multi-step operational task. Patch and reboot, in that order. Snapshot a disk, then resize it. Rotate a credential, isolate a compromised EC2 (Elastic Compute Cloud, Amazon's virtual machines) instance, put an S3 (Simple Storage Service, the AWS object store) control back the way it belongs. The procedure runs identically whether a person clicks it or an event fires it at midnight. AWS ships hundreds of *managed* runbooks, named with the prefixes AWS- and AWSConfigRemediation-; you write *custom* ones for your own procedures. Either way, the operational knowledge that used to live on a wiki page and inside one senior engineer's head turns into reviewed, testable, reusable code.

This is infrastructure-as-code (writing your infrastructure down in files you commit, review and version, instead of clicking through a console) applied to operations, and you get three things out of it. Consistency: no skipped steps, no wrong order. Auditability: every execution and every individual step lands in the Automation history and in CloudTrail (the AWS record of who called which API, and when). Least privilege: the runbook runs under one dedicated IAM (Identity and Access Management) role, not under the wide permissions of whoever happened to press the button. That last property is the one that makes it safe to hand the runbook a trigger and walk away.

What's inside a runbook: steps, action providers, and a borrowed role

A runbook has four parts: a schemaVersion (0.3 is the one for Automation), a parameters block, an assumeRole, and an ordered mainSteps list. Every step names an action provider, which is the built-in verb that step runs. A handful come up constantly. aws:executeAwsApi calls any single AWS API (Application Programming Interface, the machine-callable version of every console button), and it is the Swiss-army step. aws:executeScript runs inline Python or PowerShell. aws:runCommand runs a command *inside* an instance through the SSM agent, the small program AWS installs on the machine itself. aws:branch does conditional routing. aws:approve pauses for a human. aws:waitForAwsResourceProperty polls until a resource reaches the state you named. Then there is assumeRole, usually passed in as the AutomationAssumeRole parameter. That is the identity every step executes under, and it is the security crux of the whole feature. Scope that role to exactly the API calls your steps make. When an *event* fires the runbook there is no human and no login session, so the role is the permission boundary, which makes it the blast radius too.

schemaVersion: '0.3'
description: Isolate a compromised EC2 instance and snapshot its volumes.
assumeRole: '{{ AutomationAssumeRole }}'
parameters:
  InstanceId:           { type: AWS::EC2::Instance::Id }
  QuarantineSgId:       { type: String }
  AutomationAssumeRole: { type: AWS::IAM::Role::Arn }
mainSteps:
  - name: assertRunning
    action: aws:assertAwsResourceProperty
    inputs:
      Service: ec2
      Api: DescribeInstances
      InstanceIds: ['{{ InstanceId }}']
      PropertySelector: '$.Reservations[0].Instances[0].State.Name'
      DesiredValues: ['running']
  - name: isolate            # aws:executeAwsApi = call exactly one API
    action: aws:executeAwsApi
    inputs:
      Service: ec2
      Api: ModifyInstanceAttribute
      InstanceId: '{{ InstanceId }}'
      Groups: ['{{ QuarantineSgId }}']
  - name: snapshotVolumes    # aws:executeScript = inline Python
    action: aws:executeScript
    inputs:
      Runtime: python3.11
      Handler: handler
      InputPayload: { InstanceId: '{{ InstanceId }}' }
      Script: |
        def handler(events, context):
            import boto3
            ec2 = boto3.client('ec2')
            r = ec2.describe_instances(InstanceIds=[events['InstanceId']])
            vols = [m['Ebs']['VolumeId']
                    for i in r['Reservations'][0]['Instances']
                    for m in i['BlockDeviceMappings']]
            for v in vols:
                ec2.create_snapshot(VolumeId=v,
                    Description=f"forensic-{events['InstanceId']}")
            return {'snapshotted': vols}

aws ssm create-document \
  --name "Custom-QuarantineInstance" \
  --document-type "Automation" \
  --document-format YAML \
  --content file://quarantine.yml
# {
#     "DocumentDescription": {
#         "Name": "Custom-QuarantineInstance",
#         "DocumentVersion": "1",
#         "Status": "Creating",
#         "DocumentType": "Automation",
#         "SchemaVersion": "0.3"
#     }
# }
 
# Publish a revised version later; executions can pin --document-version
# so a document edit never silently changes a scheduled job:
aws ssm update-document --name "Custom-QuarantineInstance" \
  --document-format YAML --content file://quarantine.yml \
  --document-version '$LATEST'

Run it on one instance, then on the fleet

start-automation-execution kicks off a run and hands you back an AutomationExecutionId. get-automation-execution shows you how far each step got. To act on many resources at once you use rate control. --targets picks resources by tag, --target-parameter-name feeds each match into a parameter, and --max-concurrency plus --max-errors throttle the blast. max-concurrency caps how many run at the same moment, either an absolute number or a percentage of the matched targets, and it defaults to a cautious 10. max-errors sets how many failures the batch will tolerate before it stops launching on new targets. Set it to 0 and the batch halts on the very first failure. Those two flags are your circuit breaker for fleet work. Treat them as safety devices, not throughput knobs.

aws ssm start-automation-execution \
  --document-name "Custom-QuarantineInstance" \
  --parameters '{"InstanceId":["i-0abc123def4567890"],
                 "QuarantineSgId":["sg-0quarantine01"],
                 "AutomationAssumeRole":["arn:aws:iam::111122223333:role/SsmAutomationRole"]}'
# {
#     "AutomationExecutionId": "e1a2b3c4-5d6e-7f80-9a1b-2c3d4e5f6a7b"
# }
 
aws ssm get-automation-execution \
  --automation-execution-id e1a2b3c4-5d6e-7f80-9a1b-2c3d4e5f6a7b \
  --query 'AutomationExecution.{Status:AutomationExecutionStatus,
           Steps:StepExecutions[].{Name:StepName,Status:StepStatus}}'
# {
#     "Status": "Success",
#     "Steps": [
#         { "Name": "assertRunning",   "Status": "Success" },
#         { "Name": "isolate",         "Status": "Success" },
#         { "Name": "snapshotVolumes", "Status": "Success" }
#     ]
# }

aws ssm start-automation-execution \
  --document-name "AWS-RestartEC2Instance" \
  --targets "Key=tag:PatchGroup,Values=web-prod" \
  --target-parameter-name InstanceId \
  --max-concurrency "10%" \
  --max-errors "1"
# { "AutomationExecutionId": "7f9c2a10-3b4d-4e5f-8a90-1b2c3d4e5f60" }
 
# Watch the aggregate rollout across every matched instance:
aws ssm describe-automation-executions \
  --filters "Key=ExecutionId,Values=7f9c2a10-3b4d-4e5f-8a90-1b2c3d4e5f60" \
  --query 'AutomationExecutionMetadataList[0].AutomationExecutionStatus'
# "Success"

Quotas and cost. An Automation document is capped at 64 KB of definition, and each account starts with a ceiling of 100 concurrently executing automations. That ceiling is a soft limit you can ask AWS to raise, up to 500 once *adaptive concurrency* is switched on. Rate-control fan-out counts against it, so a 5,000-instance run at --max-concurrency 50 sits comfortably underneath while --max-concurrency 100% (all 5,000 at once) does not. There is no free tier here. You pay per step, roughly $0.002 per step, plus about $0.00003 per second of runtime for aws:executeScript steps. In practice the runbook is almost never what shows up on the bill. What the steps *do* is what costs money: the EC2, KMS (Key Management Service, where your encryption keys live) and Lambda calls they make, and the CloudWatch Logs they write (CloudWatch is the AWS home for metrics, log lines and alarms).

Let events pull the trigger

The real multiplier is wiring runbooks to events. EventBridge (the AWS event bus that carries signals from one service to another) takes a signal and hands it to a runbook that responds in seconds. The signal might be an AWS Config rule flipping to NON_COMPLIANT (Config records the settings of your resources and grades them against rules), a GuardDuty finding (GuardDuty is the AWS threat-detection service), or a CloudWatch alarm. The runbook re-applies Block Public Access, quarantines an instance, restarts a service. For compliance drift the cleanest wiring is AWS Config remediation, which binds a runbook straight onto a rule so any resource that goes non-compliant gets fixed and then re-evaluated on its own. That is *self-healing infrastructure*: the system spots a problem it already knows how to fix, repairs it with nobody in the critical path, and writes down every repair. It is the change that moves a team from reactive firefighting to operations that live in version control.

# remediation.json binds an AWS-managed runbook to a Config rule:
cat remediation.json
# [
#   {
#     "ConfigRuleName": "s3-bucket-level-public-access-prohibited",
#     "TargetType": "SSM_DOCUMENT",
#     "TargetId": "AWSConfigRemediation-ConfigureS3BucketPublicAccessBlock",
#     "Automatic": true,
#     "MaximumAutomaticAttempts": 3,
#     "RetryAttemptSeconds": 60,
#     "Parameters": {
#       "AutomationAssumeRole": {"StaticValue": {"Values":
#         ["arn:aws:iam::111122223333:role/ConfigRemediationRole"]}},
#       "BucketName": {"ResourceValue": {"Value": "RESOURCE_ID"}}
#     }
#   }
# ]
 
aws configservice put-remediation-configurations \
  --remediation-configurations file://remediation.json
# {
#     "FailedBatches": []
# }
# Empty FailedBatches = the runbook is bound; non-compliant buckets now self-heal.

Guardrails: automate what you can undo, gate what you can't

Speed only helps if it doesn't cause an outage of its own. Wire every event to an unconditional action and a single false positive is enough to set off a destructive runbook. The discipline is a split. Auto-remediate the reversible, low-blast-radius cases: turn encryption back on, restart a service, re-apply Block Public Access. Route high-impact actions through an aws:approve step: deleting resources, disabling accounts, terminating instances. That step parks the execution until a named approver answers over SNS (Simple Notification Service, the AWS service that delivers the message). Pair it with tight rate control so that even an approved-but-wrong runbook cannot go off across the whole fleet at once.

Two habits separate a runbook you trust from one you dread. First, make every step idempotent, meaning you can run the whole thing twice and land in the same state instead of stacking up side effects. Retries and duplicate events are ordinary traffic, not rare accidents. Second, test it in a non-production account first, using --mode Interactive or a single tagged canary instance, before you let an event fire it while nobody is watching. A runbook is code. Give it the review, the version history and the staged rollout you would give any deployment you ship.

An Automation document is a runbook with a role attached. Reach for the AWS-owned documents whenever one already fits the job, and fork into a custom document only when your steps are genuinely your own. Version them the way you version code, and pin the version in production. Pointing a scheduled job at "latest" is how surprises ship.

EventBridge into Automation is what self-healing looks like in practice. A GuardDuty finding or a CloudWatch alarm starts a document, and the document swaps the instance into a quarantine security group or hands an open bucket to a Config remediation. The one thing you keep a person standing in front of is the step you cannot take back.

Concurrency and error targets are the flags that start to matter at fleet size. A document that patches 5,000 nodes with a concurrency of 5,000 is a self-inflicted outage with an execution ID attached to it.

Try this

In a lab account, list the Automation documents you can see, then go and look at a run that already happened. Every call below is read-only, so nothing here changes a resource.

aws ssm list-documents --filters Key=DocumentType,Values=Automation --query 'DocumentIdentifiers[:5].Name' --output table
aws ssm describe-automation-executions --max-results 3 \
  --query 'AutomationExecutionMetadataList[].{Id:AutomationExecutionId,Doc:DocumentName,Status:AutomationExecutionStatus}' --output table
aws ssm get-automation-execution --automation-execution-id 11111111-2222-3333-4444-555555555555 \
  --query 'AutomationExecution.{Status:AutomationExecutionStatus,Step:StepExecutions[].StepName}' --output json

AWS-PublishSnsNotification
AWSConfigRemediation-EnableS3BucketEncryption
---------------------------------------------
|     DescribeAutomationExecutions          |
+----------+--------------------+-----------+
|    Id    | Doc                | Status    |
+----------+--------------------+-----------+
| 1111-... | AWS-RestartEC2Instance | Success|
+----------+--------------------+-----------+

Takeaway

Write your operational procedures down as Automation documents, let events start them, scope the assume-role to exactly the API calls the steps make, and put aws:approve in front of anything you cannot take back. A wiki page has never once paged itself at 2 a.m.

Your next move: take one fix you currently do by hand while on call, turn it into a document, and put an approve step ahead of any terminate or revoke action inside it.

Runbooks are the *hands* of automated operations. The next lesson, AWS Config & governance, gives you the *eyes*: the rules that notice drift and non-compliance in the first place. The put-remediation-configurations binding you wired up above is exactly where the two shake hands. A Config rule spots the bucket that went public, and a runbook closes it.

Related