Drift detection

When reality diverges from the template.

A CloudFormation template is a blueprint: a plain-text file that lists every resource you want and how it should be set up. AWS CloudFormation (Amazon Web Services' service for building cloud infrastructure from that file) reads the blueprint and puts up the building. That live stack is the named bundle of real resources it creates: security groups (virtual firewalls), storage buckets, databases, and the rest. Then handover happens. The stack is in use, and one day someone walks in and moves a wall. They open a firewall port in the web console. They retag a bucket. They delete a subnet, and never touch the blueprint. The file still says one thing. The account says another. That gap is drift.

Drift detection is the survey crew you send to walk the building and mark every place the bricks no longer match the plans. It fixes nothing. It tells you what will surprise you the next time you deploy, and if you are defending the account, it tells you what changed while nobody was looking.

This is a different question from the one a change set answers (a change set is a preview of what a template edit would do before you apply it, covered in cf-changesets). A change set looks forward, at an edit you are about to make. Drift detection looks at the template you already deployed and compares it against the running account right now.

Kick Off a Detection Run

Detection is a job you kick off, not a guard on patrol. Nothing watches in the background. From the AWS CLI (command-line interface, the aws command you run in a shell) you ask for a scan of a whole stack, and CloudFormation goes out and reads the live settings of each resource type it supports. Starting the scan hands back a detection id, an identifier for this single run.

That call is asynchronous, which means it returns to your shell right away, before the scan has finished, and gives you a ticket instead of an answer. You then poll it, asking again on a short loop, until the run leaves the DETECTION_IN_PROGRESS state and settles on a result.

# Kick off one async drift scan for the whole stack
ID=$(aws cloudformation detect-stack-drift \
  --stack-name payments-prod \
  --query StackDriftDetectionId --output text)
 
echo "$ID"

a1b2c3d4-4e5f-11ef-9a2b-0e1f2a3b4c5d

# Wait until the scan finishes (usually a few seconds), then read the verdict
while [ "$(aws cloudformation describe-stack-drift-detection-status \
    --stack-drift-detection-id "$ID" \
    --query DetectionStatus --output text)" = "DETECTION_IN_PROGRESS" ]; do
  sleep 3
done
 
aws cloudformation describe-stack-drift-detection-status \
  --stack-drift-detection-id "$ID" \
  --query '{status:DetectionStatus, drift:StackDriftStatus, drifted:DriftedStackResourceCount}'

{
    "status": "DETECTION_COMPLETE",
    "drift": "DRIFTED",
    "drifted": 2
}

The headline is StackDriftStatus. DRIFTED means at least one resource no longer matches the template. IN_SYNC means everything CloudFormation could check still lines up. DriftedStackResourceCount is the tally. That is the summary verdict, and it tells you something moved, not what moved.

Read the Resource-Level Diff

The summary points at a problem; the resource-level results name it. Each resource carries a StackResourceDriftStatus: MODIFIED when a property changed, DELETED when the resource was removed out of band (deleted directly, outside CloudFormation), IN_SYNC when it still matches. Filter to the two that matter with --stack-resource-drift-status-filters so you are not reading past the noise.

# List only the resources that actually moved
aws cloudformation describe-stack-resource-drifts \
  --stack-name payments-prod \
  --stack-resource-drift-status-filters MODIFIED DELETED \
  --query 'StackResourceDrifts[].{id:LogicalResourceId, type:ResourceType, status:StackResourceDriftStatus}'

[
    {
        "id": "ApiSecurityGroup",
        "type": "AWS::EC2::SecurityGroup",
        "status": "MODIFIED"
    },
    {
        "id": "FlowLogsBucket",
        "type": "AWS::S3::Bucket",
        "status": "DELETED"
    }
]

Read that the way a defender would. ApiSecurityGroup came back MODIFIED and FlowLogsBucket came back DELETED. A firewall rule was edited, and on the same stack a logging bucket disappeared. Someone opened a hole and removed a place evidence would have been written. Each modified resource lists PropertyDifferences, and every difference names four things: the PropertyPath (which field moved), the ExpectedValue (what your template says), the ActualValue (what is live right now), and a DifferenceType of NOT_EQUAL (a value changed), ADD (something was added), or REMOVE (something was taken away).

# Re-check one resource live and show its property-level diff
aws cloudformation detect-stack-resource-drift \
  --stack-name payments-prod \
  --logical-resource-id ApiSecurityGroup \
  --query 'StackResourceDrift.PropertyDifferences'

[
    {
        "PropertyPath": "/SecurityGroupIngress/0/CidrIp",
        "ExpectedValue": "10.0.0.0/8",
        "ActualValue": "0.0.0.0/0",
        "DifferenceType": "NOT_EQUAL"
    }
]

There it is, in one line. Your template expected 10.0.0.0/8, and the account is serving 0.0.0.0/0. CIDR (Classless Inter-Domain Routing, the slash-notation for a block of IP addresses) makes the size plain: 10.0.0.0/8 is about 16 million private addresses you control, and 0.0.0.0/0 is every address on the internet. An inbound rule that used to admit only your own network now admits anyone, anywhere. That is the finding you want in front of you in minutes, not at the next quarterly review.

What Drift Detection Cannot See

Coverage is wide but not total, and the gaps are exactly where a careful attacker operates. Drift detection only inspects the resources this stack manages. A brand-new object created on the side, a standalone AWS::EC2::SecurityGroupIngress rule bolted onto your security group, or an IAM (Identity and Access Management, AWS's permissions system) policy attached to a role the stack does not own, never shows up, because the stack was never told it exists. A change inside a property the template does manage is different: add an entry to an inline ingress list and it surfaces as an ADD difference. The rule of thumb is that drift sees edits to what it built, and is blind to what it never built.

Which is why drift detection is one sensor, not the whole alarm system. Pair it with AWS Config recording every resource type and with CloudTrail (the log of every application programming interface (API) call made in your account), so the objects a stack never knew about are still seen by something. An attacker who edits a resource the stack manages leaves drift you can catch. An attacker who creates fresh resources, or works in a corner of the account the stack does not touch, leaves none. Build for both.

Make It Continuous

One manual scan tells you today's truth and nothing about tomorrow's. Because detection is on demand, teams that care about drift put it on a schedule. The lightest way is a managed rule (a prebuilt check AWS maintains and updates) in AWS Config (a service that continuously records your resource settings and checks them against rules). The rule re-runs detection and marks any stack it finds DRIFTED as noncompliant, so drift lands in the compliance dashboards and alerts you already watch instead of a CLI run somebody forgot. It pairs with a locked-down deployment pipeline (see cf-cicd): the pipeline keeps out-of-band changes from being the normal way to work, and the rule catches the ones that happen anyway.

Resources:
  StackDriftCheck:
    Type: AWS::Config::ConfigRule
    Properties:
      ConfigRuleName: cfn-stacks-must-not-drift
      Source:
        Owner: AWS
        SourceIdentifier: CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK
      InputParameters:
        # IAM role AWS Config assumes to call detect-stack-drift for you
        cloudformationRoleArn: !GetAtt ConfigDriftRole.Arn
      Scope:
        ComplianceResourceTypes:
          - AWS::CloudFormation::Stack
# Requires an active AWS Config configuration recorder in this account and region.

SourceIdentifier picks the AWS-owned check named CLOUDFORMATION_STACK_DRIFT_DETECTION_CHECK. cloudformationRoleArn is the ARN (Amazon Resource Name, the unique id of an AWS resource) of an IAM role that AWS Config assumes so it can call detect-stack-drift on your behalf. One requirement is easy to miss: the rule does nothing without an active configuration recorder (the AWS Config component that captures resource state) in that account and region.

Fixing Drift, and Proving You Fixed It

Detection reads and reports; it never changes a resource. Fixing drift is a separate decision, and the first question is which side is right. If the template is correct and the live change was a mistake, run a stack update to push the template's values back over reality. If the live change was intended, update the template to describe it and deploy that, so the blueprint and the building agree again and the next scan comes back clean.

One more trap catches people here, and it looks like success. CloudFormation compares your new template against the last template it deployed, not against the live account. Submit the exact same template and it reports 'No updates are to be performed,' runs nothing, and leaves the drift untouched. To make it re-assert a drifted value you have to change the template so CloudFormation actually touches that resource (even a small edit to its properties), or replace the resource outright. Do not assume the fix landed. Confirm it.

Confirming is one more detection run. It is read-only and cheap, so after any reconciliation, run detect-stack-drift again and check that the resource reports IN_SYNC. If it still shows MODIFIED, your update never reached it. One more operational note: drift on a nested stack (a stack created and managed by a parent stack, see cf-nested) is reported on the child stacks themselves, not summarized on the parent, so point detection at the children when a parent looks clean but you suspect otherwise.

Give deletions their own weight. A logging bucket or a flow log that comes back DELETED is an incident until someone proves it was planned, since wiping out where logs get written is a standard way to cover tracks. Put detection on a schedule, route every DRIFTED result into the alerts your team already reads, and re-run the scan after each fix. Let IN_SYNC be the thing you verified, not the thing you assumed.

Try this

Work through “Fixing Drift, and Proving You Fixed It” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: iN_SYNC is not 'nothing changed'. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related