Drift detection

Scheduled plans and alerting on out-of-band changes.

A warehouse runs on a stock ledger. Every pallet in, every pallet out, signed for. After a while the ledger gets trusted so completely that nobody walks the aisles to count anything. Then one night a driver loads two pallets and skips the paperwork. The ledger still says forty. The shelves hold thirty-eight. Nothing has broken yet, and that is the problem: every decision from here on gets made from a document that is quietly wrong.

Terraform keeps a ledger too. It is called the **state file**, a JSON (JavaScript Object Notation, a plain-text way of writing down structured data) record of every resource Terraform manages and what each one looked like the last time Terraform checked. The paperwork is `terraform plan` and `terraform apply`. Change a cloud resource without going through those two commands and the ledger and the shelves stop agreeing. That gap is **drift**: any difference between the infrastructure that actually exists and what Terraform's state says exists. It shows up after a console click made under deadline pressure (ClickOps, as people call it), a script talking straight to the cloud API (Application Programming Interface, the machine-facing control surface where every button in the console really lives), another team's automation, or an attacker holding a stolen access key. **Drift detection** is walking the aisles on a schedule, on purpose, instead of tripping over the difference during your next deploy.

Why Your Policy Scanners Never See This

Every static check you have wired up so far reads code. `tfsec` and `checkov` parse `.tf` files. Open Policy Agent (a general-purpose rules engine that you feed your own data and your own policies) evaluates a plan generated from those same files. All of them answer one question: is what we intend to build safe? A security group opened to `0.0.0.0/0` through the AWS console never appears in a `.tf` file, never lands in a pull request, and never reaches a scanner. Your pipeline stays green. Production stops matching anything anybody reviewed.

That gap is where a patient attacker lives. Someone with a working access key can widen a security group, attach a policy to a role, or flip a storage bucket to public, and the code-review trail stays spotless. Git blames nobody. The pull request history looks perfect. Drift detection is the only check in your toolchain that inspects deployed reality instead of stated intent, which makes it the only one that can see this at all.

There is a second failure, quieter and worse. `terraform apply` drags reality back toward the code, so drift nobody noticed gets reverted silently on the next deploy. That sounds like self-healing until you ask what got reverted. If the change was an engineer's 2 a.m. mitigation, the apply reopens the incident. If it was an attacker's way back in, the apply wipes the evidence before anyone photographed the scene.

What A Plan Actually Compares

`terraform plan` is a three-way comparison: your configuration, the state file, and live infrastructure. The live view comes from a step called **refresh**. For every resource already written down in state, Terraform asks the provider plugin (the code that knows how to speak AWS, Azure, GitHub, and so on) what that object looks like right now, and writes the answers into an in-memory copy of state. Reality versus state is drift. Configuration versus state is the pending change you have not applied yet. A normal plan mixes both into one diff and decides for itself which outside changes are worth mentioning, which is why the JSON plan format carries a `relevant_attributes` list whose documented job is to filter drift down to the parts that actually affected the plan.

`terraform plan -refresh-only` pulls the two apart. It reports drift and proposes no infrastructure changes at all, which is exactly what you want from a job whose only purpose is to look. It needs Terraform 0.15.4 or newer. Pair it with `-refresh=false` and Terraform refuses outright, telling you it makes no sense to use both because it would have nothing left to do. The old standalone `terraform refresh` command is deprecated, and the docs now send you to `-refresh-only` on `plan` and `apply` instead, because `refresh` rewrote state with no review step in the middle.

# Reality changes outside Terraform: a console click, a script,
# or somebody using a key they should not have.
aws ec2 authorize-security-group-ingress \
  --group-id sg-0a1b2c3d4e5f6a7b8 \
  --protocol tcp --port 22 --cidr 0.0.0.0/0

{
    "Return": true,
    "SecurityGroupRules": [
        {
            "SecurityGroupRuleId": "sgr-0c4d8b7a1e6f39204",
            "GroupId": "sg-0a1b2c3d4e5f6a7b8",
            "GroupOwnerId": "123456789012",
            "IsEgress": false,
            "IpProtocol": "tcp",
            "FromPort": 22,
            "ToPort": 22,
            "CidrIpv4": "0.0.0.0/0"
        }
    ]
}

# Now ask Terraform to compare its ledger against the shelves.
terraform plan -refresh-only -lock=false -input=false -no-color

aws_security_group.app: Refreshing state... [id=sg-0a1b2c3d4e5f6a7b8]
 
Note: Objects have changed outside of Terraform
 
Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:
 
  # aws_security_group.app has changed
  ~ resource "aws_security_group" "app" {
        id      = "sg-0a1b2c3d4e5f6a7b8"
      ~ ingress = [
          + {
              + cidr_blocks      = ["0.0.0.0/0"]
              + description      = ""
              + from_port        = 22
              + ipv6_cidr_blocks = []
              + prefix_list_ids  = []
              + protocol         = "tcp"
              + security_groups  = []
              + self             = false
              + to_port          = 22
            },
            # (1 unchanged element hidden)
        ]
        name    = "app-sg"
        # (7 unchanged attributes hidden)
    }
 
 
This is a refresh-only plan, so Terraform will not take any actions to undo
these. If you were expecting these changes then you can apply this plan to
record the updated values in the Terraform state without changing any remote
objects.

Read that closing paragraph slowly, because it is the whole safety property. A refresh-only plan will not undo anything and will not change one byte of your infrastructure. Be precise about what that does and does not mean, though. Refresh still calls the cloud, at least once per managed resource, and every one of those calls is a read. Nothing gets created, modified or destroyed. It is a report. The only cost is API traffic, and later in this lesson that traffic turns out to matter.

Turning Drift Into Something A Machine Can Trust

Nobody reads nightly logs. Add `-detailed-exitcode` and the plan starts speaking in exit codes instead: `0` means everything matched, `1` means the command errored, `2` means there is a diff. A scheduler can branch on that.

Do not build your alerting on the exit code alone. Refresh-only combined with `-detailed-exitcode` has a repeat history of returning `2` while the output plainly says "No changes". Terraform issue 35117 was filed against 1.8.1, issue 36403 against 1.10.5, issue 37406 against 1.12.2. Provider-side value normalisation and output type comparisons both set it off. The most recent fix landed for the 1.15 series, so a current binary should behave itself, and the workflow below pins one. The bug still came back three times across three release lines, and you do not control which version every workspace in the company runs. If your pager fires on a bare exit code, you will train the on-call to ignore it inside a month, and alert fatigue is exactly how real drift walks past a team.

So save the plan and read the structured version. Terraform's JSON plan format carries a top-level `resource_drift` array, documented as the changes Terraform detected when it compared the most recent state to the prior saved state. That array, not the exit code, is your evidence.

terraform plan -refresh-only -detailed-exitcode \
  -lock=false -input=false -no-color -out=drift.tfplan
echo "exit=$?"

aws_security_group.app: Refreshing state... [id=sg-0a1b2c3d4e5f6a7b8]
 
Note: Objects have changed outside of Terraform
[... full drift report elided ...]
 
Saved the plan to: drift.tfplan
 
To perform exactly these actions, run the following command to apply:
    terraform apply "drift.tfplan"
 
exit=2

# jq is a command-line JSON processor; apt install jq if you lack it.
terraform show -json drift.tfplan \
  | jq '[.resource_drift[] | {address, type, actions: .change.actions}]'

[
  {
    "address": "aws_security_group.app",
    "type": "aws_security_group",
    "actions": [
      "update"
    ]
  }
]

Exit code `2` with an empty array means you hit the false positive, and the job should stay quiet. Entries in the array mean real drift with resource addresses attached, which is something you can route, rank and ticket. If you would rather have a live feed than a saved file, the streaming JSON output emits one `resource_drift` message per changed resource as it goes.

terraform plan -refresh-only -lock=false -input=false -json \
  | jq -r 'select(.type == "resource_drift") | .["@message"]'

aws_security_group.app: Drift detected (update)

The Scheduled Job And The Key It Holds

A drift job runs unattended, in the middle of the night, holding credentials to production. Treat it like a night watchman: give them a torch and a radio, not the keys to the safe. A plan only reads, so the job's cloud role should be read-only, plus read access to wherever the state file lives. Do not tell yourself that read-only makes a compromise harmless. Whoever steals that role walks away with a complete map of your infrastructure and, through the state backend, a file stuffed with secrets. It is a far better day for you than write access would have been. It is still a bad day.

name: drift-detection
 
on:
  schedule:
    - cron: "17 6 * * *"    # 06:17 UTC; off-the-hour spreads provider API load
  workflow_dispatch: {}
 
permissions:
  id-token: write           # mint an OIDC token for AWS; no long-lived keys
  contents: read
 
jobs:
  drift:
    runs-on: ubuntu-24.04
    steps:
      - uses: actions/checkout@v4
 
      - uses: hashicorp/setup-terraform@v3
        with:
          terraform_version: 1.15.8
          terraform_wrapper: true      # required for steps.<id>.outputs.exitcode
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/tf-drift-readonly
          aws-region: eu-west-1
 
      # -lockfile=readonly: a drift job must never upgrade a provider version
      - run: terraform init -input=false -lockfile=readonly
 
      - name: Refresh-only plan
        id: drift
        continue-on-error: true        # exit 2 is data, not a build failure
        run: |
          terraform plan -refresh-only -detailed-exitcode \
            -lock=false -input=false -no-color -out=drift.tfplan
 
      - name: Confirm real drift, then alert
        if: steps.drift.outputs.exitcode == '2'
        run: |
          if ! terraform show -json drift.tfplan \
               | jq -e '.resource_drift | length > 0' > /dev/null; then
            echo "::notice::exit 2 with empty resource_drift - nothing to report"
            exit 0
          fi
          terraform show -json drift.tfplan \
            | jq -r '.resource_drift[].address' \
            | while read -r addr; do echo "::error::drift detected: $addr"; done
          exit 1

Four details in that file are doing real work. `terraform_wrapper: true` is the only reason `steps.drift.outputs.exitcode` exists: the action installs a shell wrapper around the `terraform` binary that captures stdout, stderr and the exit code as step outputs, and switching the wrapper off makes that `if` condition silently never match. `-lock=false` skips the state lock, because a read-only plan does not need one and a nightly check should never block a real deploy or sit waiting behind one. `-lockfile=readonly` stops the job quietly pulling a newer provider, though it will fail the init if your committed `.terraform.lock.hcl` holds no checksums for the runner's platform, so generate them once with `terraform providers lock -platform=linux_amd64`. And `id-token: write` is what lets the runner use OIDC (OpenID Connect, a standard way for one system to prove a workload's identity to another without any shared password) instead of a static access key.

One more thing before you point that job at production. A refresh-only plan prints resource attributes, and providers put real values in them: connection strings, generated passwords, private key material. Terraform masks whatever the provider flagged as sensitive, but plenty of secrets sit in attributes nobody flagged. Treat the job log exactly like the state file. Do not upload `drift.tfplan` with `actions/upload-artifact`, do not echo a raw plan into a chat webhook, and remember that in a public repository the Actions logs are public too.

OIDC is only as good as the trust policy at the other end. The condition below is the part people get wrong. Pin the `sub` claim (short for subject, the field that names exactly which workflow is asking) with `StringEquals` to one repository and one branch. A wildcard there means any GitHub Actions workflow on the internet can assume your production role.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
      },
      "Action": "sts:AssumeRoleWithWebIdentity",
      "Condition": {
        "StringEquals": {
          "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
          "token.actions.githubusercontent.com:sub": "repo:acme/infra:ref:refs/heads/main"
        }
      }
    }
  ]
}

The First Question Is Who, Not What

An alert that says "a security group changed" is half a finding. The half that decides your next hour is who changed it, when, and from where. In AWS that answer lives in CloudTrail, the service that writes down every API call made in your account, and you can pivot straight from the resource ID in the drift report to the events that touched it.

aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0a1b2c3d4e5f6a7b8 \
  --start-time 2026-07-20T00:00:00Z \
  --max-results 5 \
  --query 'Events[].CloudTrailEvent' --output text \
  | jq -r '[.eventTime, .eventName, .userIdentity.arn, .sourceIPAddress] | @tsv'

2026-07-20T22:14:03Z	AuthorizeSecurityGroupIngress	arn:aws:sts::123456789012:assumed-role/BreakGlassAdmin/alex.rivera	203.0.113.47
2026-07-19T09:02:55Z	CreateTags	arn:aws:sts::123456789012:assumed-role/tf-apply/GitHubActions	18.130.44.12

Now you have a name, a timestamp and a source address. A break-glass role (the emergency identity people assume when their normal access is not enough, deliberately noisy to use) driven by a named engineer at 22:14 is a conversation over coffee tomorrow. The same event from an access key nobody knew was still live, from an address in a country you do not operate in, is an incident, and the drifted resource is now evidence you must not touch.

Two Ways To Close The Loop

Once a human has classified the drift, there are only two honest endings. If reality is wrong, revert it through the normal pipeline: a standard plan works out exactly the steps that put the resource back as coded, and a reviewer sees that diff before anything runs. If reality is right, codify it: edit the `.tf` so the code matches what is deployed, get that reviewed, then accept the drifted values into the record with a refresh-only apply. That second command writes state and changes nothing in the cloud. It still reads from the cloud, because it refreshes first, but it will not create, modify or destroy a single object.

# Ending A: reality is wrong. Revert via a reviewed, normal plan.
terraform plan -input=false -out=revert.tfplan

aws_security_group.app: Refreshing state... [id=sg-0a1b2c3d4e5f6a7b8]
 
Note: Objects have changed outside of Terraform
[... drift report elided ...]
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
 
Terraform will perform the following actions:
 
  # aws_security_group.app will be updated in-place
  ~ resource "aws_security_group" "app" {
        id      = "sg-0a1b2c3d4e5f6a7b8"
      ~ ingress = [
          - {
              - cidr_blocks = ["0.0.0.0/0"]
              - from_port   = 22
              - protocol    = "tcp"
              - to_port     = 22
            },
            # (1 unchanged element hidden)
        ]
        # (7 unchanged attributes hidden)
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.
 
Saved the plan to: revert.tfplan
 
To perform exactly these actions, run the following command to apply:
    terraform apply "revert.tfplan"

# Ending B: reality is right. Edit the .tf to match it first, then
# record the new values. This changes STATE ONLY, never infrastructure.
terraform apply -refresh-only

aws_security_group.app: Refreshing state... [id=sg-0a1b2c3d4e5f6a7b8]
 
Note: Objects have changed outside of Terraform
[... drift report elided ...]
 
Would you like to update the Terraform state to reflect these detected
changes?
  Terraform will write these changes to the state without modifying any real
  infrastructure.
  There is no undo. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Apply complete! Resources: 0 added, 0 changed, 0 destroyed.

Where Drift Detection Is Blind

A plan can only inspect resources already written down in state. A rogue IAM (Identity and Access Management, the service that decides who is allowed to do what in your account) user, an unmanaged virtual machine an attacker spun up, a security group conjured out of thin air: Terraform never knew those objects existed, so refresh never asks about them and no exit code is ever produced. Catching unmanaged resources needs an inventory layer sitting above Terraform, something like AWS Config, Azure Resource Graph, or a CSPM (Cloud Security Posture Management, a tool that continuously inventories a cloud account and grades it against rules) product. The open-source `driftctl` was built for exactly this job and its README now says the project is in maintenance mode, so plan accordingly. HCP Terraform (HashiCorp Cloud Platform, the hosted version of Terraform) covers managed drift through its health assessments feature on the Standard and Premium editions.

The second blind spot is one you built yourself. A `lifecycle { ignore_changes = [...] }` block tells Terraform to stop comparing specific attributes, permanently. Every entry is a hiding place, so audit that list the way you would audit a list of doors somebody removed the alarm sensors from.

grep -rn --include='*.tf' 'ignore_changes' .

./modules/app/main.tf:41:    ignore_changes = [tags, ingress]
./modules/rds/main.tf:78:    ignore_changes = [password, engine_version]
./network/nat.tf:23:    ignore_changes = all

Line 41 means your nightly job will never report a port opened on that security group. Line 78 means a rotated database password never surfaces. Line 23 means that whole resource is invisible to drift detection forever. None of these are automatically wrong. Every one of them needs a comment naming who decided it and why.

Scale is the third limit. Refresh costs at least one provider read per resource, so a workspace holding two thousand objects makes at least two thousand API calls, and dozens of workspaces all firing at 06:00 will hit provider rate limits and turn a five-minute job into an hour of `RequestLimitExceeded` retries. Stagger the cron minute (cron is the Unix scheduler that runs a job at a fixed time; the minute field is the first number) across workspaces, drop `-parallelism` below its default of 10 if you are getting throttled, and consider splitting the schedule: an hourly `-target` run against the handful of resources that matter most for security, plus one full sweep overnight.

One last assumption worth saying out loud. Drift detection compares reality against state, which means it trusts state completely. Anyone with write access to your state backend can edit the ledger to match the shelves, and the nightly job will hand back a clean bill of health. Versioning and object lock on the state bucket, plus an alarm on any write that did not come from your apply role, are what keep that door shut.

Prove The Detector Actually Fires

An untested detector is a decoration. Once a quarter, plant a canary: a small, deliberate, reversible change made purely so you can watch the alarm go off. Then confirm the whole pipe reacted, the plan, the exit code, the JSON and the alert.

# Deliberate, reversible drift on a resource Terraform manages
aws ec2 create-tags --resources sg-0a1b2c3d4e5f6a7b8 \
  --tags Key=drift-canary,Value=2026-07-21
 
terraform plan -refresh-only -lock=false -input=false \
  -detailed-exitcode -out=canary.tfplan > /dev/null
echo "exit=$?"
terraform show -json canary.tfplan | jq '.resource_drift | length'
 
aws ec2 delete-tags --resources sg-0a1b2c3d4e5f6a7b8 --tags Key=drift-canary

exit=2
1

If that comes back `exit=0` and `0`, your detector is asleep. The usual culprit is an `ignore_changes` entry covering `tags`. Finding it during a drill costs you ten minutes. Finding it during an investigation costs you the investigation.

Preventing drift beats detecting it wherever you can get it. Make the production console read-only with an AWS Service Control Policy (an organisation-wide guardrail that a local account admin cannot switch off) or Azure Policy, and put emergency write access behind a break-glass role whose use posts to your security channel the second it happens. Then rank whatever still slips through by blast radius, meaning how much damage that change could do if it turns out to be hostile: a changed IAM policy, security group or KMS (Key Management Service, where your encryption keys and the rules about who may use them live) key earns an incident, a changed tag earns a ticket. The next lesson looks at the other half of this problem, the modules and providers your configuration downloads and then runs with those same production credentials.

Related