Drift detection & remediation
When reality diverges from code.
Your office has a floor plan pinned to the wall. It says the back door is locked and the server room needs a badge. Then a contractor props the back door open on a hot afternoon and nobody closes it. The floor plan still says locked. It is now fiction, and everyone who trusts it is being quietly misled. Infrastructure fails the same way, and the failure has a name: drift.
Infrastructure as Code (IaC, meaning you describe servers, networks and cloud resources in version-controlled text files instead of clicking around a web console) promises that the files tell you what exists. Drift is any gap between that promise and reality. Someone opened a firewall rule at 2am to unblock a customer. An operator ran kubectl edit (kubectl is the command-line client for Kubernetes) against a live object. A backup tool attached a policy to a role. Those are out-of-band changes, meaning changes that did not come through the pipeline. Terraform records what it built in a state file, a JSON ledger mapping every resource in your code to a real resource ID in the cloud. Once reality stops matching that ledger, your code lies about what exists, your next apply does something nobody predicted, and an attacker's change looks exactly like a colleague's.
Seeing Drift in a Plan
The command that finds drift is one you already run. terraform plan reads your state file, asks the cloud provider what those resources look like right now, and reports the difference. Two flags turn it into a detector a script can act on. The flag -refresh-only says: do not propose changes to my infrastructure, only tell me how reality has moved away from my recorded state. The flag -detailed-exitcode changes the exit code so a machine can read the verdict, where 0 means the diff is empty, 1 means the command itself errored, and 2 means there is a diff. Add -lock=false, because a read-only check should never hold the state lock and block a real deploy, and -no-color so the output stays readable in a log. A plan refreshes in memory only. It does not write the refreshed values back to your state backend, and that is what makes it safe to point at production on a schedule.
# does reality still match what Terraform recorded?$ cd /opt/infra/live/prod$ terraform plan -refresh-only -detailed-exitcode -lock=false -no-color
aws_instance.app: Refreshing state... [id=i-04f3a9c2b1d7e5a60]aws_security_group.web: Refreshing state... [id=sg-0a1b2c3d4e5f67890]Note: Objects have changed outside of TerraformTerraform detected the following changes made outside of Terraform since thelast "terraform apply" which may have affected this plan:# aws_security_group.web has changed~ resource "aws_security_group" "web" {id = "sg-0a1b2c3d4e5f67890"name = "web-sg"+ ingress {+ cidr_blocks = [+ "0.0.0.0/0",]+ description = "temp"+ from_port = 22+ ipv6_cidr_blocks = []+ prefix_list_ids = []+ protocol = "tcp"+ security_groups = []+ self = false+ to_port = 22}# (1 unchanged block hidden)# (8 unchanged attributes hidden)}This is a refresh-only plan, so Terraform will not take any actions to undothese. If you were expecting these changes then you can apply this plan torecord the updated values in the Terraform state without changing any remoteobjects.─────────────────────────────────────────────────────────────────────────────Note: You didn't use the -out option to save this plan, so Terraform can'tguarantee to take exactly these actions if you run "terraform apply" now.
Read that carefully. Terraform is not offering to fix anything, because you asked for refresh-only. It is telling you that port 22 (SSH, the remote shell port) is now reachable from 0.0.0.0/0, which means every address on the internet, on a security group your code restricts to the office range. Nothing in Git changed. Reality changed. Someone typed a description of "temp" and went home.
# the exit code is the part a script can act on$ terraform plan -refresh-only -detailed-exitcode -lock=false -no-color >/dev/null; echo "exit=$?"# and the drift itself is machine-readable$ terraform plan -refresh-only -lock=false -no-color -out=plan.bin >/dev/null$ terraform show -json plan.bin \| jq -r '.resource_drift[] | "\(.address) \(.change.actions | join(","))"'
exit=2aws_security_group.web update
That jq line (jq is a command-line JSON processor) is the reason to save the plan file. resource_drift is a top-level array in Terraform's JSON plan output, one entry per resource whose real state moved, each carrying the resource address and the action that would reconcile it. Post it to a chat channel, open a ticket from it, or count it as a metric. What you should not do is grep the human-readable text. That formatting changes between Terraform versions, and it was written for people, not parsers.
Give It a Timer, Not a Reminder
A smoke alarm that only works when somebody remembers to press the test button is decoration. Drift detection is the same. Hand the schedule to the machine. On a modern Linux box that means a systemd timer (systemd is the program running as process ID 1 that starts and supervises everything else on the system, and its timers are the built-in replacement for cron). Two small files, and the check runs whether or not anyone is thinking about it.
[Unit]Description=Terraform drift detection (prod)Wants=network-online.targetAfter=network-online.target[Service]Type=oneshotUser=driftGroup=drift# systemd creates and owns /var/lib/tf-drift for this unit,# and exports its path to the process as $STATE_DIRECTORYStateDirectory=tf-driftWorkingDirectory=/var/lib/tf-drift# Type=oneshot has NO start timeout by default, so a wedged run hangs foreverTimeoutStartSec=20min# a profile naming a read-only role, kept outside the home directoryEnvironment=AWS_CONFIG_FILE=/etc/tf-drift/aws-configEnvironment=AWS_PROFILE=drift-readonlyEnvironment=TF_IN_AUTOMATION=1Environment=CHECKPOINT_DISABLE=1Environment=TF_PLUGIN_CACHE_DIR=/var/lib/tf-drift/plugin-cacheExecStart=/usr/local/bin/tf-drift-check# exit 2 means "drift found", not "the unit is broken"SuccessExitStatus=2# the checker must not be able to change anything, including this hostNoNewPrivileges=yesPrivateTmp=yesProtectSystem=strictProtectHome=yesProtectKernelTunables=yesRestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX
[Unit]Description=Nightly Terraform drift detection[Timer]OnCalendar=*-*-* 03:00:00# spread the load and avoid a thundering herd of API callsRandomizedDelaySec=15m# if the box was off at 03:00, run at next bootPersistent=true[Install]WantedBy=timers.target
Five details there earn their place. Type=oneshot tells systemd this is a task that runs and exits, not a daemon to keep alive. StateDirectory=tf-drift makes systemd create /var/lib/tf-drift owned by the drift user, which gives the job somewhere writable while ProtectSystem=strict keeps the rest of the filesystem read-only to it. SuccessExitStatus=2 is the one people trip over: without it, exit code 2 marks the unit failed, and your journal fills with red for the completely normal case of finding drift. TimeoutStartSec is there because oneshot units have no start timeout at all by default, so a plan that wedges against a hanging cloud API stays wedged until a human notices. And AWS_CONFIG_FILE exists because ProtectHome=yes hides the home directory, taking ~/.aws with it. Put the profile somewhere root owns, name it explicitly, and the sandbox stops fighting the credentials.
One quirk of the file format bites everyone once. systemd has no inline comments. Every # sits on its own line. A comment tacked onto the end of a value becomes part of the value, and you spend an afternoon working out why the job is looking for a directory called "/var/lib/tf-drift # the working dir".
The identity this runs as should be read-only in the cloud, and genuinely read-only. A machine that wakes up at 3am holding permission to change production is a lovely thing for an attacker to find. Better still, the drift-readonly profile holds no keys of its own. It names a read-only role and picks up the machine's own identity from the instance metadata service to assume it, so there is nothing on disk worth stealing. Drift checking needs to read the state backend and describe resources. It never applies anything.
#!/usr/bin/env bash# Read-only drift check. Compares reality against Git HEAD. Applies nothing.# Deliberately no `set -e`: exit code 2 from terraform is a result, not a crash.set -uo pipefailREPO="https://github.com/example/infra.git"WORK="${STATE_DIRECTORY:-/var/lib/tf-drift}"DIR="$WORK/repo"export TF_PLUGIN_CACHE_DIR="${TF_PLUGIN_CACHE_DIR:-$WORK/plugin-cache}"mkdir -p "$TF_PLUGIN_CACHE_DIR"# let clone and init errors reach stderr, so they land in the journalrm -rf "$DIR"git clone --quiet --depth 1 --branch main "$REPO" "$DIR" || exit 1cd "$DIR/live/prod" || exit 1terraform init -input=false -lockfile=readonly >/dev/null || exit 1terraform plan -refresh-only -input=false -lock=false -no-color \-detailed-exitcode -out=plan.bin >/dev/nullrc=$?if [ "$rc" -eq 2 ]; thenterraform show -json plan.bin \| jq -r '.resource_drift[] | "DRIFT \(.address) \(.change.actions | join(","))"'fiexit "$rc"
Cloning fresh from Git on every run is deliberate. If the checker compares reality against a local checkout somebody edited six weeks ago, you are measuring drift against a stale copy and the result means nothing. Git is the reference, so read the reference from Git. The -lockfile=readonly flag on init makes the run fail rather than silently rewrite your provider version pins, which is a small supply-chain guard on a job nobody watches.
$ sudo systemctl daemon-reload$ sudo systemctl enable --now tf-drift.timer
Created symlink /etc/systemd/system/timers.target.wants/tf-drift.timer → /etc/systemd/system/tf-drift.timer.
# the next morning: did it fire, and when does it fire again?$ systemctl list-timers tf-drift.timer
NEXT LEFT LAST PASSED UNIT ACTIVATESWed 2026-07-22 03:11:22 UTC 9h left Tue 2026-07-21 03:07:41 UTC 14h ago tf-drift.timer tf-drift.service1 timers listed.Pass --all to see loaded but inactive timers, too.
# what did last night's run actually say?$ journalctl -u tf-drift.service --since "yesterday" --no-pager
Jul 21 03:07:41 ops-runner-01 systemd[1]: Starting Terraform drift detection (prod)...Jul 21 03:08:16 ops-runner-01 tf-drift-check[24187]: DRIFT aws_security_group.web updateJul 21 03:08:16 ops-runner-01 systemd[1]: tf-drift.service: Deactivated successfully.Jul 21 03:08:16 ops-runner-01 systemd[1]: Finished Terraform drift detection (prod).
Drift Is a Detection Signal, So Ask Who Did It
Reverting the change and moving on is the mistake. A security group that opened itself is either a colleague in a hurry or somebody with stolen credentials building a way back in, and in a plan those two look identical. The difference lives in the door log. Every office building has one: a badge reader that quietly records who went through which door and when. In AWS that log is CloudTrail, and it records every API call (application programming interface call, the request that every tool, script and console click eventually sends to AWS) made in your account.
# who touched this security group in the last few days?$ aws cloudtrail lookup-events \--lookup-attributes AttributeKey=ResourceName,AttributeValue=sg-0a1b2c3d4e5f67890 \--start-time 2026-07-18T00:00:00Z \--query 'Events[].[EventTime,EventName,Username]' --output text# then pull the full record for the suspicious one$ aws cloudtrail lookup-events \--lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress \--max-results 1 --query 'Events[0].CloudTrailEvent' --output text \| jq '{who: .userIdentity.arn, ip: .sourceIPAddress, agent: .userAgent,mfa: .userIdentity.sessionContext.attributes.mfaAuthenticated}'
2026-07-19T22:41:08+00:00 AuthorizeSecurityGroupIngress ops-oncall2026-07-14T09:02:55+00:00 AuthorizeSecurityGroupIngress terraform-apply{"who": "arn:aws:sts::123456789012:assumed-role/BreakGlassAdmin/ops-oncall","ip": "203.0.113.47","agent": "console.ec2.amazonaws.com","mfa": "false"}
Four fields decide your next move. The ARN (Amazon Resource Name, the unique identifier of the principal that made the call) shows the break-glass admin role, not the pipeline role behind the legitimate change five days earlier. The user agent points at the EC2 console, so a person clicked this rather than a pipeline running it. For comparison, calls from the Terraform AWS provider carry a user agent containing APN/1.0 HashiCorp/1.0 Terraform/, which makes the two easy to tell apart. The source IP tells you whether that person was where you expect. And mfaAuthenticated is the string "false", which should turn this from a cleanup into an incident.
Two constraints worth knowing before you lean on that field. For sessions that came from an external identity provider, mfaAuthenticated often reads "false" even when the human did use MFA (multi-factor authentication, the second proof beyond a password), because the second factor happened at the provider and not at AWS. So treat it as a strong hint in a plain IAM setup and a weak one in a federated single sign-on setup. The second constraint is time: lookup-events searches only the last 90 days of management events, so a quarterly drift review can find the change long after the evidence for who made it has aged out. Nightly is the floor.
Kubernetes: A Guard Who Walks the Corridor
Terraform checks on a schedule. Argo CD checks continuously, like a guard who walks the same corridor every few minutes and re-locks whatever got opened. It is a GitOps controller, meaning a program that lives inside your Kubernetes cluster and treats one Git repository as the only legitimate description of what should be running. It compares the live objects in the cluster against the manifests in Git, marks the application OutOfSync when they differ, and, if you let it, puts them back.
apiVersion: argoproj.io/v1alpha1kind: Applicationmetadata:name: platform-infranamespace: argocdspec:project: platformsource:repoURL: https://github.com/example/infra.gittargetRevision: mainpath: clusters/proddestination:server: https://kubernetes.default.svcnamespace: platformsyncPolicy:automated:selfHeal: true # revert live changes that are not in Gitprune: true # delete live objects Git no longer declaressyncOptions:- CreateNamespace=true
selfHeal: true is what turns detection into automatic remediation. The controller re-syncs from Git about five seconds after it spots a difference, tunable with --self-heal-timeout-seconds on the application controller. It re-compares everything every three minutes by default, set by timeout.reconciliation in the argocd-cm ConfigMap, though it usually notices sooner because it also watches the Kubernetes API for change events. prune: true lets it delete live objects that Git no longer declares, which is powerful and exactly as dangerous as it sounds on a cluster where other tooling creates objects too.
$ kubectl get applications.argoproj.io -n argocd$ argocd app diff platform-infra
NAME SYNC STATUS HEALTH STATUSplatform-infra OutOfSync Healthy===== rbac.authorization.k8s.io/ClusterRoleBinding /platform-admins ======11,13d10< - apiGroup: rbac.authorization.k8s.io< kind: User< name: [email protected]
Argo CD runs that comparison as diff <live> <git>, so lines marked < are what the cluster actually has and lines marked > are what Git says it should have. This one says somebody added a user to a ClusterRoleBinding (a cluster-wide grant of a role to a set of users or service accounts) by talking to the API server directly, and Git knows nothing about it. That is a textbook persistence move: one small object, no new workload, no image for a scanner to catch, and it survives pod restarts and node reboots. With self-heal on, it does not survive Argo CD.
$ kubectl -n argocd logs statefulset/argocd-application-controller \--since=5m | grep platform-infra
time="2026-07-21T03:11:41Z" level=info msg="Refreshing app status (comparison expired), level (2)" application=argocd/platform-infratime="2026-07-21T03:11:41Z" level=info msg="Comparing app state (cluster: https://kubernetes.default.svc, namespace: platform)" application=argocd/platform-infratime="2026-07-21T03:11:46Z" level=info msg="Initiated automated sync to '9f4c1ab'" application=argocd/platform-infratime="2026-07-21T03:11:48Z" level=info msg="Updating operation state. phase: Running -> Succeeded, message: '' -> 'successfully synced (all tasks run)'" application=argocd/platform-infratime="2026-07-21T03:11:48Z" level=info msg="Update successful" application=argocd/platform-infra
Which Side Wins
Most drift is not an attack, so you need a rule for the ordinary case. The default is that the code wins: apply, and put reality back. The exception is the manual change that was correct and the code that was wrong, where a thoughtless revert takes production down for the second time in one night. That call belongs to a person who understands the change, and it is worth making deliberately instead of letting whoever runs the next apply decide by accident.
Reverting one resource looks like this. A targeted apply is the emergency form of the fix, and Terraform will tell you so in the output. The everyday form is re-running your normal pipeline apply, because that keeps the audit trail in the same place as every other change.
$ terraform apply -target=aws_security_group.web
aws_security_group.web: Refreshing state... [id=sg-0a1b2c3d4e5f67890]Terraform used the selected providers to generate the following executionplan. Resource actions are indicated with the following symbols:~ update in-placeTerraform will perform the following actions:# aws_security_group.web will be updated in-place~ resource "aws_security_group" "web" {id = "sg-0a1b2c3d4e5f67890"- ingress {- cidr_blocks = [- "0.0.0.0/0",]- description = "temp"- from_port = 22- protocol = "tcp"- to_port = 22}# (1 unchanged block hidden)}Plan: 0 to add, 1 to change, 0 to destroy.╷│ Warning: Resource targeting is in effect││ You are creating a plan with the -target option, which means that the result│ of this plan may not represent all of the changes requested by the current│ configuration.││ The -target option is not for routine use, and is provided only for│ exceptional situations such as recovering from errors or mistakes, or when│ Terraform specifically suggests to use it as part of an error message.╵Do you want to perform these actions?Terraform will perform the actions described above.Only 'yes' will be accepted to approve.Enter a value: yesaws_security_group.web: Modifying... [id=sg-0a1b2c3d4e5f67890]aws_security_group.web: Modifications complete after 1s [id=sg-0a1b2c3d4e5f67890]Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
If instead the manual change was the right one, adopting it means editing the code so the rule lives in Git, then applying normally. Notice what does not achieve that. terraform apply -refresh-only writes reality into the state file and leaves your configuration untouched, so the very next ordinary plan proposes undoing the change all over again. Refresh-only apply is for accepting values your config does not manage, not for blessing a change you want to keep. And when the drift is a whole resource somebody built by hand that ought to be managed, Terraform 1.5 and later can adopt it: write an import block naming the resource address and its real ID, then run terraform plan -generate-config-out=adopted.tf and Terraform drafts the HCL (HashiCorp Configuration Language, the syntax Terraform files are written in) for you.
Then verify, because "I ran apply" and "the drift is gone" are different claims.
$ terraform plan -refresh-only -detailed-exitcode -lock=false -no-color; echo "exit=$?"
aws_instance.app: Refreshing state... [id=i-04f3a9c2b1d7e5a60]aws_security_group.web: Refreshing state... [id=sg-0a1b2c3d4e5f67890]No changes. Your infrastructure still matches the configuration.Terraform has checked that the real remote objects still match the result ofyour most recent changes, and found no differences.exit=0
Remove the Ability to Drift
Every hour you spend chasing drift is a symptom of something upstream: too many people can change production by hand. Detection is the backstop, not the cure. The cure is closing the manual path for almost everyone. Read-only console access becomes the norm, write access belongs to the pipeline role, and the emergency path is a separate role that pages a human the moment it is used. A tenant can cut as many copies of their office key as they like, but the building's master lock policy still decides which doors any key opens. In AWS that master policy is a Service Control Policy (SCP, an organization-wide ceiling on what any principal in an account may do, regardless of what their own IAM policy allows).
{"Version": "2012-10-17","Statement": [{"Sid": "OnlyThePipelineTouchesSecurityGroups","Effect": "Deny","Action": ["ec2:AuthorizeSecurityGroupIngress","ec2:RevokeSecurityGroupIngress","ec2:AuthorizeSecurityGroupEgress","ec2:RevokeSecurityGroupEgress","ec2:ModifySecurityGroupRules"],"Resource": "*","Condition": {"ArnNotLike": {"aws:PrincipalArn": ["arn:aws:iam::*:role/terraform-apply","arn:aws:iam::*:role/break-glass-admin"]}}}]}
Three caveats before you ship that policy. SCPs never grant permissions, they only take them away, so this is a ceiling and not a substitute for the IAM policies underneath. They do not apply to the organization's management account, or to service-linked roles, so both of those need their own controls and neither is covered by the deny above. And you have to leave a break-glass principal outside the deny, or your first real outage becomes an outage you are locked out of fixing. Then wire an alert to any use of that role. A break-glass session at 3am with no incident ticket attached is one of the highest-quality security signals you will ever get, precisely because it should almost never happen.
Expect the first scheduled run to be noisy. Provider defaults, tags added by a cost tool, an autoscaling group whose desired capacity moves on its own: a fresh check against an older codebase can return dozens of findings. Triage that list once. Fix the code where the code is wrong, add narrow ignore_changes with a comment explaining why where the external change is legitimate, and get to a clean exit 0. From then on the number worth watching is how long a manual change survives before something notices it. Under 24 hours, drift is an operational annoyance. Over a month, it is an intruder's runway.
Try this
Run terraform plan -refresh-only -detailed-exitcode -lock=false -no-color on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.
Takeaway
The trap worth remembering here: a Green Drift Check Is Not a Clean Account. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.