Courses Infrastructure as Code & automationAdvanced · GitOps for infrastructure

Drift detection & remediation

When reality diverges from code.

Advanced12 min · lesson 20 of 23

Drift is when the real infrastructure no longer matches your code — someone made a manual change in the console, a resource was modified out-of-band, or a process altered something. Drift is the enemy of IaC’s core promise (the code is the truth), because once reality diverges, your code lies about what exists, applies behave unpredictably, and the next person cannot trust the repository. Detecting and remediating drift is what keeps IaC honest over time, not just on day one.

DRIFT: DETECT AND RECONCILE

1Drift occurs

manual console change — reality no longer matches code

2Detect

scheduled plan -detailed-exitcode returns exit 2

3Decide

code wins (revert) OR capture the change back in code

4Converge

apply so code and reality match again

The durable fix is prevention — remove console write access so out-of-band changes can't happen.

terminal

# detect drift: does reality still match the code + state?
$ terraform plan -detailed-exitcode
  # aws_security_group.web will be updated in-place
  ~ ingress { + cidr_blocks = ["0.0.0.0/0"] }   # someone opened this in the console!
Exit code: 2      # 2 = drift detected (0 = no changes, 1 = error) — great for CI checks

Detecting drift continuously

A terraform plan that shows changes when you did not change the code is drift — the plan is proposing to "fix" reality back to your code. You catch it deliberately by running plan on a schedule (nightly, in CI) with -detailed-exitcode, which returns a distinct exit code when drift exists, and alerting on it. GitOps tools help here too: Argo CD shows an application as "OutOfSync" when the cluster drifts from Git, and can auto-heal. The goal is that drift surfaces as an alert within hours, not as a nasty surprise during an unrelated apply weeks later.

Remediating: code wins, usually

When drift is found, you reconcile — and the default is that the code wins: apply Terraform to bring reality back to the declared state, undoing the manual change. But sometimes the manual change was a legitimate emergency fix, and the right move is to bring it into the code (update the config to match, so it is captured and reviewed) rather than revert it. Either way you converge code and reality; what you must not do is leave them diverged. And the deeper fix is prevention: lock down console write access so manual changes cannot happen, making the code the only path.

The real fix for drift is removing the ability to drift

Detecting and reverting drift is reactive; the durable solution is preventing out-of-band changes in the first place. Restrict human write access to production infrastructure (read-only console access for most; changes only through the reviewed IaC pipeline), so the code is genuinely the only way to change things and drift becomes nearly impossible. Combined with GitOps (no local applies) and scheduled drift detection as a backstop, you get infrastructure that actually stays as described. Drift is usually a symptom of too many people able to change production directly — fix that, and drift mostly disappears.

GitOps: Atlantis, Argo CD & FluxTesting IaC: Terratest & InSpec