Terraform in CI/CD & automation

Plan on PR, apply on merge, safely.

A Terraform run from a laptop is the infrastructure equivalent of letting yourself into the building after hours with a master key. You move the furniture, you leave, and the only record is whatever you happen to remember. Nobody signed you in. Nobody watched. Multiply that by six engineers, each with a permanent cloud access key sitting in a file in their home directory, and you have an estate nobody can audit and anybody can flatten. Moving Terraform into CI/CD (continuous integration and continuous delivery, the automated system that checks and ships your changes for you) puts a front desk in front of that door. Every change gets read as a diff before it happens, and the keys become day passes that expire, handed out by a machine you control, instead of cut keys living forever on machines you do not.

What makes this trustworthy is boring paperwork. A pharmacist fills the prescription the doctor actually wrote. They read the paper, walk to the shelf, and come back with what is on it. They do not invent a fresh prescription from memory somewhere between the counter and the shelf. Terraform gives you that same guarantee with a saved plan file. terraform plan -out=tfplan writes the exact intended change set to disk. terraform apply tfplan carries out that file and nothing else. What a reviewer approved is what runs.

Plan on the Pull Request

On every pull request (a proposed code change, opened so other people can read it before it lands), CI checks out the branch and runs a plan. Two environment variables make Terraform behave sensibly on a machine with no keyboard attached. TF_INPUT=0 stops it ever pausing to ask a question, because a prompt on a build server is a job that hangs until it times out. TF_IN_AUTOMATION=1 tells Terraform nobody is reading the terminal, so it drops the chatty "now run terraform apply" suggestions from its output. Any non-empty value works there; 1 is the convention. Add -lock-timeout so a run that collides with another one waits its turn instead of dying on the spot.

$ export TF_INPUT=0 TF_IN_AUTOMATION=1
$ terraform init -lockfile=readonly
$ terraform plan -lock-timeout=5m -out=tfplan

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v6.55.0...
- Installed hashicorp/aws v6.55.0 (signed by HashiCorp)
 
Terraform has been successfully initialized!
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 
Terraform will perform the following actions:
 
  # aws_vpc_security_group_ingress_rule.web_ssh will be created
  + resource "aws_vpc_security_group_ingress_rule" "web_ssh" {
      + arn                    = (known after apply)
      + cidr_ipv4              = "0.0.0.0/0"
      + from_port              = 22
      + id                     = (known after apply)
      + ip_protocol            = "tcp"
      + region                 = (known after apply)
      + security_group_id      = "sg-0a1b2c3d4e5f60718"
      + security_group_rule_id = (known after apply)
      + tags_all               = (known after apply)
      + to_port                = 22
    }
 
Plan: 1 to add, 0 to change, 0 to destroy.
 
Saved the plan to: tfplan

Two details in that run are worth slowing down for. Terraform takes a lock on the state file even for a plan, so two pull requests planning at the same moment queue up rather than reading state while somebody else is writing it. And -lockfile=readonly makes init fail loudly instead of quietly rewriting .terraform.lock.hcl when the configuration asks for provider versions the committed lock file does not cover, which turns a silent provider swap into a visible diff. That flag has one common failure mode. The lock file records a checksum per operating system and CPU, so a lock generated only on someone's Mac will break the Linux runner with a complaint that the package does not match any recorded checksum. Fix it once with terraform providers lock -platform=linux_amd64 -platform=darwin_arm64 and commit the result.

The Gate Reads the Plan, Not the Code

Scanning the .tf files is like reading the packing list. Reading the plan is like looking inside the loaded truck. The list catches plenty, but the plan is the thing that is actually about to happen: it carries computed values, resolved variables, and the full list of what will be created, changed and destroyed. terraform show -json turns the binary plan into JSON (JavaScript Object Notation, a plain-text data format that other tools can read), and a policy engine such as Conftest, which runs Open Policy Agent rules, tests that JSON and exits non-zero when a rule fails. Non-zero exit fails the CI step. A failed step blocks the merge. That is the entire enforcement mechanism.

$ terraform show -json tfplan > tfplan.json
$ conftest test --policy policy/ --all-namespaces tfplan.json; echo "exit=$?"

FAIL - tfplan.json - terraform - aws_vpc_security_group_ingress_rule.web_ssh opens port 22 to 0.0.0.0/0
 
1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions
exit=1

That 0.0.0.0/0 is CIDR notation (a way of writing a range of network addresses), and this particular one means the whole internet. For the human half of the review, post terraform show -no-color tfplan as a comment on the pull request. The reviewer and the policy engine are then reading the same plan file, rendered two different ways, which is exactly the point. Be careful which rendering you publish. The human-readable form masks sensitive attributes as (sensitive value). The JSON form carries the real values, which is precisely why a policy engine can test them. Post the text. Never post the JSON.

Apply the File, Not a Fresh Idea

Applying a saved plan is a different operation from applying a configuration. terraform apply on its own works out a new plan and asks you to confirm it. terraform apply tfplan skips both steps: there is nothing to confirm, because the decision was made at review time, and nothing to recompute, because the actions are already in the file. This is why -auto-approve has no place in an apply job. It means "work out a brand new plan right now and do whatever it says", which throws away everything the review bought you.

$ terraform apply -lock-timeout=5m tfplan

aws_vpc_security_group_ingress_rule.web_ssh: Creating...
aws_vpc_security_group_ingress_rule.web_ssh: Creation complete after 1s [id=sgr-0a1b2c3d4e5f60718]
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Now the interesting failure. Suppose a colleague's pull request merged and applied while yours sat waiting for approval. Your saved plan was computed against a version of the state that no longer exists, so its actions may no longer make any sense. The pharmacist checks the date on the prescription. Terraform does the same, because the plan file records which state snapshot it was built from.

$ terraform apply -lock-timeout=5m tfplan     # another apply landed first

╷
│ Error: Saved plan is stale
│ 
│ The given plan file can no longer be applied because the state was changed
│ by another operation after the plan was created.
╵

Treat that error as the system working. The right response is to re-run the plan, look at the new diff, and get it approved again. The wrong response, and people really do reach for it, is to make the apply job compute its own plan with -auto-approve so the error goes away. One line, and the review boundary is gone. Prevent the collision instead: put a concurrency group around the pipeline so only one Terraform run touches a given state at a time.

The Whole Pipeline in One File

name: terraform
on:
  pull_request:
  push:
    branches: [main]
 
permissions:              # the default for every job; a job may narrow it, never widen it
  contents: read
  id-token: write         # may ask GitHub for an OIDC token; grants nothing else
 
concurrency:              # one Terraform run against this state at a time
  group: terraform-prod
  cancel-in-progress: false
 
env:
  TF_INPUT: "0"
  TF_IN_AUTOMATION: "1"
 
jobs:
  plan:
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/tf-ci-plan    # read-only
          role-session-name: gha-${{ github.run_id }}
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "1.15.8" }
      - run: terraform init -lockfile=readonly
      - run: terraform plan -lock-timeout=5m -out=tfplan
      - run: terraform show -json tfplan > tfplan.json
      - run: conftest test --policy policy/ --all-namespaces tfplan.json
      - uses: actions/upload-artifact@v4
        if: github.event_name == 'push'     # never publish a PR plan as an artifact
        with: { name: tfplan, path: tfplan, retention-days: 1 }
 
  apply:
    needs: plan
    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
    environment: production      # required reviewers hold this job until approved
    runs-on: ubuntu-22.04
    steps:
      - uses: actions/checkout@v4
      - uses: actions/download-artifact@v4
        with: { name: tfplan }
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/tf-ci-apply   # write
          role-session-name: gha-${{ github.run_id }}
          aws-region: us-east-1
      - uses: hashicorp/setup-terraform@v3
        with: { terraform_version: "1.15.8" }
      - run: terraform init -lockfile=readonly
      - run: terraform apply -lock-timeout=5m tfplan

Read the permissions block first, and read it as a ceiling rather than a floor. Whatever you list there is the most any job in this workflow can have, and an individual job can only ask for less. id-token: write does not grant write access to anything. It lets the job ask GitHub for a signed identity token, which is the whole OIDC (OpenID Connect, a standard way for one system to prove who it is to another without sharing a password) mechanism. Declaring permissions at the top also stops you inheriting whatever the repository default happens to be, which on older repositories is generous.

There is a gap here worth being straight about. The plan a reviewer reads belongs to the pull request run. The plan that applies is produced by a second run after the merge. They are nearly always identical, and the policy gate and the stale-plan check both run again on that second one, so the gap is covered rather than ignored. If you want the reviewed bytes to be the applied bytes with no second run at all, that is what Atlantis buys you: it plans on the pull request, you comment atlantis apply, it applies that exact plan, and the merge happens afterwards. HCP Terraform, GitLab and Spacelift build the same shape with their own vocabulary.

Two Roles, One That Reads and One That Changes

In a well-run warehouse, the person walking the aisles with a clipboard counting stock is not the person holding the keys to the loading bay. Split your pipeline the same way. Give the plan job a role that cannot change anything: read access to the cloud API and to the state bucket, and nothing else. The apply job gets a second role holding the write permissions, and only that job ever assumes it. The split matters because the plan job is the one that runs on pull requests, including pull requests from people you have never met. If a plan can create resources, a plan is an apply with extra steps.

Both roles should be assumed through OIDC rather than holding a static key, and the difference is the difference between a visitor badge and a cut key. The runner asks GitHub for a token that says "I am this workflow, in this repository, on this ref", hands it to AWS STS (Security Token Service, the service that issues temporary credentials), and gets back keys that expire in an hour or less. No permanent secret exists to leak. All the security of that exchange lives in one document, the role's trust policy in IAM (Identity and Access Management, the AWS service that decides who is allowed to do what), and that document is the thing to audit.

$ aws iam get-role --role-name tf-ci-apply \
    --query 'Role.AssumeRolePolicyDocument.Statement[0]' --output json

{
    "Effect": "Allow",
    "Principal": {
        "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
        "StringEquals": {
            "token.actions.githubusercontent.com:aud": "sts.amazonaws.com"
        },
        "StringLike": {
            "token.actions.githubusercontent.com:sub": "repo:acme/infra:*"
        }
    }
}

That StringLike line is far too loose. The sub claim (short for subject, the line in the token that says who is asking) is the only thing distinguishing one workflow from another, and repo:acme/infra:* matches every branch, every tag, every pull request and every environment in the repository. Any workflow anyone can trigger there can mint apply credentials. Pin it to the exact context you expect. A job declaring environment: production gets the subject repo:acme/infra:environment:production, so match that string with StringEquals and nothing else. The plan job has no environment, so its subject is repo:acme/infra:pull_request on pull request runs and repo:acme/infra:ref:refs/heads/main on pushes. Then go hunting across your accounts for the catastrophic version of this mistake: a trust policy with no sub condition at all, or with sub set to *, can be assumed by any GitHub Actions workflow on the internet.

A Plan Runs Code You Have Not Read

terraform plan sounds passive. It is not. Running it downloads provider plugins and executes them, executes any data "external" block by running the program that block names on the runner, and clones whatever module sources the configuration points at. All of that is code, all of it comes from the branch being planned, and on a pull request all of it can be written by whoever opened the pull request. A contributor who adds a module sourced from their own Git URL, or who swaps required_providers to a lookalike namespace, gets code execution on your runner with whatever credentials that job is holding.

The Plan File Is a Credential

A plan file embeds a copy of the state it was computed against, and state holds resource attributes in the clear: database passwords, generated private keys, tokens a provider handed back. A tfplan artifact is a photocopy of the contents of your safe, wearing a friendlier name. Anyone who can download it can run terraform show -json on it and read the lot. That is why the workflow above uploads the artifact only on pushes, sets retention-days to 1, keeps the file inside a single run, and never prints plan JSON to the build log. Check whether your CI provider serves artifacts to anyone who can read the repository, because on a public repository that is everyone. If a plan artifact has been exposed, rotate every credential the state contained rather than deciding it was probably fine. And if you push plans to object storage instead of artifacts, encrypt them and lock that bucket down exactly as tightly as the state bucket.

What the Cloud Side Should Show You

Set role-session-name to something carrying the workflow run identifier and your cloud audit log stops being anonymous. Every API call the apply makes arrives stamped with a session name you can trace to a run, which traces to a merge commit, which traces to a reviewed pull request. Here is that chain in AWS CloudTrail, the log of every API call made in the account.

$ aws cloudtrail lookup-events --max-results 1 \
    --lookup-attributes AttributeKey=EventName,AttributeValue=AuthorizeSecurityGroupIngress \
  | jq -r '.Events[].CloudTrailEvent | fromjson
           | [.eventTime, .userIdentity.arn, .sourceIPAddress] | join("  ")'

2026-07-21T09:14:23Z  arn:aws:sts::123456789012:assumed-role/tf-ci-apply/gha-9284713  20.29.134.17

That identity is your detection hook. Anything that changes managed infrastructure without carrying the tf-ci-apply session name is out-of-band change: a human clicking around the console, a forgotten access key, or an intruder. Alert on it. The other half of the loop is a night watchman, a scheduled job that walks the estate rattling door handles and changes nothing. It runs a plan under the read-only role with -detailed-exitcode, which returns 0 for no changes, 1 for an error, and 2 for a non-empty diff. Leave -out off it, so the job never writes a plan file worth stealing.

$ terraform plan -lock-timeout=5m -detailed-exitcode; echo "exitcode=$?"

Note: Objects have changed outside of Terraform
 
Terraform detected the following changes made outside of Terraform since the
last "terraform apply" which may have affected this plan:
 
  # aws_instance.web has been changed
  ~ resource "aws_instance" "web" {
        id       = "i-0f1e2d3c4b5a69780"
      ~ tags     = {
          + "owner" = "changed-in-the-console"
        }
      ~ tags_all = {
          + "owner" = "changed-in-the-console"
        }
        # (30 unchanged attributes hidden)
    }
 
 
Unless you have made equivalent changes to your configuration, or ignored the
relevant attributes using ignore_changes, the following plan may include
actions to undo or respond to these changes.
 
─────────────────────────────────────────────────────────────────────────────
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
 
Terraform will perform the following actions:
 
  # aws_instance.web will be updated in-place
  ~ resource "aws_instance" "web" {
        id       = "i-0f1e2d3c4b5a69780"
      ~ tags     = {
          - "owner" = "changed-in-the-console" -> null
        }
      ~ tags_all = {
          - "owner" = "changed-in-the-console" -> null
        }
        # (30 unchanged attributes hidden)
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.
exitcode=2

Run that nightly and page on exit code 2. One caveat: a change Terraform would not act on can show up in the "changed outside of Terraform" note while still leaving the plan empty and the exit code at 0, so read the note as well as the number. Handled that way, the loop closes. Git states what should exist, the pipeline is the only thing permitted to change it, and anything that shifts underneath you surfaces as a diff nobody asked for.

Go and attack your own pipeline before someone else does. Open a throwaway pull request adding a security group rule open to the world and confirm the plan job fails at the policy step. Push a branch from a fork and confirm no credentials get minted for it. Pull the trust policy for your apply role and read the sub condition out loud, word by word. Then try to download yesterday's plan artifact and see whether it is still sitting there waiting for you. A pipeline you have never tried to defeat is a pipeline you are guessing about.

Try this

Run terraform init -lockfile=readonly on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a pull request plan is untrusted code execution. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related