A secure Terraform pipeline, end to end
Everything applied: OIDC, plan/apply split, scan + policy gates, protected state.
An airport runs its checks in a deliberate order, cheapest first. A machine reads your boarding pass at the door in half a second. A scanner looks inside your bag. A person compares your face against a passport. Only at the end does anyone open a door to an aircraft, and that door opens for one flight, for a few minutes, for people who already cleared everything before it. A Terraform pipeline earns its keep the same way. The cheap checks catch most bad outcomes at almost no cost, and the one step that can break production stays locked until every earlier step has passed.
This lesson bolts every control from the course into one file you can copy. What you are protecting is the pipeline's power to change your cloud: whoever can make `terraform apply` run on their behalf owns your infrastructure as completely as if they held your root credentials. That is the prize behind a friendly-looking pull request, and it is the same button a tired engineer leans on at 6pm on a Friday. The pipeline below leaves both of them very little to work with. No long-lived keys anywhere, a plan produced by a role that cannot write, and a person standing between the plan and the change.
Three rules that set the order
Rule one: credential-free checks go first. Formatting, syntax, static scanning and policy unit tests need no cloud access at all. They finish in seconds and cost you nothing when they fail. A run that dies twelve seconds in with a clear message is a good run. Rule two: credentials arrive as late and as narrow as the work allows. Every step that runs before the credential step is a step an attacker cannot use to steal one. When credentials do arrive they are temporary, scoped to a single job, and split in two: one role that can read your cloud, a separate role that can change it.
Rule three: the gate inspects the exact thing that will be applied. It does not read the source files. It does not read a fresh plan computed twenty minutes later either. It reads the saved plan the apply step will consume. Terraform can write a plan to a file, export that file as JSON (JavaScript Object Notation, the plain-text data format both scanners and policy engines read), and apply the same file afterwards without recomputing anything. Every control in the middle of this pipeline reads that one artifact, which closes the gap where the world moves between the check and the change.
Stage one: the checks that hold no credentials
Formatting and syntax checks are the boring ones, which is why people drop them. Keep them. `terraform fmt -check -recursive` changes nothing, prints the name of every file that is not in canonical form, and exits with status 3, and a non-zero exit is how CI (continuous integration, the service that runs your pipeline on every push) decides a step failed. `terraform validate` needs provider schemas, so it needs `terraform init` first. Pass `-backend=false` and Terraform installs providers without touching your state file or asking for a single credential.
terraform fmt -check -recursiveecho "exit: $?"
main.tfexit: 3
A lock file is a receipt for what you installed. `.terraform.lock.hcl` records the provider versions Terraform chose and the checksums of the exact packages it verified, so tomorrow's run gets the same code instead of whatever the registry serves that morning. Commit it, then add `-lockfile=readonly` in the pipeline so `init` refuses to upgrade anything on the quiet: widen a version constraint without updating the lock file and the run stops and tells you, rather than pulling a provider nobody reviewed. One trap catches everybody once. Checksums are recorded per platform, so a lock file written on a Mac fails on a Linux runner until you run `terraform providers lock -platform=linux_amd64 -platform=darwin_arm64` and commit the result.
terraform init -backend=false -lockfile=readonlyterraform validate
Initializing provider plugins...- Reusing previous version of hashicorp/aws from the dependency lock file- Installing hashicorp/aws v6.55.0...- Installed hashicorp/aws v6.55.0 (signed by HashiCorp)Terraform has been successfully initialized!Success! The configuration is valid.
Now scan the code. Checkov reads your HCL (HashiCorp Configuration Language, the `.tf` syntax you write Terraform in) and applies a few hundred prewritten rules for the mistakes everyone makes: unencrypted volumes, security groups open to the whole internet, buckets that accept public access. Trivy does the same job from the other direction and absorbed tfsec, whose checks now ship inside `trivy config`. Either tool belongs here in the credential-free zone, and here it blocks on failure. Watch what it says about the bucket you are shipping.
terraform {required_version = ">= 1.10"backend "s3" {bucket = "acme-tfstate-prod"key = "platform/reports/terraform.tfstate"region = "eu-central-1"encrypt = truekms_key_id = "arn:aws:kms:eu-central-1:111122223333:key/9d2f0c41"use_lockfile = true # native S3 locking (1.10+); dynamodb_table is deprecated}required_providers {aws = {source = "hashicorp/aws"version = "~> 6.0"}}}variable "block_public_access" {description = "Whether this bucket refuses public ACLs and policies."type = booldefault = true}resource "aws_s3_bucket" "reports" {bucket = "acme-quarterly-reports"tags = { owner = "platform" }}resource "aws_s3_bucket_public_access_block" "reports" {bucket = aws_s3_bucket.reports.idblock_public_acls = var.block_public_accessblock_public_policy = var.block_public_accessignore_public_acls = var.block_public_accessrestrict_public_buckets = var.block_public_access}
# scoped to the four public-access rules so the output fits here;# a real run keeps every rule switched oncheckov -d . --quiet --compact \--check CKV_AWS_53,CKV_AWS_54,CKV_AWS_55,CKV_AWS_56
terraform scan results:Passed checks: 4, Failed checks: 0, Skipped checks: 0
Green, and wrong. The four switches are not literal values, they read a variable whose default is `true`. The production values live in `prod.tfvars`, and Terraform reads that file only when you pass `-var-file`, so the scanner never sees it and reports a compliant bucket from the default. You can hand Checkov the same file with `--var-file prod.tfvars` and it will read it, which closes this one hole and no others. Nothing helps when the value arrives from a data source, a remote state lookup, a workspace, or anything the provider computes at apply time. This is the ordinary way a public bucket reaches production past a green scan: nobody lied, the real value arrived later. Which is why the pipeline scans twice.
Stage two: plan with a role that cannot change anything
A visitor badge printed at reception, valid for the afternoon, beats a key cut for the building. That is the trade OIDC (OpenID Connect, a standard way for one system to prove who it is to another) makes for you. GitHub signs a short-lived token describing the run, including which repository, which branch or pull request, and which environment it belongs to. AWS (Amazon Web Services) trusts GitHub's signature, checks those fields against a rule you wrote, and swaps the token for temporary credentials that expire in an hour by default. Nothing durable sits in the repository, so there is no key to leak in a log, nothing to rotate, and nothing a compromised action can steal that still works tomorrow.
Two roles, two rules, and the second rule is the interesting one. The plan role is assumable by runs on the default branch and by pull request runs, and it carries read-only permissions. The apply role is assumable only by a job whose token says `environment:production`, and GitHub mints that claim only for a job that has already cleared the environment's reviewer rule. The write role is unreachable until a person clicks Approve. Not discouraged. Unreachable. Check one detail before you copy these strings into your own account: repositories created from mid-July 2026 onward use an immutable subject claim that also carries numeric owner and repository IDs, in the shape `repo:acme@1234/infra@5678:environment:production`, so read the `sub` from a real token of your own rather than typing it from memory.
# Roles live in the IAM (Identity and Access Management) repo, applied by a# separate pipeline. aws_iam_openid_connect_provider.github is declared once# per account and trusts token.actions.githubusercontent.com.locals {gh = "token.actions.githubusercontent.com"repo = "repo:acme/infra"}# Plan role: read-only, reachable from pull requests and from main.resource "aws_iam_role" "tf_plan" {name = "gh-terraform-plan"assume_role_policy = jsonencode({Version = "2012-10-17"Statement = [{Effect = "Allow"Action = "sts:AssumeRoleWithWebIdentity"Principal = { Federated = aws_iam_openid_connect_provider.github.arn }Condition = {StringEquals = {"${local.gh}:aud" = "sts.amazonaws.com"# a list here means OR, and every value is matched in full"${local.gh}:sub" = ["${local.repo}:ref:refs/heads/main","${local.repo}:pull_request",]}}}]})}# Apply role: only a job that cleared the production environment can assume it.resource "aws_iam_role" "tf_apply" {name = "gh-terraform-apply"assume_role_policy = jsonencode({Version = "2012-10-17"Statement = [{Effect = "Allow"Action = "sts:AssumeRoleWithWebIdentity"Principal = { Federated = aws_iam_openid_connect_provider.github.arn }Condition = {StringEquals = {"${local.gh}:aud" = "sts.amazonaws.com""${local.gh}:sub" = "${local.repo}:environment:production"}}}]})}
Read-only has one exception worth knowing before you write the permissions. `terraform plan` takes the state lock, the way a shared document locks while somebody edits it, so the plan role needs write access to the lock object and to nothing else in the bucket. With native locking in S3 (Simple Storage Service, the Amazon object store holding your state file) that means `s3:GetObject`, `s3:PutObject` and `s3:DeleteObject` on the lock object, which is the state key with `.tflock` on the end. Add `s3:ListBucket` on the bucket, `s3:GetObject` on the state object, and `kms:Decrypt`, `kms:Encrypt` and `kms:GenerateDataKey` when the bucket is encrypted with KMS (Key Management Service, where the encryption key lives). Plans never persist state, so `s3:PutObject` on the state object itself stays with the apply role. Get this wrong in the generous direction and your read-only role can quietly rewrite state, which is the whole game.
# The only writes the plan role gets: the lock object, and nothing else.resource "aws_iam_role_policy" "tf_plan_state" {name = "state-read-and-lock"role = aws_iam_role.tf_plan.idpolicy = jsonencode({Version = "2012-10-17"Statement = [{# narrow further with a StringLike condition on s3:prefix if you likeEffect = "Allow"Action = "s3:ListBucket"Resource = "arn:aws:s3:::acme-tfstate-prod"},{Effect = "Allow"Action = "s3:GetObject" # read state, never write itResource = "arn:aws:s3:::acme-tfstate-prod/platform/reports/terraform.tfstate"},{Effect = "Allow"Action = ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"]Resource = "arn:aws:s3:::acme-tfstate-prod/platform/reports/terraform.tfstate.tflock"},{Effect = "Allow"Action = ["kms:Decrypt", "kms:Encrypt", "kms:GenerateDataKey"]Resource = "arn:aws:kms:eu-central-1:111122223333:key/9d2f0c41"},]})}# Plus a read-only policy for the resources the plan inspects.# arn:aws:iam::aws:policy/ReadOnlyAccess is the blunt version: it can read the# contents of every bucket in the account, so scope it down when you can.
terraform init -lockfile=readonlyterraform plan -input=false -lock-timeout=300s \-var-file=prod.tfvars -out=tfplan.binary
Plan: 2 to add, 0 to change, 0 to destroy.Saved the plan to: tfplan.binaryTo perform exactly these actions, run the following command to apply:terraform apply "tfplan.binary"
Stage three: the gate reads the plan, not the code
`terraform show -json` turns the saved plan into a document with a `resource_changes` array: every resource, its address, the actions planned for it, and the attribute values as they will be after the change. Anything that came from a variable, a workspace or a data source is resolved by now. This is the artifact worth arguing with, because it is the only description of the change that cannot drift from what apply will do. The tool below is `jq`, a small command-line program for slicing JSON, and it is pulling out the four switches the static scan was so happy about.
terraform show -json tfplan.binary > tfplan.jsonjq -c '.resource_changes[]| select(.type=="aws_s3_bucket_public_access_block")| {address, actions: .change.actions,blocks: (.change.after| {block_public_acls, block_public_policy,ignore_public_acls, restrict_public_buckets})}' tfplan.json
{"address":"aws_s3_bucket_public_access_block.reports","actions":["create"],"blocks":{"block_public_acls":false,"block_public_policy":false,"ignore_public_acls":false,"restrict_public_buckets":false}}
There it is, in the open. Run the scanner over the same file and it agrees, this time with the real values. Two tools looking at one document will often shout about the same mistake, so decide up front which one is allowed to stop the run. In this pipeline the scanner annotates the plan (`--soft-fail`, exit 0) and your own policies hold the gate, because those rules are versioned in your repository, unit-tested, and written in your organization's language rather than a vendor's.
checkov -f tfplan.json --quiet --compact --soft-fail \--check CKV_AWS_53,CKV_AWS_54,CKV_AWS_55,CKV_AWS_56
terraform_plan scan results:Passed checks: 0, Failed checks: 4, Skipped checks: 0Check: CKV_AWS_53: "Ensure S3 bucket has block public ACLS enabled"FAILED for resource: aws_s3_bucket_public_access_block.reportsFile: /tfplan.json:0-0Check: CKV_AWS_54: "Ensure S3 bucket has block public policy enabled"FAILED for resource: aws_s3_bucket_public_access_block.reportsFile: /tfplan.json:0-0Check: CKV_AWS_55: "Ensure S3 bucket has ignore public ACLs enabled"FAILED for resource: aws_s3_bucket_public_access_block.reportsFile: /tfplan.json:0-0Check: CKV_AWS_56: "Ensure S3 bucket has 'restrict_public_buckets' enabled"FAILED for resource: aws_s3_bucket_public_access_block.reportsFile: /tfplan.json:0-0
package mainimport rego.v1switches := ["block_public_acls","block_public_policy","ignore_public_acls","restrict_public_buckets",]# Every public access block arrives with all four switches on, whether it is# being created or edited. Missing, null or not-yet-known counts as off:# the resource has to prove it is safe.deny contains msg if {some rc in input.resource_changesrc.type == "aws_s3_bucket_public_access_block"some action in rc.change.actionsaction in {"create", "update"}on := [s | some s in switches; rc.change.after[s] == true]count(on) != count(switches)msg := sprintf("%s blocks %d of %d public access settings", [rc.address, count(on), count(switches)])}# No bucket ACL (access control list, the older per-object permission model)# hands the bucket to the internet.deny contains msg if {some rc in input.resource_changesrc.type == "aws_s3_bucket_acl"some action in rc.change.actionsaction in {"create", "update"}rc.change.after.acl in {"public-read", "public-read-write"}msg := sprintf("%s sets a public ACL (%s)", [rc.address, rc.change.after.acl])}
Conftest is the small command that feeds a file to OPA (Open Policy Agent, a general-purpose policy engine) and turns the verdict into an exit code, and Rego is the query language those rules are written in. Notice the shape of the first rule. It counts the switches that are explicitly `true` and denies unless all four are, so a missing attribute, a null, or a value Terraform cannot compute until apply time fails closed instead of sliding through. Rules that fail closed produce the occasional false alarm. Rules that fail open produce incidents. Policies are code, so test them like code: keep a denying case and a passing case in `policy/public_access_test.rego` with rule names starting with `test_`, and `conftest verify` runs them back in the credential-free stage, before any of this goes near your cloud.
conftest test tfplan.json --policy policy/echo "exit: $?"
FAIL - tfplan.json - main - aws_s3_bucket_public_access_block.reports blocks 0 of 4 public access settings2 tests, 1 passed, 0 warnings, 1 failure, 0 exceptionsexit: 1
The whole thing, as one workflow file
Read the file for its permissions before you read it for its steps. The top-level grant is `contents: read`, and a job-level `permissions` block replaces that grant rather than adding to it, which is why the jobs that talk to AWS name `contents: read` again alongside `id-token: write`. Three jobs carry that write permission; a job without it cannot ask GitHub for an identity token at all. A pull request opened from a fork gets a read-only token and no secrets, so it never receives the write-level id-token permission the exchange needs, and a stranger's code fails at the credential step rather than at your cloud. Each checkout also drops the git credential it was handed once the files are on disk, so no later step can borrow it. The `environment: production` line is where a reviewer stops the run, and in that environment's settings you also limit deployments to the default branch so no side branch can queue an approval. One last habit: version tags on actions move, so pin them to a commit hash once you have read the code you are running.
name: terraformon:pull_request:push:branches: [main]schedule:- cron: "17 6 * * *" # daily drift check, 06:17 UTCpermissions:contents: read # a job-level block replaces this, it does not add to itconcurrency:group: terraform-production # one run at a time against one state filecancel-in-progress: falseenv:TF_VERSION: "1.15.8"AWS_REGION: eu-central-1ACCOUNT: "111122223333"jobs:# 1. no credentials exist anywhere in this jobstatic:if: github.event_name != 'schedule'runs-on: ubuntu-lateststeps:- uses: actions/checkout@v7 # pin to a commit hash in productionwith:persist-credentials: false- uses: hashicorp/setup-terraform@v4with:terraform_version: ${{ env.TF_VERSION }}terraform_wrapper: false # keep stdout byte-exact and exit codes raw- run: terraform fmt -check -recursive- run: terraform init -backend=false -lockfile=readonly- run: terraform validate- name: Checkov on the .tf filesuses: bridgecrewio/checkov-action@v12with:directory: .quiet: truecompact: true- name: Unit-test the policiesrun: |docker run --rm -v "$PWD:/project" \openpolicyagent/conftest:v0.68.2 verify --policy policy/# 2. read-only credentials, saved plan, policy gateplan:needs: staticruns-on: ubuntu-latestpermissions:id-token: write # only this job may ask GitHub for a tokencontents: readsteps:- uses: actions/checkout@v7with:persist-credentials: false- uses: hashicorp/setup-terraform@v4with:terraform_version: ${{ env.TF_VERSION }}terraform_wrapper: false- uses: aws-actions/[email protected]with:role-to-assume: arn:aws:iam::${{ env.ACCOUNT }}:role/gh-terraform-planrole-session-name: plan-${{ github.run_id }}aws-region: ${{ env.AWS_REGION }}- run: terraform init -lockfile=readonly # S3 backend, locking on- run: |terraform plan -input=false -lock-timeout=300s \-var-file=prod.tfvars -out=tfplan.binary- run: terraform show -json tfplan.binary > tfplan.json- name: Scan the plan (reports, does not block)uses: bridgecrewio/checkov-action@v12with:file: tfplan.jsonquiet: truecompact: truesoft_fail: true- name: Policy gaterun: |docker run --rm -v "$PWD:/project" \openpolicyagent/conftest:v0.68.2 test tfplan.json --policy policy/- uses: actions/upload-artifact@v7with:name: tfplan-${{ github.run_id }}path: tfplan.binary # the JSON never leaves the runnerretention-days: 1# 3. a human, then write credentialsapply:needs: planif: github.event_name == 'push' && github.ref == 'refs/heads/main'runs-on: ubuntu-latestenvironment: production # required reviewers hold the job herepermissions:id-token: writecontents: readsteps:- uses: actions/checkout@v7with:persist-credentials: false- uses: hashicorp/setup-terraform@v4with:terraform_version: ${{ env.TF_VERSION }}terraform_wrapper: false- uses: actions/download-artifact@v8with:name: tfplan-${{ github.run_id }}- uses: aws-actions/[email protected]with:role-to-assume: arn:aws:iam::${{ env.ACCOUNT }}:role/gh-terraform-applyrole-session-name: apply-${{ github.run_id }}aws-region: ${{ env.AWS_REGION }}- run: terraform init -lockfile=readonly# no -var-file: the values are already baked into the reviewed plan,# and Terraform rejects variables when you apply a saved plan- run: terraform apply -input=false -lock-timeout=300s tfplan.binary# 4. drift, on the schedule trigger only, with the read-only roledrift:if: github.event_name == 'schedule'runs-on: ubuntu-latestpermissions:id-token: writecontents: readissues: writesteps:- uses: actions/checkout@v7with:persist-credentials: false- uses: hashicorp/setup-terraform@v4with:terraform_version: ${{ env.TF_VERSION }}terraform_wrapper: false- uses: aws-actions/[email protected]with:role-to-assume: arn:aws:iam::${{ env.ACCOUNT }}:role/gh-terraform-planaws-region: ${{ env.AWS_REGION }}- run: terraform init -lockfile=readonly- id: checkrun: |set +eterraform plan -input=false -no-color -detailed-exitcode \-var-file=prod.tfvars -out=drift.binaryecho "code=$?" >> "$GITHUB_OUTPUT"exit 0- if: steps.check.outputs.code == '2'run: |gh issue create --title "Terraform drift in production" \--body "Scheduled plan found changes: $GITHUB_SERVER_URL/$GITHUB_REPOSITORY/actions/runs/$GITHUB_RUN_ID"env:GH_TOKEN: ${{ github.token }}
The run: blocked, then green
Run 8412 came from a pull request that flipped one line in `prod.tfvars`. The static job went green, because the code still looked fine. The plan job assumed the read-only role, produced a plan, exported it, let the scanner annotate it, and stopped dead at the gate step with the failure you saw above and exit code 1. The apply job never started, so the write role was never assumed, so the token that could create a public bucket was never issued. The fix is one line.
git diff -- prod.tfvars
diff --git a/prod.tfvars b/prod.tfvarsindex 034bd37..863d27d 100644--- a/prod.tfvars+++ b/prod.tfvars@@ -1 +1 @@-block_public_access = false+block_public_access = true
Run 8413 takes the same path with the corrected value. The plan JSON now reports all four settings as `true`, the scanner passes, and the gate returns nothing to complain about. Only then does the apply job appear in the run graph, waiting on a reviewer.
checkov -f tfplan.json --quiet --compact --soft-fail \--check CKV_AWS_53,CKV_AWS_54,CKV_AWS_55,CKV_AWS_56conftest test tfplan.json --policy policy/echo "exit: $?"
terraform_plan scan results:Passed checks: 4, Failed checks: 0, Skipped checks: 02 tests, 2 passed, 0 warnings, 0 failures, 0 exceptionsexit: 0
# apply job, after a reviewer approved the production environmentterraform apply -input=false -lock-timeout=300s tfplan.binary
Acquiring state lock. This may take a few moments...aws_s3_bucket.reports: Creating...aws_s3_bucket.reports: Creation complete after 4s [id=acme-quarterly-reports]aws_s3_bucket_public_access_block.reports: Creating...aws_s3_bucket_public_access_block.reports: Creation complete after 1s [id=acme-quarterly-reports]Apply complete! Resources: 2 added, 0 changed, 0 destroyed.
Drift, on a timer
Everything above governs changes that arrive through the pipeline. The console does not care about your pipeline. Someone with production access clicks four checkboxes during an incident at 2am, the incident ends, and nobody remembers. The scheduled job is the smoke alarm in the hallway: same repository, same read-only role, one plan a day. `-detailed-exitcode` gives Terraform three answers instead of two, and the job keys off the middle one. Zero means reality matches the code, 1 means the plan itself failed, 2 means something changed. Exit 2 opens an issue with a link to the run, and someone reads the diff over coffee instead of over a breach notification.
terraform plan -input=false -no-color -detailed-exitcode \-var-file=prod.tfvars -out=drift.binaryecho "exit: $?"
Note: Objects have changed outside of TerraformTerraform detected the following changes made outside of Terraform since thelast "terraform apply" which may have affected this plan:# aws_s3_bucket_public_access_block.reports has been changed~ resource "aws_s3_bucket_public_access_block" "reports" {~ block_public_policy = true -> false}Terraform will perform the following actions:# aws_s3_bucket_public_access_block.reports will be updated in-place~ resource "aws_s3_bucket_public_access_block" "reports" {~ block_public_policy = false -> true}Plan: 0 to add, 1 to change, 0 to destroy.Saved the plan to: drift.binaryexit: 2