DRY remote state

One backend block for the whole repo.

Terraform's state file is a ledger. Every resource it built, every id the cloud handed back, every attribute it read: one long document in JSON (JavaScript Object Notation, a plain text format for structured data). A remote backend is where that ledger lives. It sits in an object store instead of on your laptop, in a bucket, which is a named container for files in a service like Amazon S3 (Simple Storage Service, the file store of AWS, short for Amazon Web Services). Terraform insists that each root module declare its own backend, in a block that cannot contain a variable, a local, or any expression at all. Fifty modules, fifty near-identical blocks. Change the bucket name and you are editing fifty files by hand. The one you miss keeps writing to the old bucket, quietly, for months. (OpenTofu, the open source fork of Terraform, loosened this in version 1.8 and does allow variables there. Terraform still does not.)

Terragrunt turns the ownership around. You write the backend once, at the top of the repo, and every unit below inherits it. A unit is one directory holding a terragrunt.hcl file, written in HCL (HashiCorp Configuration Language, the same syntax Terraform uses), and it maps to exactly one state file. The mailroom of a large apartment building works the same way. One set of sorting rules for the whole building, one locked box per apartment, and the box number falls out of the apartment number instead of being handed out by someone with a clipboard. One place to read, one place to harden, and every tenant still has their own locked box.

One Block At The Root

Put a single remote_state block in a root config file at the top of your live directory, named root.hcl by convention. It names the backend type, the bucket, the region, encryption, and locking. The argument that does the real work is generate. Leave generate out and Terragrunt hands those settings to Terraform as -backend-config flags on init, which works, but they live only for the length of that one command. Set generate and Terragrunt writes an actual backend.tf file into the unit's working directory a moment before it calls Terraform, on every single run. Nothing is kept between runs, nothing is committed, so what Terraform sees cannot drift away from what the root declares.

# One backend definition for the entire repo. Every unit includes this file.
remote_state {
  backend = "s3"
 
  generate = {
    path      = "backend.tf"            # written into the unit's working dir
    if_exists = "overwrite_terragrunt"   # replace only files Terragrunt signed
  }
 
  config = {
    bucket = "acme-tfstate-${get_aws_account_id()}"
    key    = "${path_relative_to_include()}/terraform.tfstate"
    region = "us-east-1"
 
    encrypt      = true   # server-side encryption of the state object at rest
    use_lockfile = true   # native S3 locking (OpenTofu 1.10+, Terraform 1.10+)
    # dynamodb_table = "terraform-locks"   # the old lock table, now deprecated
  }
}

Read that config map one line at a time, because each entry is a control. get_aws_account_id() asks AWS which account your current credentials belong to, so if each environment sits in its own account, the bucket name changes with the credentials and the account boundary does the isolating for you. encrypt turns on server-side encryption, so S3 encrypts the object at rest. use_lockfile is the occupied sign on the door: while one run holds the lock, nobody else gets in. Without it, two people running apply at the same moment can interleave writes and shred the ledger. And key is the address of this unit's own box, which is the line worth staring at longest.

include "root" {
  path = find_in_parent_folders("root.hcl")   # walk up until root.hcl is found
}
 
terraform {
  source = "git::[email protected]:acme/modules.git//vpc?ref=v1.4.0"
}
 
inputs = {
  # CIDR = Classless Inter-Domain Routing, the notation for an address range
  cidr_block = "10.20.0.0/16"
}
 
# Note what is absent: no backend block, no bucket, no state key.

The Folder Path Is The State Key

Inheriting one backend would be useless if every unit wrote to the same address, because they would take turns overwriting each other. path_relative_to_include() is what keeps them apart. It returns the path from the directory holding the included root.hcl down to the unit you are standing in. So live/prod/us-east-1/vpc, the unit that builds the VPC (Virtual Private Cloud, your own private network inside AWS), resolves to prod/us-east-1/vpc, and the key becomes prod/us-east-1/vpc/terraform.tfstate. The folder tree is the state tree. Adding a new component costs one small file and zero decisions about where its state goes.

$ cd ~/live/prod/us-east-1/vpc
$ terragrunt init >/dev/null
$ cat "$(find .terragrunt-cache -name backend.tf | head -1)"

# Generated by Terragrunt. Sig: nIlQXj57tbuaRZEa
terraform {
  backend "s3" {
    bucket       = "acme-tfstate-123456789012"
    key          = "prod/us-east-1/vpc/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true
    use_lockfile = true
  }
}

Two things in that file matter. The Sig line is Terragrunt's signature on its own work, and if_exists = "overwrite_terragrunt" tells it to replace a file carrying that signature and refuse to touch anything else. The second is where the file landed. Because this unit sets terraform { source }, Terraform runs inside .terragrunt-cache and the generated backend never appears in your source tree. A unit with no source gets backend.tf written right beside its terragrunt.hcl, which is why the generated filenames belong in .gitignore whether you think you need it or not.

# Resolve the config without running Terraform at all
$ cd ~/live/prod/us-east-1/vpc
$ terragrunt render --format json | jq -r '.remote_state.config | .bucket, .key'
$ cd ../eks
$ terragrunt render --format json | jq -r '.remote_state.config | .bucket, .key'

acme-tfstate-123456789012
prod/us-east-1/vpc/terraform.tfstate
acme-tfstate-123456789012
prod/us-east-1/eks/terraform.tfstate

terragrunt render merges every include and evaluates every function, then prints the configuration the unit actually ended up with. Piped through jq (a small command line filter for JSON) it answers the only question that matters here in under a second: same bucket, different key. The second unit there is eks (Elastic Kubernetes Service, the managed Kubernetes offering from AWS), sitting next to the network it runs on. This is the command to reach for when you are reviewing someone else's pull request, before anything touches the cloud.

One Key Layout Makes Least Privilege Possible

Terraform records what it manages in full, resolved, in clear text. A database password you passed as an input sits in the state file as a readable string. So does a generated private key, a service account token, the body of a secret you pulled in with a data source. Reading production state is close to a full credential dump of the estate, which makes that bucket one of the highest value objects you own. The prefix each unit writes to is the boundary you get to defend, and it exists only because one expression in one file decides it.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOwnEnvironmentStateOnly",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-tfstate-123456789012/dev/*"
    },
    {
      "Sid": "ListOnlyOwnPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-tfstate-123456789012",
      "Condition": { "StringLike": { "s3:prefix": ["dev/*"] } }
    },
    {
      "Sid": "NeverTouchProdState",
      "Effect": "Deny",
      "Action": "s3:*",
      "Resource": "arn:aws:s3:::acme-tfstate-123456789012/prod/*"
    }
  ]
}

That policy can be written at all because the keys are predictable. In AWS IAM (Identity and Access Management, the rules deciding which identity may touch which resource) an explicit Deny beats every Allow anywhere else, including one a well-meaning colleague grants next quarter. Those long arn: strings are ARNs (Amazon Resource Names), the unique address of a resource. Now think like the attacker. Someone phishes a developer laptop, assumes that role, and the first move is always to list the state bucket and pull production. They hit the deny. If you turned on S3 server access logging or CloudTrail (the AWS audit log of API calls, where an API is the programmatic interface the cloud exposes) data events for the bucket, the attempt lands in a log with a principal, a source address, and the exact key. That is a high quality alert: any GetObject under prod/ by anything other than the production pipeline role.

# What the attempt looks like from a developer's credentials,
# using the AWS CLI (command line interface)
$ aws s3 cp s3://acme-tfstate-123456789012/prod/us-east-1/vpc/terraform.tfstate -

fatal error: An error occurred (AccessDenied) when calling the GetObject operation:
User: arn:aws:sts::123456789012:assumed-role/dev-engineer/sachin is not authorized
to perform: s3:GetObject on resource:
"arn:aws:s3:::acme-tfstate-123456789012/prod/us-east-1/vpc/terraform.tfstate"
with an explicit deny in an identity-based policy

Bootstrap The Backend Once

The bucket has to exist before Terraform can write to it, and there is a genuine chicken and egg problem: the thing that creates your state store cannot keep its own state there. terragrunt backend bootstrap reads the same remote_state block and creates whatever is missing. Older versions folded this into init, asking for a yes if you were sitting at a terminal and creating the bucket without a word in a pipeline, where non-interactive mode is on. Making it a command of its own (or an explicit --backend-bootstrap flag on a run) means creating cloud infrastructure is something your pipeline gates and logs, rather than something that happens because someone typed init while pointed at the wrong account. It is idempotent, meaning a second run changes nothing, and --all walks every unit under the current directory.

$ cd ~/live/prod/us-east-1/vpc
$ terragrunt backend bootstrap

14:22:07.318 info Bootstrapping remote state for unit prod/us-east-1/vpc
14:22:09.664 info Creating S3 bucket acme-tfstate-123456789012
14:22:11.402 info Enabling versioning on S3 bucket acme-tfstate-123456789012
14:22:12.117 info Enabling server-side encryption on S3 bucket acme-tfstate-123456789012
14:22:13.055 info Blocking all public access to S3 bucket acme-tfstate-123456789012
14:22:14.230 info Enabling bucket-wide TLS enforcement on S3 bucket acme-tfstate-123456789012

Those defaults are the point. Versioning, encryption, a full public access block, and a bucket policy that refuses plain HTTP so state can only move over TLS (Transport Layer Security, the encryption behind HTTPS). Terragrunt also gives you switches to turn each one off: skip_bucket_versioning, skip_bucket_public_access_blocking, skip_bucket_enforced_tls. Every one of those is a control disabled for every unit in the repo at once, usually added by someone trying to make an error message go away. A one-line pull request that adds a skip_ option to root.hcl deserves the same reading as one that opens a port.

# Verify the backend is what you think it is
$ aws s3api get-bucket-versioning --bucket acme-tfstate-123456789012
$ aws s3api list-objects-v2 --bucket acme-tfstate-123456789012 \
    --query 'Contents[].Key' --output text | tr '\t' '\n'

{
    "Status": "Enabled"
}
dev/us-east-1/eks/terraform.tfstate
dev/us-east-1/vpc/terraform.tfstate
prod/us-east-1/eks/terraform.tfstate
prod/us-east-1/vpc/terraform.tfstate

One object per unit, named after its folder, and versioning on so a truncated write or a malicious overwrite can be rolled back and compared. During an active run you will also see a small companion object at the same path with a .tflock suffix. That is the native S3 lock that use_lockfile creates. A .tflock left behind by a crashed run is what an "Error acquiring the state lock" message means, and because the lock is a plain object you can read it to see which user and which operation held it, then delete it once you are certain that run is dead. Five second diagnosis instead of a mystery.

A Backend File You Did Not Sign

Here is a quiet attack worth understanding. Someone with commit access adds a hand-written backend.tf to a shared module, pointing state at a bucket in an account they control. Every team that consumes that module starts posting its state, secrets and all, to the attacker, and plans keep working perfectly, so nobody notices. The signature check is what stops it. With if_exists = "overwrite_terragrunt", Terragrunt overwrites only files carrying its own Sig line, and refuses to run when it finds a foreign file sitting at the path it was told to generate.

$ terragrunt init; echo "exit=$?"

14:31:02.887 error Can not generate terraform file:
/home/sachin/live/prod/us-east-1/vpc/.terragrunt-cache/JgTQ1PLwvBSD/qN4aP2xR9hKm/vpc/backend.tf
already exists
exit=1

That refusal is a tamper alarm, so treat it as one instead of reaching for if_exists = "overwrite" to make it stop. Back it with a repository rule: git ls-files '*backend.tf' '*provider.tf' should return nothing at all, and a pipeline step that fails when it returns something catches the file at review time rather than at run time. Generated files are build output, and build output does not belong in version control.

# Terragrunt working files and generated config: never commit these
.terragrunt-cache/
backend.tf
provider.tf
terragrunt.rendered.json
*.tfplan

Moving A Unit Is A State Migration

Because the key comes from the folder path, renaming a folder changes the address of the state file. Terragrunt keeps no memory of where the unit used to live, so you move the object yourself first. terragrunt backend migrate takes a source unit and a destination unit and copies the state between the two resolved keys. When both sides use the same S3 backend it copies through the AWS SDK (software development kit, the code library AWS ships) without invoking Terraform at all, and it refuses to run against a bucket with versioning switched off unless you pass --force.

$ cd ~/live/prod/us-east-1
$ cp -R vpc network
$ terragrunt backend migrate vpc network
$ cd network && terragrunt plan
# only once the plan below is clean:  rm -rf ../vpc

14:47:09.551 info Migrating state from prod/us-east-1/vpc/terraform.tfstate
14:47:09.552 info Migrating state to prod/us-east-1/network/terraform.tfstate
14:47:11.084 info State migrated successfully
 
Acquiring state lock. This may take a few moments...
No changes. Your infrastructure matches the configuration.
 
Terraform has compared your real infrastructure against your configuration
and found no differences, so no changes are needed.

Catch A Collision Before It Ships

Two units resolving to the same key is the failure that hurts most, because both write to one file and each apply erases the other's resources from the ledger. It happens two ways. A unit hard-codes key with a literal string and overrides the root. Or, more quietly, someone copies root.hcl into every environment folder so dev can set its own region. find_in_parent_folders stops at the first match on the way up, so units under live/dev now measure from live/dev/root.hcl and units under live/prod measure from live/prod/root.hcl. live/dev/us-east-1/vpc and live/prod/us-east-1/vpc both collapse to us-east-1/vpc/terraform.tfstate, and two environments share one state file. Render every unit and look for duplicates.

$ cd ~/live
$ terragrunt render --all --format json --write
$ jq -r '.remote_state.config.key' $(find . -name terragrunt.rendered.json) \
    | sort | tee /tmp/keys.txt | uniq -d
$ wc -l < /tmp/keys.txt

us-east-1/vpc/terraform.tfstate
8

Eight units, and uniq -d printed a key that shows up more than once. That single line of output is a production incident that has not happened yet. Any output from uniq -d should fail the build, and the same check re-run after a merge tells you the tree is still sane. It costs a couple of seconds and it is the only automated way to know that fifty units are still fifty separate ledgers.

Guard the line that decides all of this. Put live/root.hcl behind a code owners rule so the platform or security team reviews any edit to the key expression or the skip_ options, and make the duplicate-key check a required job on every pull request. One expression in one file decides where every secret-bearing state document in the repo lands, and whether an IAM policy can tell dev from prod by prefix. Read a change to it the way you read a change to a firewall rule.

Try this

Run terragrunt init >/dev/null on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a renamed folder is a brand new state file. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related