Generating backend & provider config

Stop copy-pasting boilerplate.

Fifty folders in one repository, and fifty copies of the same twelve lines. Every Terraform module needs two bits of wiring before it can do anything useful. A provider block (settings for the plugin that talks to Amazon, Azure or Google, mostly which account and which region) and a backend block (where to keep the state file, the ledger that records every resource the tool has created). Copy that wiring into fifty places and you have fifty places to edit when the region changes or the state bucket moves. Terragrunt's generate block is a rubber stamp for it. You cut the stamp once, and Terragrunt presses it onto each module's working directory in the moment before it calls the engine (the tofu or terraform binary that does the real work). Terragrunt calls each of those folders a unit: one directory holding one terragrunt.hcl file, describing one deployment. HCL is HashiCorp Configuration Language, the same syntax Terraform itself uses. The unit stays clean, the module it points at often holds nothing but main.tf and variables.tf, and the repetitive plumbing lives in one file a reviewer can actually read.

Boilerplate Rots Quietly

Duplication is annoying. What makes it a security problem is that copies drift and nobody notices. Forty-nine of your backend blocks say encrypt = true and one, added in a hurry eighteen months ago, does not. That unit has stopped asking S3 (Amazon's object storage service, where the state file lives) to encrypt the object on the way in. Nothing breaks. Nothing warns you. S3 has applied its own default encryption to new objects for years, so the file is not lying there in plaintext, but that one unit is now protected by whatever the bucket happens to do rather than by the key you chose. Tighten the bucket policy later to demand a specific KMS key (Key Management Service, Amazon's service for keys you control and can audit) and that unit is the one failing at three in the morning. It matters because state is the most sensitive file in the repo: database passwords, generated private keys, and a complete inventory of your estate sitting in readable JSON. The provider block rots the same way. One unit still pinned to the old account, or missing the guard that checks which account your credentials actually resolved to, and a routine apply reaches somewhere it was never meant to touch. Nobody reviews fifty identical files. People skim them. Nobody skims one.

Stamping A Provider Block

A generate block lives in a terragrunt.hcl and describes a file to put on disk. Four things matter. The label after generate is a name you pick, so one config can carry several stamps. path is the filename to write, relative to the module's working directory. contents is the literal body of the file, almost always a heredoc (text fenced by <<EOF ... EOF, so you can paste real HCL without escaping anything). if_exists decides what happens when something is already sitting at that path, and Terragrunt makes you say it out loud. There is no default to fall back on: leave it out and the parse fails with generate block "provider" is missing required attribute "if_exists".

include "root" {
  path = find_in_parent_folders("root.hcl")
}
 
terraform {
  source = "git::ssh://[email protected]/acme/tf-modules.git//vpc?ref=v1.4.0"
}
 
# Written to provider.tf in the module's working dir, before the engine runs
generate "provider" {
  path      = "provider.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
provider "aws" {
  region = "eu-west-1"
 
  # Refuse to run if these credentials resolve to any other account
  allowed_account_ids = ["123456789012"]
 
  default_tags {
    tags = {
      managed_by = "terragrunt"
    }
  }
}
EOF
}
 
inputs = {
  name       = "prod"
  cidr_block = "10.0.0.0/16"
}

Working directory is the phrase to hold onto. When a unit sets terraform { source = ... }, Terragrunt copies the module into a hidden .terragrunt-cache/ folder beside your terragrunt.hcl and runs the engine inside that copy, the way you photocopy a form before filling it in rather than writing on the original. Generated files land in the copy, not in your unit folder, sitting next to the module's own .tf files as though they had always belonged. Ask Terragrunt to show its work and you can watch it happen. Terragrunt writes its logs to standard error (the second output stream every command has, kept separate from its normal output), which is what the 2>&1 is for.

cd ~/live/prod/eu-west-1/vpc
terragrunt plan --log-level debug 2>&1 | grep -i 'generated file'

14:07:22.318 DEBUG  Generated file /home/eng/live/prod/eu-west-1/vpc/.terragrunt-cache/yBmMEuFHwVMDMHmZfTBpBqzXEIE/THBFOKzKNXCS_gWlQx1QMDrqK4M/vpc/backend.tf.
14:07:22.319 DEBUG  Generated file /home/eng/live/prod/eu-west-1/vpc/.terragrunt-cache/yBmMEuFHwVMDMHmZfTBpBqzXEIE/THBFOKzKNXCS_gWlQx1QMDrqK4M/vpc/provider.tf.

cat .terragrunt-cache/*/*/vpc/provider.tf

# Generated by Terragrunt. Sig: nIlQXj57tbuaRZEa
provider "aws" {
  region = "eu-west-1"
 
  # Refuse to run if these credentials resolve to any other account
  allowed_account_ids = ["123456789012"]
 
  default_tags {
    tags = {
      managed_by = "terragrunt"
    }
  }
}

That first line is a signature, and it earns its keep further down. Two things to know about it now. It is a fixed marker, the same sixteen characters in every file Terragrunt has ever written, not a checksum of the contents, so changing the body below it invalidates nothing. And Terragrunt only ever reads line one when it goes looking for that marker. Now look at what the stamp bought you. allowed_account_ids is present in every unit that includes this config, for free, and it costs nothing to keep there. Set the wrong profile, inherit a stale role in a pipeline, or run on a build machine whose credentials someone has quietly swapped, and the provider stops before it reads a single resource.

AWS_PROFILE=sandbox terragrunt plan

╷
│ Error: configuring Terraform AWS Provider: AWS account ID not allowed: 210987654321
│ 
│   with provider["registry.terraform.io/hashicorp/aws"],
│   on provider.tf line 2, in provider "aws":
│    2: provider "aws" {
│ 
╵

A hard stop before the plan, not a warning buried in a diff that a tired reviewer approves at 18:00. Note the line number in that error. The signature occupies line 1, so the provider block starts on line 2, which is a quick way to tell generated code from hand-written code when you are reading an error at speed. Hand-write that guard fifty times and it will be present in forty-two of them. Stamp it and the count is exact, and you can prove the count in one command.

The Backend Gets Its Own Helper

You could stamp the backend with a plain generate block, and it would work. Terragrunt gives you something better: remote_state, a block that understands backends and carries its own generate attribute. Describe the backend once, in the root config every unit includes, and Terragrunt writes the matching backend.tf into each module and points the engine's init step (the setup command that downloads plugins and connects to the state store) at it. Leave the generate attribute off and Terragrunt falls back to passing -backend-config flags to init instead, which works but leaves nothing on disk for you or an auditor to read. Pick one mechanism for a given backend, never both.

remote_state {
  backend = "s3"
 
  generate = {
    path      = "backend.tf"
    if_exists = "overwrite_terragrunt"
  }
 
  config = {
    bucket  = "acme-tfstate-prod"
    key     = "${path_relative_to_include()}/terraform.tfstate"
    region  = "eu-west-1"
    encrypt = true
 
    # Native S3 locking (Terraform 1.10+ / OpenTofu 1.10+).
    # Keep dynamodb_table beside it only while you migrate off DynamoDB.
    use_lockfile = true
 
    # Terragrunt-only keys. These never reach the generated backend.tf.
    skip_bucket_versioning = false
    s3_bucket_tags = {
      owner = "platform"
    }
  }
}

The key is the path of the state object inside the bucket, and path_relative_to_include() is what stops units from stepping on each other. Think of it as a room number worked out from where the room sits in the building. It returns the path from the directory holding the included root config down to the current unit. A unit at live/prod/eu-west-1/vpc that includes live/root.hcl gets prod/eu-west-1/vpc, so its state object is prod/eu-west-1/vpc/terraform.tfstate. Every unit lands somewhere different, automatically, and nobody has to remember to edit a string after copying a folder.

That config map is not passed through verbatim, which surprises people. Terragrunt pulls out its own keys first, the ones the engine's S3 backend has never heard of, and writes what remains sorted alphabetically. skip_bucket_versioning, s3_bucket_tags and about twenty siblings control how Terragrunt provisions the bucket and the DynamoDB lock table (DynamoDB is Amazon's key-value database, used here as the traffic light that stops two people applying at once), not how the engine talks to them. Recent versions make that provisioning an explicit act. Run terragrunt backend bootstrap, or pass --backend-bootstrap on a normal run, and Terragrunt creates the bucket with versioning on, server-side encryption on, public access blocked, and a policy that refuses plain HTTP so state can only move over TLS (Transport Layer Security, the encryption behind the padlock in a browser). Each skip_* flag switches one of those protections off. Treat a pull request adding skip_bucket_public_access_blocking = true or skip_bucket_enforced_tls = true as a finding, not a preference.

cat .terragrunt-cache/*/*/vpc/backend.tf

# Generated by Terragrunt. Sig: nIlQXj57tbuaRZEa
terraform {
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    encrypt      = true
    key          = "prod/eu-west-1/vpc/terraform.tfstate"
    region       = "eu-west-1"
    use_lockfile = true
  }
}

Read that output as a receipt. The skip_* and tag keys are gone, filtered out as expected. The key has been computed into a real path, so you can see with your own eyes which object this unit will write. Terragrunt also ran the file through the HCL formatter on the way out, which it does for .tf, .hcl and .tofu files by default, and that is why the = signs line up when nothing in your config asked for it. This file is the single best thing to look at when someone asks whether staging and prod really are separated.

if_exists And The Mark That Makes It Safe

The # Generated by Terragrunt. Sig: ... line is not decoration. It is a maker's mark, the way a potter presses a stamp into the base of a bowl, and it is what makes one of the if_exists values safe to use. overwrite_terragrunt opens whatever file is already sitting at that path and reads the first line. Ends with the mark? Overwrite it, we made it. Anything else? Stop, because that file is somebody's hand-written work and clobbering it silently would be theft. You get ERROR: The file path /...​/provider.tf already exists and was not generated by terragrunt, then the run dies with Can not generate terraform file: /...​/provider.tf already exists.

Four values exist and you must pick one. overwrite_terragrunt is what you want nearly always. skip leaves an existing file alone, which sounds gentle and hides staleness: the module keeps using whatever was there while your terragrunt.hcl quietly becomes fiction. overwrite replaces unconditionally, mark or no mark. error refuses if anything at all exists.

generate "provider" {
  path              = "provider.tf"
  if_exists         = "overwrite_terragrunt"  # replace only files we stamped
  disable_signature = false                   # keep the maker's mark (default)
  comment_prefix    = "# "                    # how that mark is commented out
  contents          = <<EOF
provider "aws" {
  region = "eu-west-1"
}
EOF
}
 
# if_exists values (required, there is no default):
#   overwrite_terragrunt   replace only files whose first line carries the mark
#   skip                   keep whatever is already there, generate nothing
#   overwrite              replace unconditionally, mark or not
#   error                  fail if any file exists at that path

comment_prefix exists because the mark has to be a comment in whatever language you are generating, and it defaults to # . Generating a .tfvars.json file? JSON (JavaScript Object Notation, the plain data format with all the braces) has no comment syntax at all, so you set disable_signature = true and the mark is left out. Fair trade, but know the bill. The file Terragrunt writes now has an ordinary first line, so on the next run overwrite_terragrunt reads that line, fails to recognise its own work, and refuses to continue. On any file where you switch the mark off, if_exists has to be overwrite, and you should mean it, because overwrite will flatten a colleague's hand-written file without a word.

When The Module Brings Its Own Provider

Here is the failure you will actually hit, and if_exists cannot save you from it. Plenty of community modules ship a provider "aws" block inside their own source, usually tucked into main.tf. Terragrunt drops your generated provider.tf into that same directory. Different filename, so there is no path clash for the mark check to catch. Generation succeeds. The engine then parses every .tf file in the directory as one configuration and finds two default providers with the same name.

terragrunt plan

╷
│ Error: Duplicate provider configuration
│ 
│   on provider.tf line 2:
│    2: provider "aws" {
│ 
│ A default provider configuration for "aws" was already given at
│ main.tf:11,1-15. If multiple configurations are required, set the "alias"
│ argument for alternative configurations.
╵

The real fix is upstream, in the module. A reusable module should declare required_providers (which plugins it needs and which versions are acceptable) and stop there. The concrete configuration, the region, the role to assume, the account guard, belongs to whoever calls the module, which is Terragrunt. If you cannot change the module, wrap it in a thin local module of your own that leaves the provider block out, or set disable = true on the generate block for that one unit and let the module's own provider config take charge. Watch that second option. disable stops future writes, but if_disabled defaults to skip, so a provider.tf from an earlier run stays in the cache and keeps causing the same clash. Add if_disabled = "remove_terragrunt" and Terragrunt clears away the file it made. When the module ships a file at the exact same path, its own provider.tf, the mark check does fire, and Terragrunt stops instead of overwriting it.

Check It From The Outside

Two questions to answer after any change to a generate or remote_state block. Did every unit get what you meant, and is anything on disk pretending to be generated when it is not? The mark answers both, because it is a fixed string you can grep for.

# Which files in this unit did Terragrunt actually write?
grep -rl 'Generated by Terragrunt' ~/live/prod/eu-west-1/vpc
 
# Which files did somebody commit that Terragrunt is supposed to own?
git -C ~/live ls-files '*provider.tf' '*backend.tf'

/home/eng/live/prod/eu-west-1/vpc/.terragrunt-cache/yBmMEuFHwVMDMHmZfTBpBqzXEIE/THBFOKzKNXCS_gWlQx1QMDrqK4M/vpc/backend.tf
/home/eng/live/prod/eu-west-1/vpc/.terragrunt-cache/yBmMEuFHwVMDMHmZfTBpBqzXEIE/THBFOKzKNXCS_gWlQx1QMDrqK4M/vpc/provider.tf
staging/eu-west-1/rds/provider.tf

The first two lines are healthy: marked files, inside the cache, exactly where the engine will read them. The last line should stop your afternoon. Somebody committed a provider.tf. Nothing generated it, so it carries no mark, nobody reviews it, and Terragrunt will refuse to replace it on the next run. Delete it, then make sure it cannot come back.

The other thing to check after a backend change is that no two units share a state file. List the bucket and count the keys against the units you expect.

aws s3 ls s3://acme-tfstate-prod --recursive | awk '{print $4}'

prod/eu-west-1/rds/terraform.tfstate
prod/eu-west-1/vpc/terraform.tfstate
staging/eu-west-1/vpc/terraform.tfstate

One key per unit, each mirroring the folder layout, which is what a correct path_relative_to_include() looks like from the outside. A single ./terraform.tfstate sitting at the top of the bucket means something else entirely.

Make CI Refuse The Stale Copy

One check in your pipeline keeps the whole scheme honest. CI is continuous integration, the automated job that runs on every push. If a file Terragrunt is supposed to write ever lands in Git, fail the build before anyone plans against it.

#!/usr/bin/env bash
set -euo pipefail
 
# Terragrunt owns these filenames. They must never be committed.
# The leading * matches at any depth, so nested units are covered too.
committed=$(git ls-files '*provider.tf' '*backend.tf')
 
if [ -n "$committed" ]; then
  echo "Terragrunt generates these files. They must not be in git:" >&2
  echo "$committed" >&2
  exit 1
fi

bash ci/no-generated-files.sh
echo "exit code: $?"

Terragrunt generates these files. They must not be in git:
staging/eu-west-1/rds/provider.tf
exit code: 1

Run it in the same job as terragrunt hcl validate, before any plan executes, so a forgotten copy never gets the chance to decide which account your credentials point at.

Try this

Run terragrunt plan --log-level debug 2>&1 | grep -i 'generated file' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a committed provider.tf is an attack surface. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related