The execution model

How Terragrunt calls Terraform.

Terragrunt has no plan engine. No state file parser, no provider plugins, no resource graph of your infrastructure anywhere in it. Every terragrunt command you type is a few seconds of preparation followed by handing the real work to a completely different program. A prep cook does the same job: chop the onions, measure the stock, line up the pans, then slide the tray down to the line cook who works the fire. Terragrunt chops. OpenTofu or Terraform cooks.

One caveat, and it matters if you are the person signing off on this pipeline. Terragrunt is not entirely hands-off with your cloud account. It ships real AWS and Google Cloud client code for two narrow jobs: creating the state backend if it does not exist yet (an S3 bucket, a lock table, a GCS bucket), and assuming an IAM role (Identity and Access Management, the cloud's permission system) so the child process starts life with temporary credentials. So Terragrunt can hold and spend credentials of its own. What it never does is plan or apply your resources. That part is always the child.

The handoff between the two is the interesting part if you have to audit an infrastructure pipeline. Generated backend configuration, resolved input values, module code pulled off the internet ninety seconds ago, credentials borrowed from a role: all of it crosses into the child process in a single instant. Read that instant and you can answer the question every incident review opens with. What exactly ran, with which values, against which account? Miss it and you are trusting a black box that holds administrator rights over your estate.

Two Programs, One Command Line

Run terragrunt plan and two processes exist. Terragrunt starts, reads terragrunt.hcl (HCL is HashiCorp Configuration Language, the syntax those config files are written in), does its preparation, then launches tofu or terraform as a child process and steps back. The child's stdout and stderr (its normal output and its error output) get streamed back through Terragrunt's own logger, which is why genuine OpenTofu output arrives in your terminal tagged STDOUT. If you would rather have the child's output raw and unwrapped, --tf-forward-stdout (env TG_TF_FORWARD_STDOUT) turns that wrapping off. For a single unit, the exit code your shell sees is the child's own. And any flag Terragrunt does not recognise as one of its own gets forwarded untouched, so -target, -out, -var and -replace behave exactly as you already know them.

Common subcommands have shortcuts, so terragrunt plan works on its own. Anything without a shortcut needs the explicit form, terragrunt run -- workspace list, where the bare -- means everything past this point belongs to the child. Two renames from the command line redesign are worth having in your fingers. The old run-all is now run --all. And every TERRAGRUNT_* environment variable is now TG_*, with the --terragrunt- prefix stripped off the front of each flag.

# Shortcut form. Terragrunt owns the subcommand name and forwards
# every flag it does not recognise straight through to the child.
terragrunt plan -out tfplan -target=aws_s3_bucket.logs
 
# No shortcut for this one, so the child's command goes after a bare --
terragrunt run -- workspace list
 
# Now watch the whole handoff, step by step.
terragrunt --log-level debug plan

14:20:38.431 DEBUG  Terragrunt Version: 1.1.1
14:20:38.433 DEBUG  Did not find any locals block: skipping evaluation.
14:20:38.436 DEBUG  Running command: tofu --version
14:20:38.502 DEBUG  Downloading Terraform configurations from git::ssh://[email protected]/acme/tf-modules.git//vpc?ref=v1.4.0 into .terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA
14:20:40.118 DEBUG  Copying files from . into .terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc
14:20:40.155 DEBUG  Setting working directory to .terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc
14:20:40.157 DEBUG  Running command: tofu init
14:20:42.902 STDOUT tofu: Initializing the backend...
14:20:44.118 STDOUT tofu: OpenTofu has been successfully initialized!
14:20:44.121 DEBUG  Running command: tofu plan
14:20:46.660 STDOUT tofu: No changes. Your infrastructure matches the configuration.

Six debug lines tell you the whole story: which binary, which module, which ref, which directory, which subcommand, in what order. Keep --log-level debug (env TG_LOG_LEVEL) in your pocket. It is the difference between guessing what a pipeline did and reading it.

Which Binary Actually Runs

tofu is a name, not an address. Write a name on an envelope with no street under it and delivery depends entirely on who answers to that name first. Terragrunt looks for a binary called tofu on your PATH (the ordered list of directories your shell searches through when you type a command) and falls back to terraform if that is the only one installed. Whichever one it finds gets launched with your cloud credentials sitting in its environment and no restrictions on what it may do with them.

That is a real attack path, not a theoretical one. Anything that can write a file named tofu into a directory appearing earlier in PATH owns your next apply. A continuous integration runner that prepends a cached ./bin, a helper script that pushes ~/.local/bin to the front, a poisoned base image: any of those beats /usr/local/bin in the search order without a single character changing in your Terragrunt config. The fix is boring and it works. Pin an absolute path with --tf-path (env TG_TF_PATH), which nothing can reorder, then check the binary's checksum against the release you meant to install.

# What would actually be executed, in PATH order?
which -a tofu terraform
 
# Pin it to an absolute path so PATH order stops mattering.
export TG_TF_PATH=/usr/local/bin/tofu
terragrunt --log-level debug plan 2>&1 | grep 'Running command'
 
# And confirm the binary is the one you shipped.
sha256sum /usr/local/bin/tofu

/usr/local/bin/tofu
/usr/bin/terraform
14:31:02.118 DEBUG  Running command: /usr/local/bin/tofu --version
14:31:02.140 DEBUG  Running command: /usr/local/bin/tofu init
14:31:04.552 DEBUG  Running command: /usr/local/bin/tofu plan
2f5c0f4e6f7a4d24a4d8b8e9ea2f8ff31f6ffb4e1b6f2f5c3d5b0c8a6e7d1a92  /usr/local/bin/tofu

The Cache Is the Real Working Directory

Terragrunt almost never runs OpenTofu in the folder you are standing in. It behaves like a photocopy shop. Your originals stay where they are, a working copy gets assembled on a bench out the back, and all the messy work happens on the copy. If your unit declares a terraform { source = ... } block, Terragrunt fetches that module (copying a local path, or pulling a remote reference through go-getter, the same download library Terraform uses), copies every file from your unit directory on top of it, drops in whatever your generate and remote_state blocks produce, and starts the child process there. That directory is where .terraform/, the provider lockfile, and any saved plan actually live.

The two levels of hash directory under .terragrunt-cache are both meaningful, and knowing which is which saves you a lot of guessing. The first is a base64-encoded SHA-1 (a short fixed-length fingerprint of a piece of text) of the absolute path of the unit you are standing in, which stops two concurrent Terragrunt runs in different directories from trampling each other. The second is a SHA-1 of the source URL with its query string stripped off, so the same module repo always lands in the same folder no matter which ref you asked for. The ref gets tracked separately, in a small file named .terragrunt-source-version that holds a hash of that query string.

cd ~/live/prod/vpc
terragrunt plan > /dev/null
find .terragrunt-cache -maxdepth 3 | sort

.terragrunt-cache
.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo
.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA
.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/.terragrunt-source-version
.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc

ls -1a .terragrunt-cache/*/*/vpc

.
..
.terraform
.terraform.lock.hcl
.terragrunt-module-manifest
backend.tf
main.tf
outputs.tf
terragrunt.hcl
variables.tf

Notice terragrunt.hcl sitting in there. Terragrunt copies every file from your unit directory into the cache, so a stray .tfvars, a leftover .env, or a private key somebody dropped next to your config all get copied along and become readable by the child process. Everything a defender wants to inspect lives in that directory too: the generated backend.tf, the .terraform.lock.hcl provider lockfile, the downloaded provider binaries under .terraform/, and any saved plan file. A saved plan holds resource attribute values in the clear, including the ones your state marks sensitive. Put .terragrunt-cache in .gitignore and in whatever your CI wipes between jobs, or move it off the repo entirely with --download-dir (env TG_DOWNLOAD_DIR).

How Values Cross the Line

Your inputs block does not become a .tfvars file. Think of a note pinned to the doorframe rather than a letter handed over in person: Terragrunt writes each input into the child process's environment as TF_VAR_<name>, and Terraform reads them off the frame on its own way in. Plain strings pass through raw. Everything else (lists, maps, numbers, booleans) gets JSON-encoded first, because JSON (a plain-text format for structured data) is what Terraform expects to find in an environment variable. Null has no representation in an environment variable at all, so Terragrunt puts those in a temporary .terragrunt-null-vars.auto.tfvars.json file inside the working directory and deletes it when the run ends.

terraform {
  source = "git::ssh://[email protected]/acme/tf-modules.git//vpc?ref=v1.4.0"
}
 
inputs = {
  environment   = "prod"
  instance_type = "t3.micro"
  azs           = ["eu-west-1a", "eu-west-1b"]
  tags          = { team = "platform", pci = "true" }
  kms_key_arn   = null
}

To see the handoff instead of reasoning about it, ask Terragrunt to write it down. --inputs-debug (env TG_INPUTS_DEBUG) dumps every resolved input to a file and prints the exact command needed to reproduce the run with Terragrunt out of the picture.

terragrunt run --log-level debug --inputs-debug -- plan

14:52:10.004 INFO   Debug mode requested: generating debug file terragrunt-debug.tfvars.json in working dir /home/ops/live/prod/vpc/.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc
14:52:10.007 DEBUG  The following variables were detected in the tofu module:
14:52:10.007 DEBUG  [environment instance_type azs tags kms_key_arn]
14:52:10.009 INFO   Variables passed to tofu are located in "/home/ops/live/prod/vpc/.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc/terragrunt-debug.tfvars.json"
14:52:10.009 INFO   Run this command to replicate how tofu was invoked:
14:52:10.009 INFO   	tofu -chdir="/home/ops/live/prod/vpc/.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc" plan -var-file="/home/ops/live/prod/vpc/.terragrunt-cache/YwxRh1t9S1MqUyuStjvIF7JSSJo/SfpMDREs7SBbO1m5qnvv01kOtWA/vpc/terragrunt-debug.tfvars.json"

cd .terragrunt-cache/*/*/vpc
ls -l terragrunt-debug.tfvars.json
cat terragrunt-debug.tfvars.json

-rw------- 1 ops ops 191 Jul 21 14:52 terragrunt-debug.tfvars.json
{
  "azs": [
    "eu-west-1a",
    "eu-west-1b"
  ],
  "environment": "prod",
  "instance_type": "t3.micro",
  "kms_key_arn": null,
  "tags": {
    "pci": "true",
    "team": "platform"
  }
}

That replicate line is the whole execution model written out as one command: OpenTofu, run by hand, in the directory Terragrunt built, with the variables Terragrunt resolved. Paste it and you find out in thirty seconds whether a failure belongs to your Terragrunt config, the module, or the provider. Note where the file landed. It goes in the working directory Terragrunt ended up using, which with a source block means inside .terragrunt-cache, not next to your terragrunt.hcl. Read the log line rather than guessing. The mode is 0600, readable and writable only by the user who owns it, but it is still plain text holding every resolved value including anything sensitive, and nothing cleans it up for you. Gitignore it and delete it when you are done. For the config side rather than the variable side, terragrunt render --format json --write resolves the whole terragrunt.hcl, includes and locals and dependencies and all, into terragrunt.rendered.json in the current directory. -w is the short form of --write.

Environment variables are not a private channel. On Linux every process's environment sits in /proc/<pid>/environ, readable by the user who owns the process and by root. More to the point, everything inside the child's process tree inherits it: each provider plugin (a separate binary that Terraform downloads and executes), every local-exec provisioner, every external data source. Put a database password in inputs and you have handed it to third-party code you did not write and cannot read. Pass secrets by reference instead, with a data source that reads your secrets manager at apply time, or a provider that assumes a role, so the value never sits in the environment at all.

# Run this in a second shell while a plan is in flight.
tr '\0' '\n' < /proc/$(pgrep -n tofu)/environ | grep '^TF_VAR_'

TF_VAR_environment=prod
TF_VAR_instance_type=t3.micro
TF_VAR_azs=["eu-west-1a","eu-west-1b"]
TF_VAR_tags={"pci":"true","team":"platform"}

Four variables, not five. kms_key_arn was null, so it never became an environment variable at all. It travelled in that temporary auto-loaded file inside the cache directory instead, which is exactly why the debug dump above shows five keys and this shows four.

Two Things That Run Without You Asking

Between your keystroke and OpenTofu's first line of output, two things happen that you did not request.

The first is auto-init. If Terragrunt decides the module is not initialised, it runs init for you, which is why you rarely type it. Convenient, and also a hole in a locked-down pipeline: a plan step can reach out to a provider registry and a Git host and pull down executable code before you have seen a single line of diff. If your policy says the plan stage gets no outbound network, or that providers come only from your internal mirror, run one controlled terragrunt init and pass --no-auto-init (env TG_NO_AUTO_INIT) everywhere after. Be clear about what that flag does, though. It does not stop the run. Terragrunt logs Detected that init is needed, but Auto-Init is disabled. Continuing with further actions, but subsequent terraform commands may fail. and carries on regardless, so the failure lands later and looks like something else. Treat that line as a build-breaking condition in your own tooling if you want it enforced.

The second is auto-retry. Terragrunt matches the child's output against thirty built-in regular expressions (text patterns) for transient failures, things like TLS handshake timeouts, provider registry deadlines and 429 Too Many Requests, and re-runs the whole command on a match. Three attempts total, five seconds apart, by default. You can print the exact list from HCL with get_default_retryable_errors(), and tune the behaviour yourself in an errors block with retryable_errors, max_attempts and sleep_interval_sec. Two things to know before you lean on it. A retried apply re-runs the entire apply, and every before_hook fires again, so a hook that is not safe to run twice will run twice. And one of the thirty patterns is the literal string NoSuchBucket: The specified bucket does not exist, which means that if somebody deletes your state bucket you burn three attempts and fifteen seconds before the real problem reaches your screen. --no-auto-retry (env TG_NO_AUTO_RETRY) turns the whole thing off, which is what you want anywhere a failure should be loud and immediate.

# One controlled init, against the lockfile you already reviewed.
terragrunt run --log-level debug -- init -lockfile=readonly
 
# Every run after that: no surprise network calls, no silent retries,
# no prompts, and OpenTofu's real exit code back in your hands.
terragrunt run \
  --no-auto-init \
  --no-auto-retry \
  --non-interactive \
  --log-level debug \
  -- plan -detailed-exitcode
echo "tofu exit code: $?"

14:57:44.208 DEBUG  Running command: /usr/local/bin/tofu init -lockfile=readonly
14:57:47.913 STDOUT tofu: OpenTofu has been successfully initialized!
14:58:02.114 DEBUG  Running command: /usr/local/bin/tofu --version
14:58:02.310 DEBUG  Running command: /usr/local/bin/tofu plan -detailed-exitcode
14:58:05.771 STDOUT tofu: OpenTofu will perform the following actions:
14:58:05.774 STDOUT tofu: Plan: 1 to add, 0 to change, 0 to destroy.
tofu exit code: 2

That exit code is the payoff of the passthrough. -detailed-exitcode still means what OpenTofu says it means: 0 for no changes, 1 for a failure, 2 for a diff. A nightly job that plans every unit and alerts on exit 2 is drift detection, and drift is what you see when somebody clicks around in the console at 2am or a leaked access key gets used by a stranger. Switch --log-format to key-value or json when those logs go somewhere that has to parse them rather than somewhere a human reads them.

One last trap, and it is a quiet one. Because inputs travel as TF_VAR_ environment variables, they inherit Terraform's rule for them: an environment variable with no matching variable block is ignored. No error, no warning, nothing. Type instnace_type instead of instance_type and Terragrunt cheerfully exports TF_VAR_instnace_type. Terraform finds nothing to bind it to, drops it on the floor, and falls back to the variable's default. The plan looks clean. The instance comes up the wrong size, or the encryption flag stays off. Terraform only complains about the reverse case, a required variable with no value at all. So there are two defences worth building in. Give security-relevant variables no default, so a missing input fails loudly instead of quietly. And when a value refuses to take effect, run terragrunt run --log-level debug --inputs-debug -- plan, which prints WARN: The variable instnace_type was omitted because it is not defined in the OpenTofu/Terraform module. That line is your typo, named out loud.

When you inherit a Terragrunt repo you did not write, run one unit with the log level turned up and pull four lines out of the noise: terragrunt run --log-level debug --inputs-debug -- plan 2>&1 | grep -E 'Running command|Downloading|Copying files|Variables passed'. Those four answer which binary executed, which module and which ref it came from, which directory it ran in, and where the resolved variables landed. That is how "the pipeline did something to prod" turns into a command you can paste into a ticket.

Try this

Run terragrunt plan -out tfplan -target=aws_s3_bucket.logs on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a cached module never refreshes itself. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related