Remote-state & credential security

Protect the crown jewels.

Terraform state is a locksmith's job book. It lists every lock in the building, which door each one sits on, and the day it was fitted. Terraform's copy has one extra habit: it tapes a spare key inside the back cover. State is the file the tool writes after every run so it can remember what it built, and it records the resolved value of every attribute it manages. Plenty of those attributes are passwords, tokens and private keys. That makes one plain-text file the highest-value object you own. Terragrunt raises the stakes, because a single backend (the shared storage where state lives, an S3 bucket here, S3 being Amazon's Simple Storage Service) now holds the state for every unit in every account. The remote-state lesson wired that backend up once for the whole repo. This one hardens it, proves the hardening actually took, and cuts the credentials that reach it down to something that expires on its own.

Read Your Own State Before Someone Else Does

Arguments about state security get short once somebody looks at the file. The command below pulls the current state down from the backend and prints it to the screen, unchanged. It only reads, so it is safe to run against production, but the text it prints is exactly as sensitive as the object sitting in the bucket. Pipe it through jq (a small command line tool for picking fields out of JSON, the text format state is written in) and ask for the attributes of one database.

cd live/prod/us-east-1/app/rds
 
# 'run --' hands everything after it to OpenTofu/Terraform unchanged.
# (Terragrunt before the CLI redesign: terragrunt state pull)
terragrunt run -- state pull | jq '
  .resources[]
  | select(.type == "aws_db_instance")
  | .instances[].attributes
  | {id: .identifier, user: .username, password: .password}'

{
  "id": "app-prod",
  "user": "appadmin",
  "password": "Tr0ub4dor&3-2026-05"
}

That password is not there because somebody was careless. Terraform keeps every attribute a provider hands back so the next plan has something to compare against, and marking a variable sensitive = true only masks the value in terminal output. It changes nothing inside the file. Secrets you never typed land there too. An aws_iam_access_key resource writes its secret access key into state, tls_private_key writes the whole private key, and any module that generates its own master password writes that. There are a couple of exits now. Setting manage_master_user_password = true on aws_db_instance hands the password to AWS Secrets Manager (a managed vault) and leaves only an ARN (Amazon Resource Name, the unique address of a thing in AWS) behind, and Terraform 1.11 added write-only arguments such as password_wo, which a provider consumes but never persists. Both are opt-in, and both are the exception. Work from the assumption that every secret your estate has ever touched is sitting in that bucket in readable form.

Harden the Bucket, Then Prove It

Terragrunt can create the state bucket and lock table for you, and its defaults are already sane: versioning on, server-side encryption on, a bucket policy that denies any request not carried over TLS (Transport Layer Security, the padlock behind the https:// in an address), and all four public-access blocks set. Each skip_* setting switches one of those off. Treat them like a fire alarm you never pull, because reaching for one to clear an error is how a state bucket ends up readable by strangers. Two settings take you past the defaults, and they are easy to mix up. kms_key_id tells Terraform which key to encrypt the state object with. bucket_sse_kms_key_id tells Terragrunt which key to make the bucket's default. Different knobs. Set only the first and anything else written to that bucket falls back to the AWS-managed aws/s3 key, which you cannot attach your own policy to. Point both at a customer managed key (a key in KMS, the Key Management Service, whose access policy you write yourself) living in the same account as the state.

locals {
  account_vars  = read_terragrunt_config(find_in_parent_folders("account.hcl"))
  account_id    = local.account_vars.locals.aws_account_id
  account_name  = local.account_vars.locals.account_name       # "prod", "staging"
  state_key_arn = local.account_vars.locals.state_kms_key_arn  # one key per account
}
 
# One hardened backend, inherited by every unit in the repo.
remote_state {
  backend  = "s3"
  generate = { path = "backend.tf", if_exists = "overwrite_terragrunt" }
 
  config = {
    bucket = "acme-tfstate-${local.account_id}"
    key    = "${path_relative_to_include()}/terraform.tfstate"
    region = "us-east-1"
 
    encrypt    = true                  # encrypt every object Terraform writes
    kms_key_id = local.state_key_arn   # a key you own, not aws/s3
 
    # Locking: one writer at a time, so two applies cannot shred the file.
    # dynamodb_table = "acme-tfstate-locks"   # deprecated in Terraform 1.11
    use_lockfile = true                       # S3-native lock object, no extra table
 
    # --- Terragrunt-only knobs: these shape the bucket Terragrunt creates ---
    bucket_sse_algorithm        = "aws:kms"           # bucket default encryption...
    bucket_sse_kms_key_id       = local.state_key_arn # ...uses the SAME key
    accesslogging_bucket_name   = "acme-tfstate-access-logs"
    accesslogging_target_prefix = "tfstate/"
 
    # The defaults are the control. Leaving these false is a decision, not laziness.
    skip_bucket_versioning             = false  # keep history to recover corrupt state
    skip_bucket_enforced_tls           = false  # policy denies non-TLS requests
    skip_bucket_public_access_blocking = false  # block all four public paths
    enable_lock_table_ssencryption     = true   # encrypts the lock table, if you keep one
  }
}

Locking keeps two applies from writing at the same moment and leaving you with half a ledger. A DynamoDB table (Amazon's key-value database) did that job for years. use_lockfile = true drops a small lock object into the same bucket instead, supported from Terraform 1.10 and OpenTofu 1.10 onward, and Terraform 1.11 deprecates the standalone dynamodb_table argument. Run terragrunt backend bootstrap to create or repair the bucket, its policies and the table from this same block. Then stop trusting the tool's own log lines and ask AWS what it actually built.

BUCKET=acme-tfstate-111122223333
 
terragrunt backend bootstrap   # creates or repairs bucket, policies and lock table
 
aws s3api get-bucket-versioning   --bucket "$BUCKET"
aws s3api get-public-access-block --bucket "$BUCKET" --query PublicAccessBlockConfiguration
aws s3api get-bucket-encryption   --bucket "$BUCKET" \
  --query 'ServerSideEncryptionConfiguration.Rules[0]'
aws s3api get-bucket-policy       --bucket "$BUCKET" --query Policy --output text \
  | jq -c '.Statement[] | select(.Sid == "AllowTLSRequestsOnly") | {Sid, Effect, Condition}'

{
    "Status": "Enabled"
}
{
    "BlockPublicAcls": true,
    "IgnorePublicAcls": true,
    "BlockPublicPolicy": true,
    "RestrictPublicBuckets": true
}
{
    "ApplyServerSideEncryptionByDefault": {
        "SSEAlgorithm": "aws:kms",
        "KMSMasterKeyID": "arn:aws:kms:us-east-1:111122223333:key/3f1c9e0a-7b52-4a1d-9d0e-51bb0a5f4c77"
    },
    "BucketKeyEnabled": false
}
{"Sid":"AllowTLSRequestsOnly","Effect":"Deny","Principal":"*","Condition":{"Bool":{"aws:SecureTransport":"false"}}}

Four questions, four answers you can hand an auditor. The last one is the one people misread. aws:SecureTransport set to false means the request arrived over plain HTTP, and that Deny applies to every principal including you, so a stray http:// endpoint in a script fails loudly instead of shipping your state across the wire in the clear. Hold on to that BucketKeyEnabled: false for a minute. It decides how much you get to see in the next section.

The Customer Key Is Also an Alarm Wire

Encryption at rest defeats exactly one attack: somebody reading the raw bytes without going through the S3 API. The bigger return on owning the key is the audit trail. Think of CloudTrail (the AWS service that writes down who called which API, and when) as the building's visitor book. Reads of S3 objects are recorded as data events, which are off by default and billed per event, so most accounts hold no record at all of who downloaded a state file. KMS behaves differently. Every Decrypt call is a management event, and the first copy of management events costs nothing in an account that has a trail. S3 cannot hand back plaintext without calling Decrypt first, so the key gives you the read log the bucket never had. The call carries an encryption context too, and for an object encrypted this way it holds aws:s3:arn naming the exact object, which means an alert can tell you which unit somebody read. Check that nobody has excluded kms.amazonaws.com from the trail as a cost saving, because that switch exists and it is popular.

# Who asked for state plaintext in the last day?
# (GNU date, as on Ubuntu/Debian; on macOS use: date -u -v-24H +%Y-%m-%dT%H:%M:%SZ)
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --start-time "$(date -u -d '24 hours ago' +%Y-%m-%dT%H:%M:%SZ)" \
  --max-results 3 \
  --query 'Events[].[EventTime,Username]' --output text

2026-07-21T09:14:22+00:00	tg-prod-15839201123
2026-07-21T09:14:21+00:00	tg-prod-15839201123
2026-07-21T08:58:03+00:00	jparker

Two of those are the pipeline, reading state at the start of a run and writing it at the end. The third is a person, at 08:58, pulling the plaintext of your production state by hand. That is either an engineer debugging without telling anyone or the first hour of an incident, and either way you want to know within minutes. Alert on any Decrypt against the state key whose principal is not the Terragrunt role and you have state-read detection that costs nothing to run. One trade-off comes attached. S3 Bucket Keys cache a data key for a short window and cut KMS traffic by up to 99%, which lowers the bill and thins the log at the same time, and they also coarsen the encryption context from the object ARN to the bucket ARN, so you lose the ability to say which unit was read. On a state bucket, per-read visibility is worth more than the saving. Leave BucketKeyEnabled false there.

Credentials That Expire While the Job Is Still Warm

A static access key is a key cut once and left under the mat. It works for whoever finds it, forever, and it never tells you who used it. A hotel keycard is the better model: issued at check-in, scoped to one room, dead at checkout. That is what iam_role buys you. Terragrunt calls AssumeRole (the operation that swaps your identity for a temporary one, served by STS, the Security Token Service, the desk that hands out short-lived credentials) before it touches state or runs Terraform, works with the credentials it gets back, and lets them lapse when the run ends. iam_assume_role_session_name is the piece defenders care about, because that string is stamped on every CloudTrail event the run produces. Make it carry the run identifier and one log line points at a single workflow execution instead of a shared robot account.

# Same file. Terragrunt assumes this role for state access AND for the run itself.
iam_role                     = "arn:aws:iam::${local.account_id}:role/terragrunt-state"
iam_assume_role_duration     = 3600  # seconds; the session dies after an hour
iam_assume_role_session_name = "tg-${local.account_name}-${get_env("GITHUB_RUN_ID", "local")}"
 
# Scope the role to this bucket, this key, this table. Nothing wider.
#   s3:GetObject, s3:PutObject, s3:DeleteObject  on acme-tfstate-<account>/*
#   s3:ListBucket                                on acme-tfstate-<account>
#   kms:Decrypt, kms:GenerateDataKey             on the one state key
# One role per account, so a mistake stays inside that account.

# Inside the job, after credentials are configured: who are we, and how long do we live?
aws sts get-caller-identity --query Arn --output text
 
aws sts assume-role \
  --role-arn arn:aws:iam::111122223333:role/terragrunt-state \
  --role-session-name tg-prod-15839201123 \
  --duration-seconds 43200 \
  --query 'Credentials.Expiration' --output text

arn:aws:sts::111122223333:assumed-role/terragrunt-state/tg-prod-15839201123
 
An error occurred (ValidationError) when calling the AssumeRole operation: The requested DurationSeconds exceeds the MaxSessionDuration set for this role.

The first line proves the run is wearing a temporary identity with a traceable session name. The second is the error everybody hits the first time they ask for a long session. A role ships with MaxSessionDuration of 3600 seconds, and iam_assume_role_duration cannot exceed it. The tempting fix is aws iam update-role --max-session-duration 43200. Resist that on a state role. Twelve hours is twelve hours a stolen session stays usable, and no Terragrunt run needs it. If a run genuinely takes longer than an hour, split it rather than stretching the credential. Worth knowing as well: when one role assumes another (role chaining), AWS caps the session at one hour whatever the role allows.

In CI (continuous integration, the automation that runs your pipeline on a shared machine) you can go further and hold no key at all. GitHub mints a short-lived signed token for the job, STS trades that token for the role, and nothing sensitive is ever stored. The standard behind it is OIDC (OpenID Connect, an agreed way for one system to vouch for an identity to another). The trust policy is where this is won or lost, so read it character by character. Note there are no comments in the file below, because IAM rejects policy JSON containing them.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": "repo:acme/live:ref:refs/heads/main"
      }
    }
  }]
}

Two claims do the work there, a claim being a field GitHub signs into the token. aud (audience) says who the token is meant for, and every GitHub token can carry sts.amazonaws.com, so on its own it says nothing about where the token came from. sub (subject) is the load-bearing line: repo:acme/live:ref:refs/heads/main means this repository, this branch, nothing else. Pin an environment instead, with repo:acme/live:environment:prod, and GitHub's environment approvals sit in front of your production role. Reach for StringLike with a wildcard such as repo:acme/* and you have handed the role to every repository in the org, including the one somebody forks tomorrow.

permissions:
  id-token: write   # lets the runner mint the OIDC token; without it the exchange fails
  contents: read
 
steps:
  - uses: aws-actions/configure-aws-credentials@v4
    with:
      role-to-assume: arn:aws:iam::111122223333:role/terragrunt-ci
      role-session-name: tg-prod-${{ github.run_id }}
      aws-region: us-east-1
      # no aws-access-key-id, no aws-secret-access-key, nothing in the secrets store

Secrets That Never Reach the Repo

SOPS (Secrets OPerationS, an editor for encrypted files) works like a bank statement with the amounts blacked out. The keys stay readable, the values turn into ciphertext, and a diff still shows you which secret changed without showing you the secret. A .sops.yaml file at the repo root maps a path to an encryption key, so a file under live/prod gets wrapped with the prod account's key, and a staging engineer holding only the staging key gets a decryption failure rather than a password. The wrapped data key travels inside the file, which is why the encrypted version is safe to commit and safe to rotate in place.

# The file's path decides which KMS key wraps it. No flags to forget.
creation_rules:
  - path_regex: live/prod/.*\.enc\.yaml$
    kms: arn:aws:kms:us-east-1:111122223333:key/3f1c9e0a-7b52-4a1d-9d0e-51bb0a5f4c77
  - path_regex: live/staging/.*\.enc\.yaml$
    kms: arn:aws:kms:us-east-1:444455556666:key/8ad0c1f2-6e44-4b7a-bb31-2c9e0d5a1f88

cd live/prod/us-east-1/app/rds
 
sops encrypt secrets.yaml > secrets.enc.yaml   # sops 3.9+; older builds: sops --encrypt
 
shred -u secrets.yaml   # best effort only: on a journalling filesystem or an SSD the
                        # old blocks can survive. Rotate the secret if it ever mattered.
 
head -7 secrets.enc.yaml
 
# to change a value later (decrypts into your editor, re-encrypts on save):
#   sops edit secrets.enc.yaml

db_password: ENC[AES256_GCM,data:qk9YR2xQZ1E=,iv:Qk9YR2xQZ1FzVGh0TmZLbEQ5cUx4Rzc2NGpIOGtNMWM=,tag:NXRSM3dRMXpYOWNWN2JOMg==,type:str]
api_token: ENC[AES256_GCM,data:M2p4TjFmSzQ=,iv:VGh0TmZLbEQ5cUx4Rzc2NGpIOGtNMWNRazlZUjJ4UVo=,tag:OWNWN2JOMk5YUlMzd1Exeg==,type:str]
sops:
    kms:
        - arn: arn:aws:kms:us-east-1:111122223333:key/3f1c9e0a-7b52-4a1d-9d0e-51bb0a5f4c77
          created_at: "2026-07-21T09:02:41Z"
          enc: AQICAHhwm5U3nT0h6r1sT4uY7xA9bC2dE5fG8YQk9YR2xQZ1FzVGh0TmZLbEQ5cUx4Rw==

locals {
  # Terragrunt decrypts at render time, in memory, using the run's own KMS rights.
  secrets = yamldecode(sops_decrypt_file("${get_terragrunt_dir()}/secrets.enc.yaml"))
}
 
inputs = {
  db_password = local.secrets.db_password
}

sops_decrypt_file is built into Terragrunt, so there is no wrapper script and no environment variable sitting around with a password in it. The decryption uses whatever credentials the run already assumed, which means the same OIDC session that reads state is the only thing allowed to read secrets, and revoking one revokes both. Add the plaintext filenames to .gitignore (secrets.yaml, anything ending .dec.yaml) and run a secret scanner in CI, so the one time somebody commits the unencrypted copy, the pipeline catches it rather than the internet.

Make the Object Itself Ciphertext

Everything so far protects the bucket. None of it protects a copy of state that leaves the bucket, and copies do leave: a support bundle, a backup job, an engineer redirecting state pull into a file on a laptop. OpenTofu closes that gap with client-side state encryption. The tool encrypts the state body before the backend ever sees it, so the object in S3 is ciphertext even to a principal holding s3:GetObject. Stamp it into every unit with the same generate mechanism you already use for providers. HashiCorp Terraform has no equivalent today, which is one of the sharper practical differences between the two.

# OpenTofu only. Written into every unit as encryption.tf before the run.
generate "encryption" {
  path      = "encryption.tf"
  if_exists = "overwrite_terragrunt"
  contents  = <<EOF
terraform {
  encryption {
    key_provider "aws_kms" "state" {
      kms_key_id = "${local.state_key_arn}"
      region     = "us-east-1"
      key_spec   = "AES_256"
    }
 
    method "aes_gcm" "state" {
      keys = key_provider.aws_kms.state
    }
 
    state {
      method   = method.aes_gcm.state
      enforced = true   # refuse to read or write plaintext state
    }
 
    plan {
      method   = method.aes_gcm.state
      enforced = true   # saved plan files carry the same secrets
    }
  }
}
EOF
}

Rolling this onto existing state takes one intermediate step, or the next run fails trying to read the plaintext file it wrote yesterday. Declare method "unencrypted" "migrate" {} and point a fallback block inside state at it, apply once so OpenTofu reads plaintext and writes ciphertext, then delete the fallback and set enforced = true. Verify by fetching the raw object with aws s3 cp rather than through OpenTofu. The body should be a short JSON envelope holding encrypted_data, encryption_version and meta, with no readable password anywhere in it.

Count the People Who Can Ask for Plaintext

Two gates stand between an engineer and your database passwords: the bucket policy plus their IAM policy on one side, and the KMS key policy on the other. A KMS key is one of the few AWS resources whose own policy is always consulted, so an IAM policy alone cannot open it unless the key policy hands that decision to IAM. This is what makes a customer managed key a genuine second lock, and it is why a mistake in the bucket policy alone is survivable. Print the key policy and read who is on it.

KEY_ARN=arn:aws:kms:us-east-1:111122223333:key/3f1c9e0a-7b52-4a1d-9d0e-51bb0a5f4c77
 
aws kms get-key-policy --key-id "$KEY_ARN" --policy-name default \
  --query Policy --output text \
  | jq -c '.Statement[] | {Sid, Principal, Action}'

{"Sid":"Enable IAM User Permissions","Principal":{"AWS":"arn:aws:iam::111122223333:root"},"Action":"kms:*"}
{"Sid":"AllowTerragruntState","Principal":{"AWS":"arn:aws:iam::111122223333:role/terragrunt-state"},"Action":["kms:Decrypt","kms:GenerateDataKey"]}

That first statement is the default AWS attaches to every new key, and it hands the decision straight back to IAM. Anyone in the account whose IAM policy allows kms:Decrypt now walks through the gate, which is usually every administrator and a surprising number of read-only roles. Narrow it to the principals that genuinely need the key and the second lock becomes real. Do that with care, because a key policy naming nobody who holds kms:PutKeyPolicy locks you out permanently and only AWS Support can undo it. Print the list every quarter. If it is too long to read out loud, it is too long.

Try this

Run terragrunt run -- state pull | jq ' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: sOPS protects the repo, not the state file. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related