Securing state, secrets & the supply chain

The IaC attack surface, closed.

A locksmith who fits every door in a building ends up holding two things nobody else has: a ledger of which lock is where, and a drawer of spare keys. Terraform keeps both in one file, and it rewrites that file after every run. The file is the state: Terraform's record of the resources it manages, plus the attribute values the cloud handed back when it built them. Read the state and you have the database password, the secret half of a generated access key, and a map of the whole estate.

Infrastructure code stacks risk in three places. The state and plan files Terraform writes to disk. The secrets that pass through it on their way to the cloud API (Application Programming Interface, the machine-facing door into a service that your tools call instead of clicking around a web console). And the modules and providers you downloaded but did not write. All three need closing. Leave one open and the other two stop mattering, because whoever controls Terraform controls everything Terraform built.

State Is a Credential Store

State is plain JSON (JavaScript Object Notation, a text format of labelled values that people and programs can both read). Terraform writes it so the next run knows what already exists, and it keeps whatever the cloud returned, including the values you fed in. A password on an RDS instance (Relational Database Service, Amazon's managed database) sits in there as ordinary text. So does the secret half of an IAM access key (Identity and Access Management, the AWS service that decides who is allowed to do what), and any private key generated by tls_private_key. Do not take my word for it. Pull the state and read it with jq, a small command-line tool for slicing JSON apart.

# Pull state through the backend rather than hunting for a local copy.
terraform state pull | jq '{version, terraform_version, serial}'
 
terraform state pull | jq '.resources[]
  | select(.type == "aws_db_instance")
  | .instances[0].attributes | {username, password}'

{
  "version": 4,
  "terraform_version": "1.13.3",
  "serial": 41
}
{
  "username": "appuser",
  "password": "pr0d-Rds-9f2c!aa"
}

Two details worth pausing on. Format version 4 is the state layout Terraform has used since 0.12, and serial goes up by one on every write, which is how Terraform notices a stale copy: run terraform state push with a file whose serial is behind what the backend already holds and it refuses unless you force it. Then the second block, where the password came out in the clear. Terraform does record that the value is sensitive, in a flag sitting directly beside the value it stored as readable text. The flag travels with the secret. It does not replace it.

That marking is a curtain, not a safe. Setting sensitive = true keeps a value out of the summary Terraform prints and out of the plan diff a reviewer reads. It changes nothing about what gets written to disk, and it comes off the moment you ask for the value directly.

terraform output                    # the summary listing everyone trusts
terraform output db_password        # name the output and the mask comes off
terraform output -raw db_password   # no quotes, ready to pipe somewhere

db_password = <sensitive>
"pr0d-Rds-9f2c!aa"
pr0d-Rds-9f2c!aa

Only the first line is redacted. Masking belongs to that summary listing, not to the value, so naming the output is by itself enough to print it. No special flag required. Running terraform output -json with no name is the same story from another angle: it returns the real string next to a helpful "sensitive": true label describing the thing it handed over anyway.

Lock the Backend Like It Holds Keys

Leaving state in a file beside your code is the key drawer left open on the workbench. State belongs in a remote backend instead, a shared server-side home for the file, and it needs four properties. Encrypted at rest under a key you control. Versioned, so a bad write is recoverable. Locked, so two applies cannot stomp on each other. Readable by a very short list of identities. The S3 backend (Simple Storage Service, Amazon's object store) covers all four, provided you configure the bucket to match.

terraform {
  backend "s3" {
    bucket = "acme-tf-state"
    key    = "prod/terraform.tfstate"
    region = "us-east-1"
 
    encrypt    = true    # server-side encryption on every write
    kms_key_id = "arn:aws:kms:us-east-1:111122223333:key/8f1c2d3e-4b5a-6c7d-8e9f-0a1b2c3d4e5f"
 
    use_lockfile = true                 # Terraform 1.10+: writes prod/terraform.tfstate.tflock in S3
    # dynamodb_table = "acme-tf-locks"  # the old lock table, deprecated in Terraform 1.11
  }
}

Three lines there earn their keep. encrypt = true tells S3 to encrypt the object as it is written. kms_key_id points at a key you own in KMS (Key Management Service, AWS's managed key store), which puts a second, separately audited gate in front of the file: an identity holding s3:GetObject but no kms:Decrypt on that key gets ciphertext and nothing else. And use_lockfile leans on S3's own conditional writes to drop a .tflock object next to the state, retiring the separate DynamoDB table (Amazon's key-value database) that older setups needed for locking. Now assume nothing about the bucket itself. Check it.

aws s3api get-bucket-versioning   --bucket acme-tf-state
aws s3api get-bucket-encryption   --bucket acme-tf-state
aws s3api get-public-access-block --bucket acme-tf-state

{
    "Status": "Enabled"
}
{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "arn:aws:kms:us-east-1:111122223333:key/8f1c2d3e-4b5a-6c7d-8e9f-0a1b2c3d4e5f"
                },
                "BucketKeyEnabled": true
            }
        ]
    }
}
{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

Encryption at rest defends against someone who walks off with the disk. It does nothing against someone your policy already allows to call GetObject. Read access to the state prefix is read access to every credential inside it, so write the bucket policy as a fence: name the few identities allowed through, deny everybody else by default.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OnlyPipelineAndPlatformAdmins",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::acme-tf-state",
        "arn:aws:s3:::acme-tf-state/prod/*"
      ],
      "Condition": {
        "StringNotLike": {
          "aws:PrincipalArn": [
            "arn:aws:iam::111122223333:role/tf-ci",
            "arn:aws:iam::111122223333:role/platform-admin",
            "arn:aws:iam::111122223333:role/break-glass"
          ]
        }
      }
    }
  ]
}

Two notes on the mechanics. When a role is assumed, aws:PrincipalArn resolves to the role's own ARN (Amazon Resource Name, the unique identifier AWS gives every object), not the temporary session identity, so listing the role covers every session of it. And keep a break-glass identity in that list, then actually test it, because a Deny paired with StringNotLike is perfectly happy to lock out you, your pipeline, and the person who wrote the policy.

The list has a habit of growing quietly. Another team's stack reading your outputs through a terraform_remote_state data source (a block that reads a second state file to borrow its output values) needs the same read permission on your bucket. So does the engineer who debugged a failed apply at 2am and never gave the access back. Review it on the schedule you use for production database access.

Plan Files Leak the Same Secrets

A saved plan looks harmless because it looks unreadable. Open it in an editor and you get binary noise. It is a zip archive, and building it means Terraform has already resolved every variable and read every data source, so the resolved values are sitting inside. Unpack one and you find the planned changes, a copy of your .tf source, the lock file, and two full state files: the prior state and the one before that. The human-readable plan prints password = (sensitive value). The machine-readable view of the same file does not.

terraform plan -out=tfplan >/dev/null
file tfplan
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.type == "aws_db_instance")
  | .change.after.password'

tfplan: Zip archive data, at least v2.0 to extract, compression method=deflate
pr0d-Rds-9f2c!aa

There is the pipeline trap in two lines. terraform show -json keeps the real value in .change.after and records the sensitivity separately in .change.after_sensitive, so what you saw in the diff was a rendering decision rather than a property of the file. The pull request comment shows a tidy (sensitive value), everyone relaxes, and then the job uploads tfplan as a build artifact that anyone with repository read access can download for the next ninety days. Give plan artifacts what you give state: short retention, restricted access, encrypted storage, never attached to a public build, never pasted into a comment.

Stop the Secret From Landing at All

Guarding the copy is the second-best move. Not making a copy is the best one. Hard-coding a password in a .tf file (the plain-text files Terraform reads, written in HashiCorp Configuration Language) is the worst version, because Git remembers the line long after you delete it. Pulling the value at apply time from a secrets manager is better, since the repository stays clean, but the value still lands in state. Terraform 1.10 and 1.11 added the two pieces that close the gap: ephemeral values, which live only for the length of a single run, and write-only arguments, which hand a value to the cloud API and store it nowhere.

# Old pattern: fetched at apply time, which keeps it out of Git.
# It still gets written into state.
# data "aws_secretsmanager_secret_version" "db" {
#   secret_id = "prod/db/password"
# }
 
# Terraform 1.10+: an ephemeral resource is read during the run and never persisted.
ephemeral "aws_secretsmanager_secret_version" "db" {
  secret_id = "prod/db/password"
}
 
resource "aws_db_instance" "main" {
  identifier = "prod-app"
  username   = "appuser"
 
  # Terraform 1.11+: a write-only argument goes to the AWS API and is written to
  # neither state nor the plan file. Bump the version to push a new value.
  password_wo         = ephemeral.aws_secretsmanager_secret_version.db.secret_string
  password_wo_version = 1
}

The version counter is what makes write-only arguments workable. Terraform cannot compare a value it refuses to keep, so you tell it the value changed by bumping the number. One constraint to design around: a write-only argument accepts only an ephemeral source, meaning an ephemeral resource, an ephemeral variable, or an ephemeral output. Hand it an ordinary data source and the plan fails. Apply, then look at what actually landed.

terraform apply -auto-approve >/dev/null
terraform state pull | jq '.resources[]
  | select(.type == "aws_db_instance")
  | .instances[0].attributes
  | {password, password_wo, password_wo_version}'

{
  "password": null,
  "password_wo": null,
  "password_wo_version": 1
}

Nulls where the secret used to be, and only the counter survives. Support is per resource rather than per provider, so look for an argument ending in _wo in the resource documentation before you build a design around it. Where a provider has not caught up, at least keep the value out of the repository: inject it in CI (continuous integration, the automation that builds and deploys your code on every push) as a TF_VAR_db_password environment variable pulled from your secret store, or commit it encrypted with SOPS, a tool that encrypts the values inside a YAML or JSON file while leaving the keys readable so diffs stay reviewable. Both of those still put the value in state, which is why the encrypted, tightly scoped backend above is not optional.

Every Provider Is Code You Run as Admin

A provider is a program, not a library your code imports. Terraform downloads it, starts it as a separate process on your laptop or your CI runner, talks to it over a local connection, and hands it your cloud credentials. That is the trust you give a contractor you buzz into the building with a master key. Look at what your last terraform init actually put on disk.

ls -lh .terraform/providers/registry.terraform.io/hashicorp/aws/5.100.0/linux_amd64/

total 683M
-rwxr-xr-x 1 dev dev 683M Jul 14 09:22 terraform-provider-aws_v5.100.0_x5

That is 683 megabytes of somebody else's compiled code, executed by your runner, holding a role that can create or destroy anything in the account. Modules carry the same authority with far fewer bytes: whatever a module declares gets applied with your permissions, and a terraform_data or null_resource block with a provisioner runs shell commands on the machine doing the apply. So pin everything, and pin to something that cannot move underneath you.

terraform {
  required_version = ">= 1.11"
 
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "5.100.0"        # exact, not "~> 5.0"
    }
  }
}
 
# Module blocks normally live in main.tf. They sit here so the pinning
# rules for providers and modules can be read side by side.
 
module "vpc" {
  source  = "terraform-aws-modules/vpc/aws"
  version = "5.21.0"             # registry modules take an exact version
}
 
module "baseline" {
  # A git tag is a movable label. A commit hash is not.
  source = "git::ssh://[email protected]/acme/tf-modules.git//baseline?ref=9c4f1e2a7b3d5c6f8a0b1d2e3f4a5b6c7d8e9f01"
}

The constraint "~> 5.0" means any 5.x release, so the code that runs tomorrow is not the code you reviewed today. Exact versions fix that for providers and for registry modules. Git sources need one more step. A tag is a label somebody can move: anyone who can push to that repository can re-point v1.4.2 at completely different code, and your next init would fetch it without a murmur. A commit hash cannot be re-pointed.

The Lock File Is the Checksum Promise

A version number says which package to fetch. It says nothing about what should be inside it. The lock file covers that gap, the way a tamper-evident seal on a parcel tells you more than the shipping label does. terraform init writes .terraform.lock.hcl recording the versions it picked and the checksums it saw, and every later run compares what it downloads against those checksums. Commit it. Treat any change to it in a pull request as a real review item rather than noise.

provider "registry.terraform.io/hashicorp/aws" {
  version     = "5.100.0"
  constraints = "5.100.0"
  hashes = [
    "h1:Fnaec9vA8sZ8BXVlN3Xn9Jz3zghSETIKg7ch8oXhxno=",   # extracted package, this platform
    "zh:0e8e5a3bd1f0c6e2a4a5e0b9c8d7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c1d0e9",
    "zh:1a2b3c4d5e6f708192a3b4c5d6e7f8091a2b3c4d5e6f708192a3b4c5d6e7f809",
  ]
}

The two prefixes mean different things. A zh: entry (zip hash) is the checksum of the .zip package exactly as the registry published it, taken from a checksum file the publisher signed. An h1: entry is a checksum Terraform computes over the package after unpacking it locally, so it exists only for platforms Terraform has actually installed on. That last part is where teams get caught. Ask for every platform your people and your runners use, in one go.

terraform providers lock \
  -platform=linux_amd64 \
  -platform=darwin_arm64

- Fetching hashicorp/aws 5.100.0 for linux_amd64...
- Retrieved hashicorp/aws 5.100.0 for linux_amd64 (signed by HashiCorp)
- Fetching hashicorp/aws 5.100.0 for darwin_arm64...
- Retrieved hashicorp/aws 5.100.0 for darwin_arm64 (signed by HashiCorp)
- Obtained hashicorp/aws checksums for linux_amd64; This was not previously recorded in the lock file
- Obtained hashicorp/aws checksums for darwin_arm64; This was already recorded in the lock file
 
Success! Terraform has updated the lock file.
 
Review the changes in .terraform.lock.hcl above and commit them to your
version control system if they represent changes you intended to make.

Here is how the trap springs without that command. terraform init records an h1: hash for the platform it ran on and no other. Lock on a Mac, run CI on linux_amd64, and Terraform quietly fetches the Linux package, records a fresh hash for a platform it has never seen, and carries on. The check you thought you had never ran once. Two moves close it: run terraform providers lock with a -platform flag for every architecture in play and commit the result, then run terraform init -lockfile=readonly in the pipeline so any lock file change fails the build instead of happening in silence.

Now the part all of this exists for. When the package on the other end does not match what you recorded, the run stops before a single API call is made.

terraform init

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/aws versions matching "5.100.0"...
- Installing hashicorp/aws v5.100.0...
╷
│ Error: Failed to install provider
│
│ Error while installing hashicorp/aws v5.100.0: the current package for
│ registry.terraform.io/hashicorp/aws 5.100.0 doesn't match any of the
│ checksums previously recorded in the dependency lock file; for more
│ information: https://developer.hashicorp.com/terraform/language/files/dependency-lock#checksum-verification
╵

That error is the control working, and it deserves a human answer rather than a quick -upgrade. Maybe a colleague bumped the provider and forgot to commit the lock change. Maybe your internal mirror served a package that no longer matches what the registry signed. Either way nothing gets applied until somebody explains the difference, which is what you want a supply-chain check to do.

For anything you cannot afford to have swapped underneath you, hold your own copy. terraform providers mirror ./vendor pulls the provider packages into a directory you control, and a private registry or network mirror lets you review a new version before your whole fleet can reach it. Read third-party modules the way you would read a Dockerfile before running the image: look for provisioners shelling out, calls to network endpoints you do not recognise, and any IAM policy the module attaches that is wider than the module's actual job.

Start with an inventory, because you cannot rotate what you have not listed. Pull each production state file, print every attribute whose name looks like a secret, and treat the output as a backlog.

terraform state pull | jq -r '
  .resources[] | .type as $t | .instances[]
  | .attributes | to_entries[]
  | select(.key | test("password|secret|token|private_key"))
  | "\($t).\(.key)"' | sort -u

aws_db_instance.password
aws_iam_access_key.encrypted_secret
aws_iam_access_key.secret
aws_iam_access_key.ses_smtp_password_v4
tls_private_key.private_key_pem

Every line there is a credential that has existed as readable text in a file, in a bucket version, in a plan artifact and probably in a CI cache. Rotate them. Move each one onto a write-only argument or into a secrets manager as the provider allows, and run the same command next quarter to see whether the list got shorter.

Try this

Run terraform state pull | jq '{version, terraform_version, serial}' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: masking is a display choice, and old copies survive. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related