State files are secrets

What lands in state, and who can read it.

A hotel front desk keeps a ledger: which guest is in which room, when they checked in, what they are paying. Useful, boring, necessary. Now tape a copy of every room key beside every line. Same ledger, completely different object. The second one is a burglary kit that happens to be well organised.

That second ledger is your Terraform state file. Terraform is the tool that builds cloud infrastructure from text files you write, and to do that job it keeps a record of everything it manages. The record lists your resources. It also stores their passwords, their private keys, and their API tokens (an API, or application programming interface, is how one program asks another to do something, and a token is the long random string that acts as a program's password) as readable text, in a file, on a disk.

Why Terraform Has to Remember

Your configuration says "one database". Reality has a database that AWS actually created, carrying an internal identifier nobody typed. Terraform needs a link between the sentence you wrote and the thing that exists, and the state file is that link. By default it is terraform.tfstate, a JSON document (JavaScript Object Notation, a plain-text format for structured data that you can open in any editor) sitting in your working directory.

Every terraform plan lines up three things: what your config asks for, what state believes exists, and what the cloud reports right now. The gaps become the plan. Take state away and Terraform can no longer tell "create this" apart from "this already exists", so it cheerfully builds everything a second time. That design is what makes Terraform fast and predictable. It also produces one machine-readable inventory of your entire environment, credentials included. Attackers know this, which is why the first move on a compromised build agent costs almost nothing.

# the parentheses matter: without them, -type f applies only to *.tfvars
find / \( -name '*.tfstate*' -o -name '*.tfvars' \) -type f 2>/dev/null

/home/build/.jenkins/workspace/infra-prod/terraform.tfstate
/home/build/.jenkins/workspace/infra-prod/terraform.tfstate.backup
/home/build/.jenkins/workspace/infra-prod/.terraform/terraform.tfstate
/home/build/.jenkins/workspace/infra-prod/prod.tfvars
/var/lib/gitlab-runner/builds/xk3Jd/0/platform/infra/terraform.tfstate

Every Secret Lands in State

Watch it happen on a machine you own. This config generates a password and marks the output sensitive, which is the flag most people reach for when they want a secret hidden.

terraform {
  required_providers {
    random = {
      source  = "hashicorp/random"
      version = "~> 3.7"
    }
  }
}
 
resource "random_password" "db" {
  length  = 20
  special = true
}
 
output "db_password" {
  value     = random_password.db.result
  sensitive = true
}

terraform init -input=false
terraform apply -auto-approve

Initializing the backend...
Initializing provider plugins...
- Finding hashicorp/random versions matching "~> 3.7"...
- Installing hashicorp/random v3.7.2...
- Installed hashicorp/random v3.7.2 (signed by HashiCorp)
 
Terraform has created a lock file .terraform.lock.hcl to record the provider
selections it made above.
 
Terraform has been successfully initialized!
 
random_password.db: Creating...
random_password.db: Creation complete after 0s [id=none]
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.
 
Outputs:
 
db_password = <sensitive>

The command-line tool did its job. It printed <sensitive>. Now open the file it wrote in that same second, using jq (a small program for pulling fields out of JSON on the command line).

jq '.outputs' terraform.tfstate

{
  "db_password": {
    "value": "J9!vQx2#pT7wLm4z%Rd0",
    "type": "string",
    "sensitive": true
  }
}

Read that closely. State records "sensitive": true sitting directly beside the cleartext value. The flag is a note to the printer, not a lock on the data. It changes what the CLI (command-line interface, the thing you type commands into) chooses to display and nothing else: no encryption, no hashing, no redaction, no omission. And the CLI hands the value straight over the moment you ask for it plainly.

terraform output -raw db_password

J9!vQx2#pT7wLm4z%Rd0

None of this is special to random_password. Whenever Terraform creates or refreshes anything, the provider (the plugin that speaks the language of AWS, GitHub, Postgres, and so on) hands back the resource's full attribute set, and Terraform writes that set into state exactly as it arrived. Database master passwords, private keys from the tls provider, IAM (Identity and Access Management, the AWS system that decides who may do what) access key secrets: all verbatim. terraform state pull prints the same plaintext from any backend, local or remote.

So the first genuinely useful thing you can do today is take inventory. Point this at a real workspace and see what you have been storing. The select(.value != null) line matters, because a provider schema can carry an attribute that is present but empty, and an empty field is not a stored secret.

terraform state pull | jq -r '
  .resources[] as $r
  | $r.instances[].attributes
  | to_entries[]
  | select(.value != null)
  | select(.key | test("password|secret|token|private_key"))
  | "\($r.type).\($r.name): \(.key)"'

aws_db_instance.main: password
aws_iam_access_key.ci: secret
aws_iam_access_key.ci: ses_smtp_password_v4
tls_private_key.deploy: private_key_openssh
tls_private_key.deploy: private_key_pem
tls_private_key.deploy: private_key_pem_pkcs8

Notice that one tls_private_key block produced three hits. The provider stores the same key in three encodings, and every one of them is a usable private key. That pattern is a starting point, not an answer. It misses random_password, whose secret attribute is called result. It misses anything a provider decided to name creatively. It also drops the module path, so two resources with the same name in different modules look identical in the list. Extend it for the resource types you actually run. And sit with this for a moment: a state file holding zero credentials is still worth stealing. Private address ranges, security group rules, role ARNs (Amazon Resource Names, the unique identifiers AWS gives every object), the exact shape of your network, all sorted and labelled.

Every Copy Counts, Including the Plan File

Protecting "the state file" sounds like one task. It is several, because Terraform keeps leaving copies behind it like a photocopier nobody clears. The local backend writes terraform.tfstate.backup next to the live file on every write, holding the previous version with the previous secrets. .terraform/terraform.tfstate holds your backend settings, which for some backends includes access credentials. And a saved plan file, the artifact your pipeline loves to hang on to for review, is a zip archive that embeds a full copy of the prior state plus every value about to be written.

terraform plan -out=tfplan > /dev/null
terraform show -json tfplan | jq -r '.resource_changes[]
  | select(.type == "aws_db_instance")
  | .change.after.password'

Correct-Horse-Battery-7

One command, no cloud permissions, password on screen. The JSON plan does mark that field in a separate after_sensitive section, which changes nothing about the value sitting in after. If your CI system (continuous integration, the machinery that runs builds automatically) uploads tfplan so a human can approve it before apply, that artifact needs the same care as the state bucket: short retention, restricted download, never a public pipeline.

One copy deserves its own paragraph, because people get it wrong in a way that feels right. If terraform.tfstate was ever pushed to git, deleting it in a later commit fixes nothing. Every clone, every fork, every CI cache still carries it in history, and so does every mirror you forgot about. Check before you assume you are clean.

git log --all --diff-filter=A --name-only --pretty=format: | sort -u | grep -Ei 'tfstate|tfplan'

infra/prod/terraform.tfstate
infra/prod/terraform.tfstate.backup

Order matters when that command comes back with hits. Rotate every credential in the file first, because the leak already happened and rewriting history never reaches the copies people pulled last year. Then rewrite with git filter-repo (the current tool for surgically removing a file from every commit), force-push, and ask everyone to re-clone. Then add *.tfstate* and *.tfvars to .gitignore so the next engineer cannot repeat it by accident.

Put State Behind a Backend

Local state fails three ways at once. It sits unencrypted on one laptop. Teammates cannot share it. And nothing stops two people applying at the same moment and shredding the file between them. The fix is to move it out of your desk drawer and into a shared safe with a sign-out sheet on the door. Terraform calls that safe a backend, which is its word for "where state lives". On AWS that means an S3 bucket (Simple Storage Service, Amazon's file storage).

terraform {
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "network/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true
    kms_key_id   = "arn:aws:kms:eu-west-1:111122223333:key/9f3d1c8a-4b52-4a1e-9c77-1f0a2b3c4d5e"
    use_lockfile = true  # native S3 locking, no DynamoDB table needed
  }
}

use_lockfile = true turns on native S3 locking, and that is the sign-out sheet. Before writing, Terraform creates a small <key>.tflock object next to your state and refuses to continue if one is already sitting there. The object names the user, the host, the operation, and the time. Native locking arrived in Terraform 1.10 and went generally available in 1.11, which deprecated the older DynamoDB lock table (DynamoDB is Amazon's key-value database, which earlier Terraform used purely to hold that one lock row), so on current Terraform you no longer need it. Azure Blob Storage, Google Cloud Storage, and HCP Terraform (HashiCorp's hosted service, formerly Terraform Cloud) give you the same three properties: encryption at rest, access control, locking.

terraform init -migrate-state

Initializing the backend...
 
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.
 
  Enter a value: yes
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v6.0.0
 
Terraform has been successfully initialized!

Migration copies. It does not shred. After that prompt, terraform.tfstate and terraform.tfstate.backup are still sitting in your working directory in cleartext, so delete them, and remember the same values are in your shell history and quite possibly your laptop backup. The backend block is half the job in any case. The bucket itself needs hardening.

# the first two print nothing when they succeed
aws s3api put-public-access-block --bucket acme-tfstate-prod \
  --public-access-block-configuration \
  BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
 
aws s3api put-bucket-versioning --bucket acme-tfstate-prod \
  --versioning-configuration Status=Enabled
 
aws s3api get-bucket-encryption --bucket acme-tfstate-prod

{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "arn:aws:kms:eu-west-1:111122223333:key/9f3d1c8a-4b52-4a1e-9c77-1f0a2b3c4d5e"
                },
                "BucketKeyEnabled": true
            }
        ]
    }
}

Versioning is your recovery path when someone corrupts or deletes state, and you should confirm it with aws s3api get-bucket-versioning rather than trust a screenshot of the console. The customer-managed KMS key (Key Management Service, the AWS service that stores encryption keys and decides who may use them) means reading state now needs both s3:GetObject and kms:Decrypt. Two doors, two keys. Writing needs s3:PutObject and kms:GenerateDataKey, plus s3:DeleteObject so Terraform can clear the lock object when it finishes.

Encryption at Rest Is Not the Perimeter

Be precise about what that encryption buys. Encryption at rest protects the physical disks, and the person who walks out of the data centre carrying one. It does nothing at all against a principal that AWS considers authorized. Picture a safe that swings open for anyone holding a badge the building already recognises. Anyone who can run terraform plan against that workspace holds s3:GetObject and kms:Decrypt by definition, which means they can run terraform state pull and read every password in the file. Plan permission and secret-read permission are the same permission. No setting pulls them apart.

That makes the bucket policy your real secret perimeter. Scope it to the CI role and a small break-glass group, not to "the platform team" because that was the easy group to type. Split state per environment and per component so one leaked file exposes one blast radius instead of everything: network/, data/, apps/, each with its own key path, its own KMS grant, its own reader list. Then look hard at terraform_remote_state, the data source that reads another workspace's outputs. It looks narrow, since you reference two values. It is not narrow at all. To read those two outputs the caller must be able to read the whole state file, secrets and all, so a dev workspace consuming a production output is a dev role holding production database credentials.

HashiCorp Terraform has no client-side state encryption. OpenTofu, the community fork, added it in version 1.7: state is encrypted on your machine before it leaves, and the backend only ever sees ciphertext. On Terraform, your backend and IAM controls are the whole story.

Watching Who Reads It

Reads are the event you care about, and S3 does not record them by default. The bucket keeps a visitor book for management actions (who changed the policy, who switched on versioning) but no camera on the door. Object-level access is what CloudTrail (the AWS service that logs API calls) calls a data event, and data events stay off until you turn them on per trail.

aws cloudtrail put-event-selectors --trail-name org-trail \
  --advanced-event-selectors '[
    {"Name":"Management events",
     "FieldSelectors":[{"Field":"eventCategory","Equals":["Management"]}]},
    {"Name":"State bucket object access",
     "FieldSelectors":[
       {"Field":"eventCategory","Equals":["Data"]},
       {"Field":"resources.type","Equals":["AWS::S3::Object"]},
       {"Field":"resources.ARN","StartsWith":["arn:aws:s3:::acme-tfstate-prod/"]}]}]'

{
    "TrailARN": "arn:aws:cloudtrail:eu-west-1:111122223333:trail/org-trail",
    "AdvancedEventSelectors": [
        {
            "Name": "Management events",
            "FieldSelectors": [
                {"Field": "eventCategory", "Equals": ["Management"]}
            ]
        },
        {
            "Name": "State bucket object access",
            "FieldSelectors": [
                {"Field": "eventCategory", "Equals": ["Data"]},
                {"Field": "resources.type", "Equals": ["AWS::S3::Object"]},
                {"Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::acme-tfstate-prod/"]}
            ]
        }
    ]
}

Notice the first selector, and do not drop it. put-event-selectors replaces the whole set rather than adding to it, so sending advanced selectors without a management-events entry silently switches management logging off on that trail. Once data events flow, the detection nearly writes itself: alert on any GetObject against that bucket prefix by a principal that is not the CI role. Reading an old copy is the same call with a version ID attached, so it lands in the same place. ListObjectVersions works on the bucket rather than an object, which means it arrives as a management event instead, and it deserves an alert of its own. A terraform state pull at 03:00 from a developer's personal session is exactly the signal you want, and without data events it leaves no trace whatsoever.

Keeping Secrets Out of State Entirely

The strongest control is to never write the value down. Read a phone number aloud to someone who copies it into their own book, and your notepad stays blank. Terraform 1.10 added ephemeral resources, values that exist only for the length of a single run and are never persisted to state or to a plan file. Terraform 1.11 added write-only arguments, resource arguments carrying a _wo suffix that accept a value at apply time and throw it away afterwards. Inside a managed resource, a write-only argument is the only place an ephemeral value is allowed to go, so the two features are built to work as a pair.

ephemeral "random_password" "db" {
  length = 20
}
 
resource "aws_db_instance" "main" {
  identifier          = "app-prod"
  engine              = "postgres"
  instance_class      = "db.t4g.micro"
  allocated_storage   = 20
  username            = "app"
  password_wo         = ephemeral.random_password.db.result
  password_wo_version = 1   # bump this number to rotate
}

terraform state pull | jq '.resources[]
  | select(.type == "aws_db_instance")
  | .instances[0].attributes
  | {password, password_wo, password_wo_version}'

{
  "password": null,
  "password_wo": null,
  "password_wo_version": 1
}

password_wo is null because it was never stored. password_wo_version is stored, and that plain integer is the clever part: Terraform sends the write-only value to the provider only when the version number changes. It is the reason generating a fresh random password on every run does not produce an endless diff. Bump the number and the new value goes out. Leave it alone and Terraform ignores the argument completely.

The trade-offs are real, so design around them. Provider support for _wo arguments is still spreading unevenly, so check the specific resource's documentation before you build on top of it. An ephemeral value cannot be referenced later, which means your application has to fetch the password from Secrets Manager or Vault (secret stores that hand out credentials at runtime) instead of reading a Terraform output, and that is an architecture change rather than a flag flip. In production the usual shape is to feed the write-only argument from an ephemeral read of the secret store, ephemeral "aws_secretsmanager_secret_version" rather than a generated password, so one system owns rotation and Terraform only passes the value through on its way to the database.

Run the inventory jq against every workspace you own, then run the git history check against every infrastructure repository. Both take about ten minutes, and between them they tell you whether you have a credential rotation job waiting this week. The next lesson takes the backend from configured to hardened: bucket policies and KMS grants that actually restrict, versioning treated as a recovery drill you have practised, and what to do when a killed pipeline leaves a lock behind.

Try this

Run find / \( -name '*.tfstate*' -o -name '*.tfvars' \) -type f 2>/dev/null on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: versioning keeps your old secrets alive too. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related