Secrets in IaC: SOPS & Vault

Keep credentials out of state and code.

Terraform keeps a notebook. Every time it builds something it writes down what it built and which settings it used, so the next run knows what already exists and what changed. That notebook is the state file. It is plain JSON (JavaScript Object Notation, a text format any person or program can read) sitting on a disk or in a storage bucket, and Terraform does not encrypt it. So when one of those settings was a database password, the password sits in the notebook, spelled out in full. This is the part of IaC (infrastructure as code, the practice of describing your servers and databases in files you commit to Git) secret handling that catches careful teams. You can keep every credential out of your .tf files and still hand the whole set to anyone with read access to state.

Where One Password Actually Goes

Follow a single password through one terraform apply and count where it comes to rest. You supply it, from a .tfvars file, an environment variable, or a lookup against a secrets manager. Terraform works out a plan, and if the pipeline saved that plan with -out=tfplan, your value is inside that file in the clear. It goes out over the wire in the provider's API call (application programming interface, the machine-to-machine request that actually creates the database). It comes back and gets written into state as an attribute of the resource. On a local backend the previous state is kept as terraform.tfstate.backup, and if Terraform cannot push the new state to the backend at all it dumps errored.tfstate into your working directory. Then the remote copy lands in the backend bucket, where who can read it is decided by IAM (identity and access management, the cloud's permission system), not by anything Terraform controls.

Two of those deserve more attention than they usually get. Pipelines routinely run terraform plan -out=tfplan, upload tfplan as a build artifact, then download it again in the apply job. That artifact holds every value the plan touched in cleartext, passwords included, and build artifacts are often readable by the whole organization and kept for weeks. Same story with TF_LOG=DEBUG or TF_LOG=TRACE, which make providers write request and response bodies into the job log, and job logs are searchable forever. Give plan artifacts the shortest retention your pipeline allows, restrict who can download them, and keep TRACE logging off any run that touches real credentials.

Read The Secret Out Of Your Own State

Prove this to yourself on a stack you own. terraform state pull fetches the current state from whatever backend you configured and prints it on stdout (standard output, meaning it lands in your terminal). jq (a small command line tool for picking JSON apart) trims it down to the part worth looking at. Notice what you did not need. No administrator rights on the cloud account, no exploit, no access to the repository. Read access to the state file was enough, which is exactly the point.

$ terraform state pull \
  | jq '.resources[]
        | select(.mode=="managed" and .type=="aws_db_instance")
        | .instances[].attributes
        | {id, username, password}'

{
  "id": "prod-orders-db",
  "username": "app",
  "password": "T7h!q2Vd9xLm0Za"
}

Marking a variable or an output sensitive = true changes exactly one thing. Terraform stops printing the value in plan and apply output. It encrypts nothing, and it keeps nothing out of state. The state format even carries a sensitive_attributes list on each resource instance, which is a note about what to blank out on screen, not a lock. It is a sticker on a filing cabinet that says do not read this aloud. The drawer is still unlocked. (OpenTofu, the community fork, added client-side state encryption in 1.7. Terraform has no equivalent, so with Terraform you are protecting the storage, never the file itself.) The working rule is blunt: whoever can read your state can read every secret Terraform has ever touched. Put that in your threat model, because it turns "give the contractor read access to the state bucket so they can debug" into "give the contractor the production database password".

Fetch It At Apply Time, Not At Commit Time

Rather than copying the key into the blueprint, write down which locksmith holds it. That is what a secrets manager lookup does. Terraform's Vault provider and the cloud secrets-manager data sources (aws_secretsmanager_secret_version, google_secret_manager_secret_version) fetch the credential during the run, so the repository holds a pointer that says "the password at secret/prod/db in Vault" and never the value behind it. Pair that with OIDC (OpenID Connect, a standard way for one system to prove who it is to another) authentication for the pipeline, where the CI (continuous integration, the automated build and deploy system) job trades a short-lived signed token for temporary cloud credentials, and no long-lived cloud key is stored anywhere either.

data "vault_kv_secret_v2" "db" {
  mount = "secret"      # where the KV (key-value) v2 engine is mounted in Vault
  name  = "prod/db"     # the secret's path inside that mount
}
 
resource "aws_db_instance" "main" {
  identifier = "prod-orders-db"
  engine     = "postgres"
  username   = "app"
  password   = data.vault_kv_secret_v2.db.data["password"]  # read at apply, never in Git
}

Now the trap that swallows most teams who get this far. A data source's result is stored in state too. Fetching from Vault took the secret out of Git and left it in the state file exactly where it was before. Same jq, different resource mode.

$ terraform state pull \
  | jq '.resources[]
        | select(.mode=="data" and .type=="vault_kv_secret_v2")
        | .instances[].attributes.data'

{
  "password": "T7h!q2Vd9xLm0Za",
  "username": "app"
}

Vault can shrink the damage instead of only moving it around. With the database secrets engine, Vault works like a hotel front desk. It cuts a fresh database user on demand, hands back a credential with a stated lease, and deletes that user when the lease expires. A password sitting in state that stopped working an hour after the apply is a very different finding from one that has been valid since 2021.

$ vault read database/creds/app-readonly

Key                Value
---                -----
lease_id           database/creds/app-readonly/8KdZq2Yf1nP0vXcT
lease_duration     1h
lease_renewable    true
password           A1a-hT9wq2ZxL0mR4sVd
username           v-token-app-read-9Qq1oXpN-1753088042

Ephemeral Values And Write-Only Arguments

Everything so far guards the copy after it exists. Ephemeral values stop the copy from being made. Terraform 1.10 added ephemeral resources along with ephemeral input variables and output values: things that live only for the length of a single run and are never written to state or into a saved plan. Terraform 1.11 added write-only arguments on resources, where the provider sends the value to the API and then drops it instead of recording what it sent. Picture a courier who reads an address off a slip, delivers the parcel, and shreds the slip on the doorstep. Both ends have to support it, so you want Terraform 1.11 or newer plus a current provider (write-only arguments landed in the AWS provider at 5.87.0, ephemeral Vault lookups in the Vault provider at 4.5.0). Where you have that, this is the only pattern that keeps a value out of state rather than guarding it after the fact.

ephemeral "vault_kv_secret_v2" "db" {   # terraform >= 1.10, vault provider >= 4.5
  mount = "secret"
  name  = "prod/db"
}
 
resource "aws_db_instance" "main" {
  identifier          = "prod-orders-db"
  engine              = "postgres"
  username            = "app"
  password_wo         = ephemeral.vault_kv_secret_v2.db.data["password"]
  password_wo_version = 1   # required alongside password_wo; bump it to send a rotated value

Verify it rather than trusting the release notes. Pull the state and look straight at the attribute you care about.

$ terraform state pull \
  | jq '.resources[]
        | select(.type=="aws_db_instance")
        | .instances[].attributes
        | {password, password_wo, password_wo_version}'

{
  "password": null,
  "password_wo": null,
  "password_wo_version": 1
}

The rough edges are real, and you should know them before you promise this to anyone. password and password_wo cannot both be set on the same resource; the provider rejects the configuration outright. An ephemeral value cannot be fed into an ordinary argument or into a root module output, and Terraform stops the run if you try, which is the feature working rather than breaking. And because the value is never recorded, Terraform has nothing to compare against and cannot tell that it changed, so rotating the password means incrementing password_wo_version to say "send it again".

SOPS For The Files You Have To Commit

Some files genuinely belong in the repository: a Helm values file, an application config with one API token in it, per-environment settings. SOPS (Secrets OPerationS) treats a file like that as a parcel with a readable label and a padlocked lid. It encrypts the values and leaves the keys and the structure in plaintext, so a pull request diff still shows that db_password changed without showing what it changed to, and merges stay reviewable. The data key that does the encrypting is itself wrapped by AWS KMS (Key Management Service, Amazon's managed key store), Google Cloud KMS, Azure Key Vault, Vault's transit engine, PGP (Pretty Good Privacy, the old standard for encrypted mail and files), or age, a small file encryption tool whose public keys are a single line of text.

# repo root: which key protects which path, so nobody has to remember
creation_rules:
  - path_regex: secrets/prod/.*\.enc\.yaml$
    encrypted_regex: '^(password|token|.*_key)$'   # encrypt these values only
    kms: 'arn:aws:kms:eu-west-1:111122223333:key/9f1c8f0e-6f2a-4a6f-8f5e-6d3f0b2a11cd'
 
  - path_regex: secrets/dev/.*\.enc\.yaml$
    encrypted_regex: '^(password|token|.*_key)$'
    age: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

$ mkdir -p ~/.config/sops/age
$ age-keygen -o ~/.config/sops/age/keys.txt
$ sops encrypt --in-place secrets/dev/app.enc.yaml
$ head -6 secrets/dev/app.enc.yaml

Public key: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p
db_host: postgres.internal
db_user: app
password: ENC[AES256_GCM,data:hR5xQ2p1ZwT8KgN4bQz3,iv:6Nn1kZ0yQ8t3RmC7vJdP2sXbA9uYfL4hEwQ0iTzNcRk=,tag:9dQ7pXsB1uKmVzOaLtG3Fw==,type:str]
sops:
    age:
        - recipient: age1ql3z7hjy54pw3hyww5ayyfg7zqgvc7w3j2elw8zmrj2kg5sfn9aqmcac8p

Each value is encrypted with AES-256-GCM (Advanced Encryption Standard at 256 bits in Galois/Counter Mode, a cipher that both hides the data and detects tampering) under one data key, and that data key is wrapped separately for every recipient listed in the sops block. That is how five engineers and one CI role each get their own way in without sharing a single key, and how you cut someone off: re-encrypt to a shorter list. The same block carries a MAC (message authentication code, a tamper-evident seal computed over the whole document). So a person who hand-edits a plaintext field, flipping tls_required from true to false, breaks decryption loudly instead of slipping it past review.

Day to day, keep the plaintext off disk. sops decrypt writes to stdout, sops edit opens your editor on a temporary decrypted copy and re-encrypts when you save, and sops exec-env hands the values to one child process as environment variables and to nothing else. Name a key TF_VAR_db_password and Terraform picks it up as the input variable db_password with no extra wiring.

$ sops filestatus secrets/dev/app.enc.yaml
$ sops exec-env secrets/dev/tfvars.enc.yaml 'terraform apply -auto-approve'

{"encrypted":true}
aws_db_instance.main: Creating...
aws_db_instance.main: Creation complete after 4m21s [id=prod-orders-db]
 
Apply complete! Resources: 1 added, 0 changed, 0 destroyed.

Two caveats to carry around. Environment variables are readable by anything running as the same user, through /proc/<pid>/environ, so exec-env is a decent handoff and a poor hiding place. And the community Terraform SOPS provider, data "sops_file", decrypts inside Terraform, which means the decrypted values get written into state and you are back at the top of this lesson.

Treat State Like A Password Store

Because state holds secrets however careful you were upstream, the backend gets the treatment you would give a safe. Encrypt at rest with a customer-managed key, so the key policy is yours to write. Keep the object private. Turn on versioning, so a truncated or corrupted state can be rolled back. And control who can read the object, not only who can write it.

terraform {
  backend "s3" {
    bucket       = "acme-tfstate-prod"
    key          = "platform/rds/terraform.tfstate"
    region       = "eu-west-1"
    encrypt      = true    # server-side encryption on every PUT
    kms_key_id   = "arn:aws:kms:eu-west-1:111122223333:key/9f1c8f0e-6f2a-4a6f-8f5e-6d3f0b2a11cd"
    use_lockfile = true    # native S3 locking, Terraform 1.10+ (replaces dynamodb_table)
  }
}

$ aws s3api get-bucket-encryption --bucket acme-tfstate-prod
$ aws s3api get-public-access-block --bucket acme-tfstate-prod \
    --query 'PublicAccessBlockConfiguration.BlockPublicAcls'

{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "arn:aws:kms:eu-west-1:111122223333:key/9f1c8f0e-6f2a-4a6f-8f5e-6d3f0b2a11cd"
                },
                "BucketKeyEnabled": true
            }
        ]
    }
}
true

Encryption at rest is the easy half, and it defends against the least likely attack: someone walking off with the disk. The half that matters is access. The KMS key policy and the bucket policy should name the pipeline role and one break-glass group (a rarely used emergency account), and nobody else. Then switch on CloudTrail (AWS's audit log of API calls) data events for that bucket, so a GetObject against terraform.tfstate by a human identity raises an alert instead of scrolling past. That single detection covers the most common real route to your production database password, which is rarely a clever exploit and usually an engineer with legitimate read access and a debugging problem. Keep the local copies out of Git as well.

*.tfstate
*.tfstate.*
errored.tfstate
.terraform/
*.tfvars
!example.tfvars
tfplan
crash.log
crash.*.log

Catch It Before The Commit, And After

Two gates, and you want both: a bouncer on the door, and a search of the building. A pre-commit hook stops a plaintext secret from reaching a commit in the first place. A history scan tells you what walked in three years ago and is still sitting there. Share the hook through the repository rather than through everyone's private .git directory, using git config core.hooksPath .githooks.

#!/usr/bin/env bash
# enable for everyone:  git config core.hooksPath .githooks
set -euo pipefail
 
staged=$(git diff --cached --name-only --diff-filter=ACM \
         | grep -E '^secrets/.*\.enc\.ya?ml$' || true)
 
for f in $staged; do
  if [ "$(sops filestatus "$f" | jq -r .encrypted)" != "true" ]; then
    echo "BLOCKED: $f is staged in plaintext. Run: sops encrypt -i $f" >&2
    exit 1
  fi
done
 
# scan the staged diff for anything that looks like a credential
# (gitleaks before 8.19 spelled this: gitleaks protect --staged)
gitleaks git --pre-commit --staged --no-banner --redact

A hook only protects people who ran that config, and anyone can skip it with git commit --no-verify, so run the same scan in CI on every pull request where it cannot be skipped. For history, gitleaks walks the commit objects instead of the working tree, which is the whole reason to run it.

$ gitleaks git --no-banner --redact -v .

Finding:     db_password = "REDACTED"
Secret:      REDACTED
RuleID:      generic-api-key
Entropy:     4.312500
File:        env/prod.tfvars
Line:        7
Commit:      4c1f9b2e8a7d6c5b4a39281706f5e4d3c2b1a098
Author:      Priya N
Email:       [email protected]
Date:        2025-01-15T14:22:41Z
Fingerprint: 4c1f9b2e8a7d6c5b4a39281706f5e4d3c2b1a098:env/prod.tfvars:generic-api-key:7
 
11:04AM INF 387 commits scanned.
11:04AM INF scanned ~412 KB (412.00 KB) in 84.2ms
11:04AM WRN leaks found: 1

Read that finding the way a defender should. The file was deleted from the working tree eighteen months ago. The credential was not. Everyone who ever cloned the repository still holds it in their local object database, and so does every fork, every CI cache, every backup, and every mirror. Rewriting history with git filter-repo reaches none of those copies.

Try this

Run vault read database/creds/app-readonly on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: rotate first, clean history last. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related