Hardened remote backends & locking

Encryption, versioning, least-privilege state access, and locks.

A gas station keeps its restroom key chained to a block of wood the size of a shoe. One key exists, one person holds it, and everyone can see who that person is. Terraform state needs both of those properties: somewhere to live that is not one person's laptop, and something that stops two people writing to it at the same moment. The previous lesson covered what sits inside that file: resource IDs, network layout, and every password, token and private key Terraform has handled, in readable text. This lesson is the storage underneath it, on Amazon Web Services (AWS) and elsewhere.

Get state off the laptop

Local state is the spare key left under the mat. `terraform.tfstate` sits in your working directory in plain text, and anything on a laptop leaks eventually: a Time Machine backup, a synced OneDrive folder, a stray `git add .`. Your teammate cannot see that file either, nothing stops the two of you applying at once, and there is no history to roll back to. A *backend* is the setting that tells Terraform to read and write state somewhere else. Moving it is the first control rather than the whole one, because a fresh S3 bucket (Simple Storage Service, Amazon's object store) is a box you own rather than a box you have locked.

terraform {
  required_version = ">= 1.11"
 
  backend "s3" {
    bucket = "acme-tfstate-prod"
    key    = "prod/network/terraform.tfstate"
    region = "eu-west-1"
 
    encrypt    = true
    kms_key_id = "arn:aws:kms:eu-west-1:111122223333:key/9f3d0c1e-4b8a-4d21-9f77-2c6e8a5b1d40"
 
    use_lockfile = true   # S3-native locking: added in 1.10, generally available in 1.11
 
    # Legacy locking. Still works, deprecated since 1.11:
    # dynamodb_table = "acme-tfstate-locks"
  }
}

Read that as an address plus four security settings. `bucket` and `key` name the object holding this stack's state. `encrypt = true` asks S3 to store it with server-side encryption. `kms_key_id` points at a key you control in KMS (Key Management Service, the AWS service that creates and guards encryption keys) rather than the default. `use_lockfile = true` turns on S3-native locking, where the lock is an object living in the same bucket. One quirk trips up everyone once: Terraform reads the `backend` block before it evaluates the rest of your configuration, so the block takes literal values only. No variables, no `${}` interpolation. Per-environment values arrive at init time, which is the last section here.

$ terraform init -migrate-state

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to
  the newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.
 
  Enter a value: yes
 
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.94.1
 
Terraform has been successfully initialized!

Terraform copied the local file up and switched over. Finish the job on the laptop: `terraform.tfstate` and `terraform.tfstate.backup` are still lying there in readable text, so delete them and add `*.tfstate*` to `.gitignore`. Other clouds have the same shape with different nouns. The `azurerm` backend keeps state as a blob and locks it with a *blob lease*, which is a short exclusive claim the storage service enforces on your behalf. HCP Terraform (HashiCorp Cloud Platform, the hosted service formerly called Terraform Cloud) keeps encrypted, versioned state per workspace and locks that workspace for the length of a run.

# A configuration has exactly one backend. Pick one of these.
 
# Azure: state is a blob; locking is a blob lease the storage service enforces.
terraform {
  backend "azurerm" {
    resource_group_name  = "rg-tfstate-prod"
    storage_account_name = "acmetfstateprod"
    container_name       = "tfstate"
    key                  = "prod/network.terraform.tfstate"
 
    # Sign in as an Entra ID identity (Microsoft's directory service) instead
    # of a shared storage account key that anyone can copy out of a pipeline.
    use_azuread_auth = true
  }
}
 
# HCP Terraform: encrypted, versioned state per workspace, locked during a run.
terraform {
  cloud {
    organization = "acme"
    workspaces { name = "prod-network" }
  }
}

Encrypt it with a key you control

Encryption at rest is the door on the storage room. Since 5 January 2023 every new S3 object is encrypted the moment it lands, using keys Amazon holds and manages for you (server-side encryption with Amazon S3 managed keys, shortened to SSE-S3). So "is it encrypted" is rarely the interesting question. Who holds the key is. Nobody outside Amazon writes a policy for those SSE-S3 keys, which leaves one permission, `s3:GetObject`, as the entire path from stranger to plaintext. Point the bucket at a customer-managed KMS key instead, and a read needs two separate grants: the object in S3, and use of the key in KMS. Two doors, two policies, two places an attacker has to be welcome.

$ aws s3api put-bucket-encryption --bucket acme-tfstate-prod \
    --server-side-encryption-configuration '{
      "Rules": [{
        "ApplyServerSideEncryptionByDefault": {
          "SSEAlgorithm": "aws:kms",
          "KMSMasterKeyID": "arn:aws:kms:eu-west-1:111122223333:key/9f3d0c1e-4b8a-4d21-9f77-2c6e8a5b1d40"
        },
        "BucketKeyEnabled": true
      }]
    }'
 
# That command prints nothing on success, so read the setting back:
$ aws s3api get-bucket-encryption --bucket acme-tfstate-prod

{
    "ServerSideEncryptionConfiguration": {
        "Rules": [
            {
                "ApplyServerSideEncryptionByDefault": {
                    "SSEAlgorithm": "aws:kms",
                    "KMSMasterKeyID": "arn:aws:kms:eu-west-1:111122223333:key/9f3d0c1e-4b8a-4d21-9f77-2c6e8a5b1d40"
                },
                "BucketKeyEnabled": true
            }
        ]
    }
}

That second door holds only if the key policy is narrow. Every KMS key carries its own resource policy, and the default one AWS writes hands everything to the account root, which lets any Identity and Access Management (IAM) policy anywhere in the account grant use of that key. That is the door propped open with a brick. Name the roles instead.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "KeyAdministrationOnly",
      "Effect": "Allow",
      "Principal": { "AWS": "arn:aws:iam::111122223333:role/kms-admin" },
      "Action": [
        "kms:Describe*", "kms:Get*", "kms:List*", "kms:Put*", "kms:Update*",
        "kms:Enable*", "kms:Disable*", "kms:Revoke*", "kms:TagResource",
        "kms:ScheduleKeyDeletion", "kms:CancelKeyDeletion"
      ],
      "Resource": "*"
    },
    {
      "Sid": "OnlyTheStateRolesMayUseTheKey",
      "Effect": "Allow",
      "Principal": {
        "AWS": [
          "arn:aws:iam::111122223333:role/tf-plan-prod",
          "arn:aws:iam::111122223333:role/tf-apply-prod"
        ]
      },
      "Action": ["kms:Decrypt", "kms:GenerateDataKey", "kms:DescribeKey"],
      "Resource": "*",
      "Condition": {
        "StringEquals": { "kms:ViaService": "s3.eu-west-1.amazonaws.com" }
      }
    }
  ]
}

Three details are worth slowing down for. `"Resource": "*"` inside a key policy means this key, because the policy is already attached to one. The `kms:ViaService` condition lets those roles use the key only through S3, so a stolen role cannot turn around and point it at other ciphertext. And the admin statement has to stay. AWS runs a lockout safety check on new key policies and refuses one that grants `kms:PutKeyPolicy` to nobody, which is exactly the policy that would leave you shut out of your own key permanently. `kms:Put*` covers that call, which turns key administration into a privileged job of its own.

Versioning, and the public access block

Versioning is the undo history on the bucket. Switch it on and every write keeps the previous copy of the object instead of painting over it. State breaks in ways that have nothing to do with attackers: a dropped connection mid-upload leaves a truncated file, an over-eager `terraform state rm` erases tracking for resources that are still running and still billing. With versions, recovery means copying the last good version back over the current object while nothing else is running, then confirming that `terraform plan` reports no changes. Without versions, recovery means rebuilding state by hand with `terraform import`, one resource at a time, in the middle of an outage. Deny `s3:DeleteObjectVersion` to everyone outside a break-glass role, or whoever corrupted the state can erase the history behind it too.

Blocking public access is the other half, and the cheapest control on this page. A state bucket reachable from the internet is a credential dump for anyone who guesses the name, and `acme-tfstate-prod` takes about five seconds to guess. Set the block at account level as well as bucket level, so a bucket somebody creates next month cannot be born wide open.

$ aws s3api put-bucket-versioning --bucket acme-tfstate-prod \
    --versioning-configuration Status=Enabled
 
$ aws s3api put-public-access-block --bucket acme-tfstate-prod \
    --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
 
# The same block at account level, covering buckets that do not exist yet:
$ aws s3control put-public-access-block --account-id 111122223333 \
    --public-access-block-configuration \
    BlockPublicAcls=true,IgnorePublicAcls=true,BlockPublicPolicy=true,RestrictPublicBuckets=true
 
# None of the three print anything on success. Read all three back:
$ aws s3api get-bucket-versioning --bucket acme-tfstate-prod
$ aws s3api get-public-access-block --bucket acme-tfstate-prod
$ aws s3control get-public-access-block --account-id 111122223333

{
    "Status": "Enabled"
}
{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}
{
    "PublicAccessBlockConfiguration": {
        "BlockPublicAcls": true,
        "IgnorePublicAcls": true,
        "BlockPublicPolicy": true,
        "RestrictPublicBuckets": true
    }
}

Least privilege on the state bucket

Anyone who can read state can read every secret inside it, so treat this bucket as the key cabinet rather than the filing cabinet. Grant per object, not per bucket. The S3 backend asks for very little: `s3:GetObject` and `s3:PutObject` on the one state key; those two plus `s3:DeleteObject` on the `.tflock` object once you use native locking; `s3:ListBucket` on the bucket; and `kms:Decrypt` with `kms:GenerateDataKey` on the key. It never needs delete on the state file itself, never needs `s3:DeleteObjectVersion`, and never needs bucket-wide read.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "OneStateFileAndItsLock",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": [
        "arn:aws:s3:::acme-tfstate-prod/prod/network/terraform.tfstate",
        "arn:aws:s3:::acme-tfstate-prod/prod/network/terraform.tfstate.tflock"
      ]
    },
    {
      "Sid": "ReleaseTheLockObject",
      "Effect": "Allow",
      "Action": "s3:DeleteObject",
      "Resource": "arn:aws:s3:::acme-tfstate-prod/prod/network/terraform.tfstate.tflock"
    },
    {
      "Sid": "ListOnlyItsOwnPrefix",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-tfstate-prod",
      "Condition": { "StringLike": { "s3:prefix": "prod/network/*" } }
    },
    {
      "Sid": "UseTheStateKey",
      "Effect": "Allow",
      "Action": ["kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:eu-west-1:111122223333:key/9f3d0c1e-4b8a-4d21-9f77-2c6e8a5b1d40"
    }
  ]
}

One caveat on that `s3:prefix` condition. It fits the layout this lesson recommends, where each configuration owns exactly one state file and nobody relies on Terraform workspaces. Workspaces keep their state under a different prefix (`env:/`), so a condition this tight would refuse `terraform workspace list`. Past that, a control you have never tested is a belief. Try the read you expect to fail. The developer role below carries a broad old `s3:*` policy, is not named in the key policy, and holds no KMS permissions of its own.

$ aws s3api get-object --bucket acme-tfstate-prod \
    --key prod/network/terraform.tfstate /dev/stdout --profile dev-engineer

An error occurred (AccessDenied) when calling the GetObject operation: User:
arn:aws:sts::111122223333:assumed-role/dev-engineer/sachin is not authorized to
perform: kms:Decrypt on the resource associated with this ciphertext because no
identity-based policy allows the kms:Decrypt action

Read which door refused. S3 was willing, KMS was not, and that is the two-key design doing its job. Notice what did not happen: nobody walked away with encrypted bytes to grind on at home, because S3 decrypts server-side or it refuses outright. Trivy (the scanner that absorbed tfsec) and Checkov both flag these settings in the Terraform code that creates the bucket, so keep that module in the scan set. Checkov's CKV_AWS_21 on its own catches a state bucket shipped without versioning. Then switch on CloudTrail data events for this bucket. CloudTrail is the log AWS keeps of who called which service and when, and object reads are missing from it by default, so a state pull you never see is a breach you never investigate.

Locking: one writer at a time

Back to that key on the block of wood. Every Terraform command that touches state follows one cycle: read the current state, work out the difference, change the cloud, write the state back. Two applies that overlap both read version N, and both write. The second write wins, so resources the first run created vanish from state while they keep running and keep billing. The next plan offers to build them a second time, and a later destroy walks straight past them. The ordinary way this happens is dull. A continuous integration pipeline (CI, the system that runs Terraform for you on every push) gets triggered twice by a quick second commit, while somebody runs `terraform apply` locally to check something.

A lock is a claim written before the change and released after it. The DynamoDB flavour (DynamoDB is Amazon's key-value database) stores one row keyed by `LockID`, set to `<bucket>/<key>`, written with a condition that means "only if this row does not already exist". The second run fails that condition and stops rather than racing. The row records who holds the claim and since when, and a companion row ending in `-md5` holds a checksum of the state, which is how Terraform notices an upload that did not land intact. S3-native locking does the same work with an object named `<key>.tflock` and no table at all. Terraform 1.10 added it, 1.11 made it generally available and deprecated `dynamodb_table`, and you can set both at once during a migration so every run takes a lock in both places.

# Only for the legacy DynamoDB lock table: older setups, or a migration window
$ aws dynamodb create-table \
    --table-name acme-tfstate-locks \
    --attribute-definitions AttributeName=LockID,AttributeType=S \
    --key-schema AttributeName=LockID,KeyType=HASH \
    --billing-mode PAY_PER_REQUEST \
    --sse-specification Enabled=true,SSEType=KMS

{
    "TableDescription": {
        "AttributeDefinitions": [
            {
                "AttributeName": "LockID",
                "AttributeType": "S"
            }
        ],
        "TableName": "acme-tfstate-locks",
        "KeySchema": [
            {
                "AttributeName": "LockID",
                "KeyType": "HASH"
            }
        ],
        "TableStatus": "CREATING",
        "CreationDateTime": "2026-03-04T09:04:11.238000+00:00",
        "ProvisionedThroughput": {
            "NumberOfDecreasesToday": 0,
            "ReadCapacityUnits": 0,
            "WriteCapacityUnits": 0
        },
        "TableSizeBytes": 0,
        "ItemCount": 0,
        "TableArn": "arn:aws:dynamodb:eu-west-1:111122223333:table/acme-tfstate-locks",
        "TableId": "8b1e0f4a-6c22-4b7e-9f31-0d5a7c93e2b1",
        "SSEDescription": {
            "Status": "ENABLED",
            "SSEType": "KMS",
            "KMSMasterKeyArn": "arn:aws:kms:eu-west-1:111122223333:key/2c7a55f1-9d0b-41e2-8a6f-b3d5e0c17a92"
        },
        "BillingModeSummary": {
            "BillingMode": "PAY_PER_REQUEST"
        }
    }
}

# A second run starts while the pipeline's apply is still writing
$ terraform apply -auto-approve

Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: operation error DynamoDB: PutItem, https response error
│ StatusCode: 400, RequestID: 9K1Q7B2VJ0RM4H8T3F,
│ ConditionalCheckFailedException: The conditional request failed
│ Lock Info:
│   ID:        3f8b9c1a-6d24-4f8b-9e0a-1a2b3c4d5e6f
│   Path:      acme-tfstate-prod/prod/network/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       runner@ci-runner-7f9
│   Version:   1.11.4
│   Created:   2026-03-04 09:12:41.882913 +0000 UTC
│   Info:
│
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

That message is the control working. `Who` names the machine and the user holding the claim, `Created` says since when, and `ID` is what any recovery command needs. With `use_lockfile` the first line differs, because S3 answers a conditional write with `412 PreconditionFailed` where DynamoDB reports a failed condition, while the lock info underneath reads the same. The honest answer is usually to wait. A lock releases itself when the holding run ends, and that includes runs that end in failure. A lock outlives its owner only when the process died without cleaning up: a cancelled job, a killed container, a laptop that went to sleep on the train.

$ terraform force-unlock 3f8b9c1a-6d24-4f8b-9e0a-1a2b3c4d5e6f

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may still be in use. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Terraform state has been successfully unlocked!
 
The state has been unlocked, and Terraform commands should now be able to
obtain a new lock on the remote state.

One environment, one bucket, one role

Give each environment its own room and its own key rather than handing out a master. Dev and prod want different buckets, different KMS keys, different roles and, where you can arrange it, different AWS accounts, so a dev run has no route to prod state at all. Backend blocks take no variables, so the pattern is partial configuration: leave `backend "s3" {}` empty in code and pass the rest at init time from a `.tfbackend` file per environment. `-reconfigure` is the flag you want here, because you are pointing Terraform at other state that already exists. `-migrate-state` carries state from one home to another, which is a different job.

# backend.tf holds only:  terraform { backend "s3" {} }
bucket       = "acme-tfstate-prod"
key          = "prod/network/terraform.tfstate"
region       = "eu-west-1"
encrypt      = true
kms_key_id   = "arn:aws:kms:eu-west-1:111122223333:key/9f3d0c1e-4b8a-4d21-9f77-2c6e8a5b1d40"
use_lockfile = true
 
# envs/dev.s3.tfbackend names a different bucket, key and KMS key. The role
# doing the reading comes from the credentials the run already holds, so the
# dev pipeline assumes a dev role the prod key policy has never heard of.

$ terraform init -reconfigure -backend-config=envs/prod.s3.tfbackend

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.94.1
 
Terraform has been successfully initialized!

The other half of this problem is who Terraform runs as. A perfectly hardened bucket hands everything over anyway if the pipeline reaching it carries a long-lived access key that one leaked log can expose. The next lesson swaps those keys for short-lived credentials issued through OIDC (OpenID Connect, the standard a CI system uses to prove its identity to AWS without storing a secret), and splits the role allowed to plan from the role allowed to apply.

Related