Remote state & locking

State for teams, done safely.

A local state file is a paper notebook in your own desk drawer. Fine while you are the only person touching the building. Add a second engineer and now there are two notebooks that disagree about which doors exist. Have both of you write in yours at the same moment and the record is worth nothing. Terraform state is that notebook: a JSON file (JavaScript Object Notation, a plain-text format for structured data) listing every resource Terraform created, the real identifier it was given in the cloud, and the settings it was built with. A shared filing cabinet with a lock on the front fixes both problems at once. In Terraform, that cabinet is called a backend.

A backend is where Terraform keeps state. The usual choices are an Amazon S3 (Simple Storage Service, Amazon's object storage) bucket, an Azure Blob Storage container, a Google Cloud Storage bucket, or HCP Terraform (HashiCorp Cloud Platform, the hosted service formerly called Terraform Cloud). Point everybody at the same backend and there is exactly one state file. That single copy is what buys you everything else: encryption at rest, a version history you can roll back to, an access log you can switch on, and locking, so two applies cannot interleave and shred each other's work.

Pointing a Configuration at Shared Storage

terraform {
  backend "s3" {
    bucket       = "acme-tf-state"
    key          = "prod/network/terraform.tfstate"
    region       = "us-east-1"
    encrypt      = true          # ask S3 to encrypt the object on write
    kms_key_id   = "arn:aws:kms:us-east-1:111122223333:key/9c8f2b1a-4d7e-4a1f-b3c2-6e5d0f7a8b91"
    use_lockfile = true          # native S3 locking, Terraform 1.10 and later
  }
}

Read that block as an address. The bucket is the cabinet. The key is the exact drawer inside it, and it doubles as the name of this stack. Setting encrypt to true tells S3 to encrypt the object as it lands, and kms_key_id points at a customer-managed key in KMS (Key Management Service, where AWS stores and guards encryption keys), so you decide who is allowed to decrypt rather than taking the shared default key that half the account can already use. That long arn:aws:kms:... string is an ARN (Amazon Resource Name), the full-length unique identifier AWS gives every resource. Now add this block to a directory that already has a local state file. Terraform will not quietly swap over. It asks first.

terraform init -migrate-state

Initializing the backend...
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.
 
  Enter a value: yes
 
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v6.7.0
 
Terraform has been successfully initialized!

Two things changed on disk. The old terraform.tfstate is still sitting in the directory, usually next to a terraform.tfstate.backup that Terraform left behind, and both are stale from this moment on. Delete them, and make sure *.tfstate* and .terraform/ are in .gitignore. The other change is .terraform/terraform.tfstate, which sounds like state and is not. It records which backend this directory is wired to. That file is how Terraform remembers the answer between commands, and reading it back is the fastest way to prove a working directory is talking to the backend you think it is.

jq -r '.backend.type, .backend.config.bucket, .backend.config.key' .terraform/terraform.tfstate

s3
acme-tf-state
prod/network/terraform.tfstate

Your State File Is a Secrets File

A locksmith's job sheet records which locks went on which doors. It also records what shape the keys were cut. Terraform state works the same way. It stores the attribute values of everything it built, and plenty of those values are secrets: a database master password, a private key generated by the tls provider, an access key minted for a service account. Marking a variable or an output as sensitive hides the value from terminal output and from plan text. That is the whole of what it does. The state file stays ordinary JSON, and anyone who can download it can read the lot.

terraform state pull | jq '{version, serial, lineage, resources: (.resources | length)}'

{
  "version": 4,
  "serial": 187,
  "lineage": "8f0c4e2a-3b19-4a77-9c1e-2d5b6a7f8e90",
  "resources": 42
}

Serial is a counter that climbs every time Terraform writes the file. Lineage is a UUID (universally unique identifier, a long random string stamped once and never changed) that marks this state as one continuous history. Terraform checks both to notice a state file that has been swapped out from under it. Now pull the same file again and look at what is actually stored inside.

terraform state pull | jq -r '.resources[] | select(.type == "aws_db_instance") | .instances[].attributes.password'

pr0d-orders-db-2026-Jul!

That is the whole attack, in one command. Someone who phishes a read-only role, digs up a forgotten CI (continuous integration, the system that builds and deploys your code automatically) token, or clones a repository where a colleague once committed state does not need to break into your database. They read the password out of the notebook, along with a complete map of every VPC (virtual private cloud, your own private network inside AWS), subnet, security group and instance you run.

So treat read access to the state bucket as close to administrator access, because that is what it is worth. The pipeline role gets s3:GetObject, s3:PutObject and s3:DeleteObject on its own key prefix, s3:ListBucket on the bucket, and kms:Decrypt plus kms:GenerateDataKey on that one key. DeleteObject is not optional once you use lock files, because releasing a lock means deleting an object. Humans get none of this by default and step into a break-glass role when they genuinely need it, the kind of role that sets an alarm off when anyone uses it. Then add a bucket policy that refuses anything arriving without TLS (Transport Layer Security, the encryption behind the s in https), switch on Block Public Access, and switch on versioning so a bad write is recoverable.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "DenyInsecureTransport",
      "Effect": "Deny",
      "Principal": "*",
      "Action": "s3:*",
      "Resource": [
        "arn:aws:s3:::acme-tf-state",
        "arn:aws:s3:::acme-tf-state/*"
      ],
      "Condition": { "Bool": { "aws:SecureTransport": "false" } }
    }
  ]
}

Then make reads visible. Object-level GetObject calls do not show up in CloudTrail (the AWS log of who called which API) unless you switch on data events for that bucket, and they are off by default because AWS bills for them. Without them, the question "who downloaded prod state last month" has no answer at all. With them, a state read by an unexpected principal, from an unexpected address, at three in the morning, is a detection you can alert on. Check the control by asking the API, not by trusting the console screenshot somebody took last year.

aws cloudtrail get-event-selectors --trail-name acme-audit \
  --query 'EventSelectors[0].DataResources'

[
    {
        "Type": "AWS::S3::Object",
        "Values": [
            "arn:aws:s3:::acme-tf-state/"
        ]
    }
]

The storage side answers just as directly. Two commands, two facts you either have or do not.

aws s3api get-bucket-versioning --bucket acme-tf-state

{
    "Status": "Enabled"
}

aws s3api get-bucket-encryption --bucket acme-tf-state \
  --query 'ServerSideEncryptionConfiguration.Rules[0].ApplyServerSideEncryptionByDefault'

{
    "SSEAlgorithm": "aws:kms",
    "KMSMasterKeyID": "arn:aws:kms:us-east-1:111122223333:key/9c8f2b1a-4d7e-4a1f-b3c2-6e5d0f7a8b91"
}

You want to see aws:kms and your own key there. If it says AES256, the bucket is falling back to the encryption S3 applies to everything by default, which protects the bytes on the disk but gives you no separate list of who is allowed to decrypt.

Locking: One Pen Chained to the Ledger

Shared storage on its own still lets two people write over each other. A bank counter solves that with a single pen chained to the desk. Terraform does the same thing: before it touches state for an operation it writes a lock record, and it deletes that record when the run ends. With S3 there are two ways to hold the pen. The old pairing is a DynamoDB table (DynamoDB is Amazon's key-value database) with a partition key named LockID, wired up through dynamodb_table. Terraform 1.10 added native locking that needs no side table at all: use_lockfile = true writes a small object beside your state, the state key with .tflock on the end, created with an S3 conditional write so exactly one writer can win the race. Terraform 1.11 deprecated dynamodb_table in favour of the lock file. It still works, but it is on the way out. The GCS, azurerm and HCP Terraform backends lock natively, with nothing extra for you to build.

terraform {
  backend "s3" {
    bucket  = "acme-tf-state"
    key     = "prod/network/terraform.tfstate"
    region  = "us-east-1"
    encrypt = true
 
    # Overlap period: with both set, Terraform takes both locks.
    # Run one apply from every workspace, then delete dynamodb_table.
    use_lockfile   = true
    dynamodb_table = "acme-tf-locks"   # deprecated in 1.11, still honoured
  }
}

The payoff shows up the first time two people move at once. Here is a second engineer starting an apply while a colleague's run is still going.

terraform apply

Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│ 
│ Error message: operation error S3: PutObject, https response error
│ StatusCode: 412, RequestID: 0P9K2M4Q7R1S3T5V, api error PreconditionFailed:
│ At least one of the pre-conditions you specified did not hold
│ Lock Info:
│   ID:        3f0c9d2e-6a1b-4e7c-9f31-0a5b8c2d4e6f
│   Path:      acme-tf-state/prod/network/terraform.tfstate.tflock
│   Operation: OperationTypeApply
│   Who:       alice@runner-ci-07
│   Version:   1.12.2
│   Created:   2026-07-21 09:14:03.114523 +0000 UTC
│   Info:      
│ 
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

Nothing was changed and nothing was corrupted. The run stopped before it touched a single resource. Read the fields before you do anything else. The 412 is HTTP's "precondition failed": Terraform asked S3 to create the .tflock object only if it did not already exist, it did exist, and S3 refused. Exactly the behaviour you want. ID is the lock identifier you would need if you ever had to force it open. Who is the operating system user and the hostname that took it, which is usually enough to find the CI job. Operation tells you whether the other run is applying or only planning. Created is UTC (coordinated universal time), not your local clock, so do the arithmetic before you accuse anyone. And note that plan takes the lock too, not only apply. That is why a plan refuses to start while somebody else is mid-apply.

The DynamoDB setup carries one extra trick worth knowing. Alongside the lock item it keeps a second item whose LockID ends in -md5, holding a checksum (a short fingerprint of the file's bytes) of the state Terraform last wrote. If S3 later hands back an object whose fingerprint does not match, Terraform stops and tells you the state data does not have the expected content, instead of planning against a half-written file and cheerfully offering to delete production. The lock file has no equivalent item, and it does not need one in the same way: that digest was built for the years when S3 was only eventually consistent, and S3 has been strongly read-after-write consistent since December 2020. The conditional write behind .tflock decides who gets the pen. It says nothing about the bytes in your state file.

Clearing a Lock That Is Genuinely Dead

Runs do die. A CI runner gets evicted mid-apply, a laptop loses its network, someone hits Ctrl-C twice. The lock record survives the process that made it, and every later run stops on it. Start by reading the record rather than guessing, because it tells you exactly who to go and ask.

aws s3 cp s3://acme-tf-state/prod/network/terraform.tfstate.tflock -

{"ID":"3f0c9d2e-6a1b-4e7c-9f31-0a5b8c2d4e6f","Operation":"OperationTypeApply","Info":"","Who":"alice@runner-ci-07","Version":"1.12.2","Created":"2026-07-21T09:14:03.114523Z","Path":"acme-tf-state/prod/network/terraform.tfstate.tflock"}

On a DynamoDB setup the same information lives in the table, keyed by bucket and state path.

aws dynamodb get-item --table-name acme-tf-locks \
  --key '{"LockID":{"S":"acme-tf-state/prod/network/terraform.tfstate"}}' \
  --query 'Item.Info.S' --output text

{"ID":"3f0c9d2e-6a1b-4e7c-9f31-0a5b8c2d4e6f","Operation":"OperationTypeApply","Info":"","Who":"alice@runner-ci-07","Version":"1.12.2","Created":"2026-07-21T09:14:03.114523Z","Path":"acme-tf-state/prod/network/terraform.tfstate"}

Runner ci-07 finished forty minutes ago with a failure, and Alice confirms the job was killed. Now, and only now, you clear the lock using the ID from the record.

terraform force-unlock 3f0c9d2e-6a1b-4e7c-9f31-0a5b8c2d4e6f

Do you really want to force-unlock?
  Terraform will remove the lock on the remote state.
  This will allow local Terraform commands to modify this state, even though it
  may still be in use. Only 'yes' will be accepted to confirm.
 
  Enter a value: yes
 
Terraform state has been successfully unlocked!

That removes the lock record and nothing else. It does not repair state. If the run died partway through writing, the object in the bucket may be behind reality, and the fix is the version history you switched on earlier. List the versions, find the last one written before the failed run started, copy it back over the current object, then run a plan and read every line of it before you agree to anything.

aws s3api list-object-versions --bucket acme-tf-state \
  --prefix prod/network/terraform.tfstate \
  --query 'Versions[:3].[VersionId,LastModified,IsLatest]' --output text

3sL4kqtJlcpXroDTDmJ.rEQ3TlZRxDA	2026-07-21T09:22:41+00:00	True
Kx9CmT8pLdN1sVbA7Ke.QUpfdndhfd8	2026-07-21T08:47:12+00:00	False
9zXbQ1oT4mNvCe7hRk.3PdLsWq0aYtG	2026-07-19T16:03:55+00:00	False

Hand-editing state, or pushing a file back with terraform state push, is the last resort. It is also the most reliable way anyone has ever found to lose track of live resources.

Splitting State to Shrink the Blast Radius

One state file for the whole company gives you one lock everybody queues behind, a plan that walks thousands of resources before it says anything useful, and a single mistyped command that can reach every resource you own. That last one is what people mean by blast radius: how much damage one bad run can do. Remote state lets you cut it down. The key in the backend block is a path, so give each stack its own. Network, data and application layers in separate files, one set per environment. Smaller states plan faster, lock independently, and can carry different permissions, which is how you remove the application pipeline's technical ability to touch the VPC at all.

acme-tf-state/
  prod/network/terraform.tfstate    # VPC, subnets, routing            (network team)
  prod/data/terraform.tfstate       # RDS databases, buckets, KMS keys (data team)
  prod/app/terraform.tfstate        # ECS services, load balancers     (app team)
  stage/network/terraform.tfstate   # same code, own state, own lock

Separated stacks still have to pass values to each other. The application stack needs the subnet IDs the network stack created. A terraform_remote_state data source reads another state file's root outputs, so the values stay current even after the network team renumbers something.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tf-state"
    key    = "prod/network/terraform.tfstate"
    region = "us-east-1"
  }
}
 
resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  subnet_id     = data.terraform_remote_state.network.outputs.private_subnet_ids[0]
}

There is a catch here that people miss until a security review finds it. Terraform downloads the whole state object to get at those outputs, so anything that can use this data source can also fetch the network state directly and read every attribute inside it. You meant to share two subnet IDs. You granted access to the file holding the network team's secrets. When the boundary between stacks actually matters, publish the values deliberately instead: write them to SSM Parameter Store (Systems Manager Parameter Store, a small AWS key-value store for configuration) or Secrets Manager, and have the consuming stack read that one parameter with an ordinary data source. The consumer then holds permission on a single value rather than a whole state file.

Bootstrapping, and Why the Backend Block Hates Variables

The bucket has to exist before any configuration can store state in it, which is a chicken-and-egg problem. Solve it with a small bootstrap configuration that builds the bucket, its key and its guardrails, applied once with local state, then committed so the setup is written down instead of remembered by whoever ran it. Protect it with prevent_destroy so a careless destroy in that directory cannot take the cabinet with it. Note what is missing from this file: no DynamoDB table, because the lock file lives in the bucket you are already creating.

resource "aws_kms_key" "state" {
  description         = "terraform state encryption"
  enable_key_rotation = true
}
 
resource "aws_s3_bucket" "state" {
  bucket = "acme-tf-state"
 
  lifecycle {
    prevent_destroy = true      # refuse to even plan a destroy of this bucket
  }
}
 
resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.state.arn
    }
    bucket_key_enabled = true
  }
}
 
resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}

The second surprise is that a backend block accepts no variables, no locals, and no interpolation of any kind. Terraform has to know where state lives before it evaluates anything else in your configuration, so the block is read very early and every value in it must be literal. The way round this is partial configuration: leave the block empty and feed the values in at init time from one file per environment, conventionally named with a .tfbackend extension.

terraform {
  backend "s3" {}   # empty on purpose: no variables are allowed in here
}

bucket       = "acme-tf-state"
key          = "prod/network/terraform.tfstate"
region       = "us-east-1"
encrypt      = true
use_lockfile = true

terraform init -reconfigure -backend-config=prod.s3.tfbackend

Initializing the backend...
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v6.7.0
 
Terraform has been successfully initialized!

Two sharp edges hide in that one command. The first: whatever you pass to -backend-config gets written into .terraform/terraform.tfstate in the working directory, resolved values and all. Put a long-lived access key in a .tfbackend file and it sits in plaintext in the workspace, and on every CI runner that has ever checked the repository out. Pass credentials through an assumed role or the environment instead, keep .terraform/ out of git, and make CI wipe its workspace between jobs.

The second edge is the difference between the two init flags. -reconfigure drops the association with the previous backend and copies nothing. -migrate-state copies the state across. Reach for -reconfigure during a genuine backend move and you can end up initialised against an empty state file, one apply away from Terraform deciding it needs to build your entire production network a second time.

Before you trust any of this, do the two-terminal drill. Open two shells against the same stack. Start an apply in the first. While it is still running, start a plan in the second. If the second one stops with Error acquiring the state lock, the cabinet is locked and the pen is chained to the desk. If it starts planning away happily, locking is not actually on, and you have found that out on a quiet Tuesday instead of during an incident.

Try this

Run terraform init -migrate-state on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: -lock=false is not a fix, and neither is a blind force-unlock. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related