Remote state & locking

State for teams, done safely.

A local state file is a paper ledger in your desk drawer. That works fine while you are the only person who writes in it. The moment a second person needs the same ledger, you photocopy it, and now there are two ledgers that disagree. Terraform state fails in exactly that way. The terraform.tfstate file on your laptop is the only record of which real cloud resources Terraform believes it created and owns, and the second a teammate or a CI runner (continuous integration, the build server that runs your pipeline with nobody watching) needs that record, one copy on one machine stops being the truth.

It breaks in three ways, and you will meet all three. A colleague applies from her copy and creates a load balancer; your copy has never heard of it, so your next plan cheerfully offers to create a second one. A laptop dies and takes the only map of production with it. Or two applies land within the same few seconds, both read the old state, both write their version back, and the second write erases the first one's resources from the record. Those resources still exist. They still bill you. Terraform now has no idea they are there. Shared storage fixes the first two problems. Locking fixes the third.

Moving State Off Your Laptop

Take the ledger out of the drawer and put it in a safe deposit box at the bank. One box, one copy, and everyone who holds a key can reach the same pages. That is a backend: the place Terraform keeps state, plus the rules for reaching it. Instead of a file sitting next to your code, state lives somewhere every human and every pipeline can get to. An S3 bucket (Simple Storage Service, Amazon's object store), an Azure Storage blob, a Google Cloud Storage bucket, or HCP Terraform (HashiCorp Cloud Platform, the hosted service). S3 is the one most teams meet first, so that is what this lesson uses. Put the configuration in its own file so nobody can miss it.

terraform {
  required_version = ">= 1.11"
 
  backend "s3" {
    bucket       = "acme-tfstate"
    key          = "prod/network/terraform.tfstate"   # path INSIDE the bucket
    region       = "us-east-1"
    encrypt      = true                                # ask S3 to encrypt every write
    kms_key_id   = "arn:aws:kms:us-east-1:111122223333:key/8c1f2b60-3e9a-4c77-9a1d-2f5b7d0e4a13"
    use_lockfile = true                                # S3-native locking
 
    # dynamodb_table = "acme-tf-locks"   # the old lock table, deprecated in TF 1.11
  }
}

key is the path of the state object inside the bucket, and writing it as a real path (prod/network/...) matters far more than it looks. That comes back at the end. encrypt = true asks S3 for server-side encryption on every write. S3 has encrypted new objects by default since early 2023, so the genuine step up is kms_key_id: point state at a customer-managed key in KMS (Key Management Service, the AWS key store) and that key's policy becomes a second door. Someone holding s3:GetObject on the bucket but no kms:Decrypt on the key gets bytes they cannot read. use_lockfile = true switches on the locking you are about to meet.

One detail catches everybody. The backend block cannot use variables, locals, or any interpolation at all, because Terraform reads it before it evaluates anything else in your configuration. If bucket names differ per environment, leave those arguments out and supply them at init time with terraform init -backend-config=prod.s3.tfbackend. Changing backend settings later needs either -reconfigure (start fresh) or -migrate-state (bring the existing state along), and picking the wrong one is how people accidentally begin with empty state.

$ terraform init -migrate-state

Initializing the backend...
 
Do you want to copy existing state to the new backend?
  Pre-existing state was found while migrating the previous "local" backend to the
  newly configured "s3" backend. No existing state was found in the newly
  configured "s3" backend. Do you want to copy this state to the new "s3"
  backend? Enter "yes" to copy and "no" to start with an empty state.
 
  Enter a value: yes
 
 
Successfully configured the backend "s3"! Terraform will automatically
use this backend unless the backend configuration changes.
 
Initializing provider plugins...
- Reusing previous version of hashicorp/aws from the dependency lock file
- Using previously-installed hashicorp/aws v5.62.0
 
Terraform has been successfully initialized!

# migration done. now look at what is still sitting in your working directory:
$ ls -l terraform.tfstate*

-rw-rw-r-- 1 dev dev 18244 Jul 21 10:02 terraform.tfstate
-rw-rw-r-- 1 dev dev 18244 Jul 21 10:02 terraform.tfstate.backup

Terraform does not clean up after itself here. The local file is still there, still holding every secret in plaintext, on a laptop, one careless git add away from a repository history you cannot rewrite. Confirm the remote copy reads back with terraform state pull, then delete both local files. While you are there, check that *.tfstate* is in .gitignore, before anyone else clones the repo.

How A State Lock Actually Works

Old service stations kept a single restroom key chained to a wooden paddle the size of a cricket bat. One key, so one person inside at a time, and no argument about who had it. A state lock is that paddle. Before Terraform touches state, it tries to create one small marker that exists only while a run is in progress. If it cannot create the marker, because somebody else got there first, it refuses to run at all.

With use_lockfile = true the marker is an object sitting next to your state, named terraform.tfstate.tflock, written with an S3 conditional put (a write the service rejects outright if the object already exists). That rejection is the lock. Terraform 1.10 shipped this mechanism and 1.11 made it the supported one, deprecating the older approach: a DynamoDB table (dynamodb_table = "acme-tf-locks", whose partition key must be a string attribute called LockID). If you are moving across, set both arguments for one release so old and new clients still block each other, then drop dynamodb_table.

# ci-runner-3 is already mid-apply against this state. you try it from your laptop:
$ terraform apply -lock-timeout=60s

Acquiring state lock. This may take a few moments...
╷
│ Error: Error acquiring the state lock
│
│ Error message: operation error S3: PutObject, https response error StatusCode:
│ 412, RequestID: 9QK2ZM1J5TRWB4XE, api error PreconditionFailed: At least one of
│ the pre-conditions you specified did not hold
│ Lock Info:
│   ID:        4f2a1c9e-1b0d-6a3f-9c21-8e77d4f0a1b2
│   Path:      acme-tfstate/prod/network/terraform.tfstate
│   Operation: OperationTypeApply
│   Who:       deploy@ci-runner-3
│   Version:   1.13.1
│   Created:   2026-07-21 10:14:02.113355 +0000 UTC
│   Info:
│
│ Terraform acquires a state lock to protect the state from being written
│ by multiple users at the same time. Please resolve the issue above and try
│ again. For most commands, you can disable locking with the "-lock=false"
│ flag, but this is not recommended.
╵

Every line of that error earns its place. Who is the user and host holding the lock. Operation says whether it is a plan or an apply. Created tells you how long it has been held, and ID is the value force-unlock would want. The -lock-timeout=60s made Terraform retry for a minute before giving up, which is what you want in a pipeline so a queued run waits its turn instead of failing the build. And yes, terraform plan takes a lock too, because it refreshes state while it runs.

# the lock is a normal S3 object. read it directly:
$ aws s3 cp s3://acme-tfstate/prod/network/terraform.tfstate.tflock - | jq .

{
  "ID": "4f2a1c9e-1b0d-6a3f-9c21-8e77d4f0a1b2",
  "Operation": "OperationTypeApply",
  "Info": "",
  "Who": "deploy@ci-runner-3",
  "Version": "1.13.1",
  "Created": "2026-07-21T10:14:02.113355Z",
  "Path": "acme-tfstate/prod/network/terraform.tfstate"
}

On the older DynamoDB setup the same JSON (JavaScript Object Notation, the text format above) lives in the Info attribute of the row whose LockID is the bucket-plus-key string, readable with aws dynamodb get-item. Either way this answers the only question that matters during a lock standoff: is this lock alive, or is it a corpse? A lock created ninety seconds ago by a job still running green in your CI dashboard belongs to a colleague, and you go talk to them. A lock created two hours ago by a runner that no longer exists is debris.

What Your State File Actually Contains

Terraform records every attribute of every resource it manages, including the ones you would never print on purpose. The initial password on an RDS database (Relational Database Service, the managed database offering), a generated private key, the plaintext of anything you fed into a resource. Marking a variable or output sensitive = true hides the value from console output and does nothing else. It has no effect whatsoever on what lands in state.

$ terraform state pull | jq -r '.resources[] | select(.type == "aws_db_instance") | .instances[].attributes | [.identifier, .password] | join("  ")'

acme-prod-orders  Tr0ub4dor-and-3-prod

Read access to that bucket is therefore read access to production secrets, and it is also a complete inventory: every account ID, subnet, security group, and hostname you run. Two things follow. Keep secrets out of state where the provider lets you, using write-only arguments such as password_wo (Terraform 1.11 and later discards those values instead of persisting them) or manage_master_user_password = true, which hands the password to Secrets Manager and leaves only an ARN (Amazon Resource Name, the unique identifier AWS gives every resource) in state. Then split the permissions so a stolen CI credential reaches one state object and nothing else.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ReadWriteOneStateObject",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject"],
      "Resource": "arn:aws:s3:::acme-tfstate/prod/app/terraform.tfstate"
    },
    {
      "Sid": "ManageOnlyItsOwnLock",
      "Effect": "Allow",
      "Action": ["s3:GetObject", "s3:PutObject", "s3:DeleteObject"],
      "Resource": "arn:aws:s3:::acme-tfstate/prod/app/terraform.tfstate.tflock"
    },
    {
      "Sid": "BackendNeedsToListTheBucket",
      "Effect": "Allow",
      "Action": "s3:ListBucket",
      "Resource": "arn:aws:s3:::acme-tfstate"
    },
    {
      "Sid": "UseTheStateKey",
      "Effect": "Allow",
      "Action": ["kms:Encrypt", "kms:Decrypt", "kms:GenerateDataKey"],
      "Resource": "arn:aws:kms:us-east-1:111122223333:key/8c1f2b60-3e9a-4c77-9a1d-2f5b7d0e4a13"
    }
  ]
}

Look closely at the delete rights, because this is where teams get it backwards. Terraform never deletes the state object, so the pipeline never needs s3:DeleteObject on it. Terraform does delete the .tflock object every time it releases a lock, so the pipeline absolutely needs s3:DeleteObject on that one key. Deny it and every run hangs on to its lock forever. Grant it across the whole prefix and a compromised runner can erase your state. The KMS statement is the other half people forget: with a customer-managed key, Terraform needs kms:Encrypt, kms:Decrypt and kms:GenerateDataKey or init fails on the first read. Engineers get read on the states they genuinely plan against and nothing on the rest. Bucket-level power (s3:PutBucketVersioning, s3:PutBucketPolicy, kms:ScheduleKeyDeletion) belongs to a break-glass role a human has to deliberately assume, never to something that runs unattended.

Building The Bucket That Holds Everything

Now the chicken-and-egg problem, the one you get with a safe whose only key is locked inside it. The bucket has to exist before Terraform can store state in it. Keep a small bootstrap configuration that creates the backend and holds its own tiny state, either locally in a repo nobody applies casually or in a second bucket you made by hand. It runs about once a year. What matters is what it turns on.

resource "aws_kms_key" "state" {
  description             = "acme terraform state"
  enable_key_rotation     = true
  deletion_window_in_days = 30
}
 
resource "aws_s3_bucket" "state" {
  bucket = "acme-tfstate"
}
 
resource "aws_s3_bucket_versioning" "state" {
  bucket = aws_s3_bucket.state.id
  versioning_configuration { status = "Enabled" }    # keep every previous write
}
 
resource "aws_s3_bucket_public_access_block" "state" {
  bucket                  = aws_s3_bucket.state.id
  block_public_acls       = true
  block_public_policy     = true
  ignore_public_acls      = true
  restrict_public_buckets = true
}
 
resource "aws_s3_bucket_server_side_encryption_configuration" "state" {
  bucket = aws_s3_bucket.state.id
  rule {
    apply_server_side_encryption_by_default {
      sse_algorithm     = "aws:kms"
      kms_master_key_id = aws_kms_key.state.arn     # customer-managed, not the AWS default
    }
    bucket_key_enabled = true                      # cheaper: one data key per bucket, not per object
  }
}
 
resource "aws_s3_bucket_policy" "tls_only" {
  bucket = aws_s3_bucket.state.id
  policy = jsonencode({
    Version = "2012-10-17"
    Statement = [{
      Sid       = "DenyInsecureTransport"
      Effect    = "Deny"
      Principal = "*"
      Action    = "s3:*"
      Resource  = [aws_s3_bucket.state.arn, "${aws_s3_bucket.state.arn}/*"]
      Condition = { Bool = { "aws:SecureTransport" = "false" } }
    }]
  })
}

Versioning is the recovery story: every write keeps the previous copy, so a truncated or wrong state becomes a restore rather than an outage. The public access block turns a future bad bucket policy into a non-event. The encryption default keeps the key requirement in force even when someone writes with a client that forgets to ask for it. The last policy denies any request that did not arrive over TLS (Transport Layer Security, the S in HTTPS). Add an aws_s3_bucket_lifecycle_configuration later if old versions pile up, but keep them long enough to survive a bad week, not a bad hour.

Seeing Who Read Your State

A building's front desk writes down who changed the locks. Almost nobody writes down who opened a filing cabinet and read a page. AWS works the same way, and it is a problem here, because an attacker with read-only credentials does not need to break anything. One GetObject against prod/network/terraform.tfstate hands them database passwords and a map of the environment, quietly, using a permission that looks harmless in a policy review. CloudTrail (the AWS audit log of API calls) records management events out of the box, so you will see who created the bucket or changed its policy. Reading an object is a data event, and data events stay off until you switch them on.

[
  {
    "Name": "Management events",
    "FieldSelectors": [
      { "Field": "eventCategory", "Equals": ["Management"] }
    ]
  },
  {
    "Name": "Terraform state object access",
    "FieldSelectors": [
      { "Field": "eventCategory", "Equals": ["Data"] },
      { "Field": "resources.type", "Equals": ["AWS::S3::Object"] },
      { "Field": "resources.ARN", "StartsWith": ["arn:aws:s3:::acme-tfstate/"] }
    ]
  }
]

$ aws cloudtrail put-event-selectors --trail-name org-trail \
    --advanced-event-selectors file://state-data-events.json \
    --query 'AdvancedEventSelectors[].Name'

[
    "Management events",
    "Terraform state object access"
]

Two traps in that one command. put-event-selectors replaces the entire selector set on the trail, which is the only reason the management selector is in the file at all. Send only the data selector and you silently stop logging management events across the whole trail, so run get-event-selectors first and copy what is already there. The second trap is that aws cloudtrail lookup-events searches management events only, so these reads will never appear there no matter how you query it. Read them where the trail delivers them, with Athena (a query engine that runs SQL directly over files in S3) or CloudWatch Logs Insights if you forward the trail. Then write two alerts: GetObject on the state prefix by any principal that is not your CI role, and any DeleteObject, PutBucketVersioning or PutBucketPolicy on that bucket at all.

One State Per Blast Radius

A master key that opens every door in the building is convenient right up to the day you lose it. That key path is doing the same job as separate room keys. One giant state for an entire account means slow plans, a single lock everyone queues behind, and one mistake that can reach anything. Splitting into prod/network, prod/data and prod/app gives each piece its own lock, its own IAM boundary (Identity and Access Management, the AWS permission system), and a blast radius the size of that piece. The app team applies all day without ever holding the lock the network team needs. When one stack genuinely needs a value from another, Terraform can read across.

data "terraform_remote_state" "network" {
  backend = "s3"
  config = {
    bucket = "acme-tfstate"
    key    = "prod/network/terraform.tfstate"   # read-only, someone else's state
    region = "us-east-1"
  }
}
 
resource "aws_instance" "api" {
  subnet_id = data.terraform_remote_state.network.outputs.private_subnet_id
}

Notice the price of that convenience. terraform_remote_state needs s3:GetObject on the whole network state object, plus kms:Decrypt on its key, which hands the app team every secret the network stack holds in order to fetch one subnet ID. For anything crossing a trust boundary, publish the specific values instead. Write them to SSM Parameter Store (Systems Manager, the AWS configuration and secret store) from the producing stack and read them back with a data "aws_ssm_parameter" block, or look the resource up by tag with an ordinary data source. Narrow reads beat convenient ones.

One last thing worth doing before you close this page. Versioning only helps if you have practised using it, so find out right now what your state's history actually looks like. The prefix filter alone would also match the .tflock object, which is why the query pins the exact key.

$ aws s3api list-object-versions --bucket acme-tfstate \
    --prefix prod/network/terraform.tfstate \
    --query 'Versions[?Key==`prod/network/terraform.tfstate`] | [:3].{when:LastModified,id:VersionId,latest:IsLatest}'

[
    {
        "when": "2026-07-21T10:16:44+00:00",
        "id": "5vN2mQ0pTt.RvXk9c1uZbYo7hJdA3wLe",
        "latest": true
    },
    {
        "when": "2026-07-21T09:02:11+00:00",
        "id": "kQ9wD7sT.aB4nR2xLmYcVfHg1JpZ0eUi",
        "latest": false
    },
    {
        "when": "2026-07-20T18:31:07+00:00",
        "id": "Xr3TuA8bN.cE5vK7mJ2sQdYw6LpZ9fHo",
        "latest": false
    }
]

Try this

Run terraform init -migrate-state on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: force-unlock is a loaded gun, and -lock=false is worse. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related