Terraform interview questions
Prep Terraform interviews from first plan to production CI: state, modules, refactoring, drift, and how secrets and provider auth stay safe.
Beginner → Intermediate → Advanced → Expert — answers are written the way you’d say them in a real interview. Advanced and Expert go deeper with war-story detail, architecture diagrams, and the follow-up an interviewer often asks next.
What is Terraform state?Beginner
The short version is: state is Terraform's memory. It's a JSON map from config addresses to real resource IDs and attributes, so Terraform can diff, own resources, and know what to update or destroy.
terraform state list terraform state show aws_instance.web # state = address -> provider ID + attributes
What is the core Terraform workflow?Beginner
I'd describe it as write → init → plan → apply. You write HCL, init pulls providers and wires the backend, plan shows the diff, apply makes it real. Destroy is when you're tearing the stack down.
terraform init terraform plan -out tf.plan terraform apply tf.plan
plan vs apply — what is the difference?Beginner
Plan figures out the create/update/replace/destroy actions; apply actually runs them. If I save a plan with -out and apply that file, I'm running exactly what we reviewed — not a surprise second plan.
terraform plan -out tf.plan terraform show tf.plan terraform apply tf.plan
Why use a remote backend with locking?Intermediate
Because local state doesn't scale for a team. Remote state is one source of truth, and locking — DynamoDB, Terraform Cloud, whatever — stops two applies from writing at once and corrupting state.
terraform {
backend "s3" {
bucket = "tf-state"
key = "prod/network.tfstate"
region = "us-east-1"
dynamodb_table = "tf-lock"
encrypt = true
}
}What does terraform init actually do?Beginner
Init installs providers and modules, configures the backend, and writes the lockfile. I re-run it after backend or provider changes. It doesn't change infrastructure by itself.
terraform init terraform init -upgrade # bump within constraints # commit .terraform.lock.hcl
Your teammate applied while you were mid-plan and the lock failed. What do you do?Advanced
I'd wait for their apply to finish and release the lock, then re-plan. I don't force-unlock unless I've verified nothing else is still holding it — unlocking over a live apply is how you corrupt state.
Backends stash a lock ID in DynamoDB or similar. force-unlock is for abandoned locks after a crashed CI job, not for racing a living apply. After an unlock I'd always re-run plan so I'm looking at current state. In practice I check who holds the lock in CI logs or the lock table, confirm that process is dead, unlock with the ID, re-init if needed, then plan again. Shorter apply scopes and one pipeline per state key keep contention rare.
terraform force-unlock <LOCK_ID> # only after confirming no other terraform process holds it terraform plan -out tf.plan
Interviewer often follows with: How would you split state keys so two teams aren't fighting the same lock?
How does remote state data source work across stacks?Intermediate
One stack writes outputs into its state; another reads them with terraform_remote_state — or better, a published module contract. I'd rather consume explicit outputs than dig through another team's full state file.
data "terraform_remote_state" "net" {
backend = "s3"
config = {
bucket = "tf-state"
key = "prod/network.tfstate"
region = "us-east-1"
}
}
# data.terraform_remote_state.net.outputs.vpc_idWhat does terraform refresh / refresh-only do?Intermediate
It reconciles state with what's actually in the cloud without applying config changes. plan -refresh-only is how I inspect drift; apply -refresh-only accepts reality into state when that's intentional.
terraform plan -refresh-only terraform apply -refresh-only
terraform fmt and validate — what do they catch?Beginner
fmt rewrites HCL to the canonical style; validate checks syntax and internal consistency — references, types — without talking to the cloud. I run both in CI before plan.
terraform fmt -check -recursive terraform init -backend=false terraform validate
What is a Terraform module?Beginner
It's a reusable package of resources with inputs and outputs. The root module calls child modules so common patterns — VPC, RDS, IAM — stay DRY and versioned instead of copy-pasted.
module "vpc" {
source = "./modules/vpc"
cidr = "10.0.0.0/16"
}
# module.vpc.subnet_idsvariables, locals, and outputs — how do they differ?Beginner
I'd break it down like this: variables are what callers pass in, locals are named expressions computed once inside the module, outputs are what you expose back out or to remote state. Mark secrets sensitive so they get redacted in the UI — they still land in state though.
variable "env" { type = string }
locals { name = "${var.env}-api" }
output "url" { value = aws_lb.web.dns_name }count vs for_each — when do you choose which?Intermediate
The short version: count makes N indexed copies; for_each keys instances by a map or set. I prefer for_each whenever identity matters — yanking one item from a count list reshuffles indexes and forces unwanted recreates.
count addresses look like aws_instance.web[1]. Drop index 0 and everything after renumbers, so Terraform plans destroy/create for survivors that didn't really change. for_each keeps stable keys via each.key, so add/remove only touches that instance. I'd only use count for identical disposable copies where a reshuffle is fine.
resource "aws_iam_user" "u" {
for_each = toset(["alice", "bob"])
name = each.key
}Data sources vs resources?Beginner
A resource creates and manages something; a data source only reads existing infra or provider data at plan time. Data sources never create or destroy.
data "aws_ami" "ubuntu" {
most_recent = true
owners = ["099720109477"]
}
# data.aws_ami.ubuntu.idWorkspaces vs separate directories/state — how do you choose?Advanced
Workspaces switch state inside one backend namespace; separate directories — or Terragrunt stacks — with different backend keys isolate blast radius. For prod vs non-prod I'd rather have separate state keys than lean on workspaces alone.
Workspaces are handy for identical configs with different state, but they share the same code and backend config, which tempts applying to the wrong workspace and couples blast radius. Directory-per-env with distinct backend keys makes the environment obvious in the path, lets IAM differ per key, and matches how most teams run CI — one pipeline per stack. I use workspaces sparingly for true clones, not as the primary multi-env strategy.
# envs/prod/network — key = "prod/network.tfstate" # envs/dev/network — key = "dev/network.tfstate" terraform workspace list # fine for soft clones, not hard isolation
Interviewer often follows with: What happens if someone runs terraform workspace select prod on a laptop with broad credentials?
How do dynamic blocks and for expressions help?Intermediate
for expressions transform collections; dynamic blocks generate repeated nested blocks — ingress rules, tags — from a list so the module stays DRY when the length varies.
dynamic "ingress" {
for_each = var.ports
content {
from_port = ingress.value
to_port = ingress.value
protocol = "tcp"
cidr_blocks = ["10.0.0.0/8"]
}
}Why avoid provisioners in production modules?Intermediate
Provisioners run imperative scripts Terraform can't really model in the plan or track in state, so they break idempotency and drift detection. I'd rather bake an AMI, use cloud-init, or hand off to config management.
local-exec and remote-exec mostly fire on create/destroy, hide side effects from the dependency graph, and fail differently across CI runners. In an interview I say: bake AMIs, use user_data/cloud-init, or Ansible — don't make SSH-from-Terraform the default path.
You need to publish an internal VPC module for three teams. How do you version and break changes safely?Expert
I'd publish to a private registry or git tags with semver, pin callers to something like ~> major.minor, and keep a changelog. Breaking input renames get a deprecation window or a new major so consumers upgrade on purpose.
Without versioning, every consumer floats on main and breaks together. Registry modules — or git ref=v1.4.2 — plus required_version and required_providers give reproducible installs. Contract-wise I want stable outputs, documented variables, examples/, and CI that runs terraform test or plan against a fixture. For a breaking change I'd bump major, keep a v1 branch for a while, and migrate stacks via PR. I never point production at latest git main as the module source.
module "vpc" {
source = "app.terraform.io/acme/vpc/aws"
version = "~> 2.3"
}Interviewer often follows with: How do you test a module before you publish a new minor?
What does lifecycle create_before_destroy do?Intermediate
It creates the replacement before destroying the old one, so you cut downtime for things like instances behind a load balancer. prevent_destroy and ignore_changes are the other lifecycle knobs I'd reach for.
resource "aws_instance" "web" {
# ...
lifecycle {
create_before_destroy = true
ignore_changes = [tags["LastTouched"]]
}
}What is drift and how do you detect it?Intermediate
Drift is when the real cloud diverges from state — usually someone clicked in the console. terraform plan or plan -refresh-only shows it; then I either revert the change or codify it, and tighten IAM so it doesn't keep happening.
terraform plan -refresh-only terraform apply -refresh-only # accept reality into state
How do you import existing infrastructure?Intermediate
I write the resource block to match reality, then terraform import — or an import block — to bind the real ID into state. A follow-up plan should be a no-op if I got it right.
import {
to = aws_s3_bucket.logs
id = "my-logs-bucket"
}
# terraform plan # expect no changesYou refactored a resource into a module and plan wants to destroy/recreate. How do you fix it without downtime?Advanced
I'd add a moved block — or run state mv — so the address updates in state without touching the cloud object. After that, plan should show a move, not destroy/create.
Terraform matches by address. Renaming aws_instance.web to module.app.aws_instance.web looks like delete-old plus create-new unless you tell it they're the same object. moved blocks are reviewable in PRs and work in CI; state mv is the CLI equivalent for one-offs. After the move I verify with plan that only the address changed. Pair with import when you're bringing brownfield into a module for the first time.
moved {
from = aws_instance.web
to = module.app.aws_instance.web
}
terraform plan # should show move, not replaceInterviewer often follows with: When would you reach for removed blocks instead?
When is -target acceptable, and why is it a smell?Advanced
I'd only use it as an incident escape hatch to apply one resource. Routine -target applies a partial graph and can leave dependent state inconsistent — usually a sign the modules are too coupled.
-target still refreshes and plans the dependency closure of the target, but it skips siblings, so outputs and cross-resource contracts can drift from what a full apply would do. I prefer smaller state files per component so a normal plan is reviewable. If I must target in an incident, I follow with a full plan in the same change window to confirm nothing else is pending.
terraform apply -replace=aws_instance.web # prefer -replace over legacy terraform taint # avoid: terraform apply -target=... in routine CI
Interviewer often follows with: How would you split a 2k-resource state so targeting isn't needed?
How does terraform plan compute the dependency graph?Expert
The short version: it parses config, builds a graph from references — and depends_on — refreshes current state, then walks that graph to emit create/update/replace/destroy. Parallelism and order come from the graph, not file order.
Implicit edges come from interpolating attributes, like subnet_id = aws_subnet.a.id. Explicit depends_on is for hidden side effects. Cycles fail at plan. Replace vs update depends on ForceNew attributes in the provider schema. That's why count index reshuffles recreate, why outputs wait on resources, and why -target is dangerous. terraform graph can visualize it; in interviews I walk parse → provider schemas → graph → refresh → diff → plan file.
terraform graph | dot -Tpng > graph.png terraform plan -out tf.plan terraform show -json tf.plan | jq '.resource_changes[].change.actions'
Interviewer often follows with: How do provider ForceNew attributes show up in a plan?
How do you force-replace a wedged resource when config did not change?Intermediate
terraform apply -replace=ADDRESS recreates that resource — the modern replacement for taint. I'd use it sparingly and be clear on destroy order and data loss first.
terraform apply -replace=aws_instance.web
How do you structure state for many environments and teams?Advanced
I split by blast radius: per environment and per component — network, data, app — each with its own backend key. Share logic via versioned modules; cross-stack links via outputs or remote state.
One giant state means one lock, one blast radius, and multi-hour plans. Split keys so a bad app apply can't destroy the VPC. I layer network → platform → app with clear ownership, map each directory to a CI pipeline, and document which stack may read whose remote state so we don't create secret circular dependencies.
# prod/network.tfstate # prod/data.tfstate # prod/app.tfstate
Interviewer often follows with: How do you handle a shared platform stack that many apps depend on?
Secrets end up in Terraform state — how do you protect them?Advanced
I treat state as sensitive: encrypt the backend, restrict IAM, never commit state or secret tfvars, mark variables sensitive, and prefer injecting secrets from Vault or SSM at apply time. sensitive = true only redacts the UI — the values still live in state.
Any random_password, provider-returned secret, or variable still lands in state JSON in plaintext or lightly obfuscated. Controls I'd put in place: S3 encryption plus bucket policy, lock table locked down, CI OIDC roles with least privilege, short-lived apply credentials, and avoid writing long-lived DB passwords into managed resources when dynamic secrets exist. Rotate anything that leaked via a state dump or CI artifact.
variable "db_password" {
type = string
sensitive = true
}
# backend encrypt = true; IAM deny s3:GetObject except apply roleInterviewer often follows with: Would you store a Vault token in tfvars for the provider?
How do you run Terraform safely in CI/CD?Advanced
Plan on every PR and post it for review; apply only on merge to a protected branch from a trusted pipeline. I'd use OIDC for short-lived cloud creds, remote state with locking, and apply the saved plan artifact — not a fresh plan on main.
Anti-patterns I push back on: apply from laptops to prod, static AWS keys in CI, apply without the reviewed plan file. The pattern that works: fmt/validate/tflint/checkov on PR → plan -out → upload the plan artifact → policy on plan JSON → approval → apply that artifact on main. Branch protection and environment approvals stop fork PRs from applying. Pin providers via a committed lockfile.
terraform plan -out=tf.plan terraform show -json tf.plan > plan.json # on main after approval: terraform apply -input=false tf.plan
Interviewer often follows with: How do you stop a compromised PR from reading production state?
How should the AWS/Azure/GCP provider authenticate in CI?Expert
I'd prefer OIDC or workload identity: the pipeline gets a short-lived token scoped to repo and branch, exchanged for cloud credentials. No long-lived access keys sitting in the secret store.
GitHub Actions or GitLab OIDC into cloud STS with a trust policy conditioned on sub — repo — ref — branch — and optionally the workflow. IAM role allows only the APIs that stack needs; network stack ≠ app stack. Locally, developers use SSO profiles, not shared keys. Terraform Cloud/Enterprise can do dynamic provider credentials. Falling back to static keys means rotation pain, leak risk, and harder attribution in CloudTrail.
# CI sets AWS_ROLE_ARN + web identity token file
provider "aws" {
region = "us-east-1"
# credentials from env / OIDC — no static keys in code
}Interviewer often follows with: How do you scope the trust policy so only main can deploy prod?
How do you enforce policy on Terraform changes?Intermediate
I run static scanners like tfsec or checkov on the code, and OPA/Conftest or Sentinel on the plan JSON in CI, so public buckets and open security groups fail before merge.
terraform show -json tf.plan > plan.json conftest test plan.json -p policy/ checkov -f plan.json
How do you manage provider versions and the lockfile?Beginner
I constrain versions in required_providers and commit .terraform.lock.hcl so every laptop and CI run uses the same provider binaries and checksums. Bumps happen on purpose with init -upgrade.
terraform {
required_version = ">= 1.5"
required_providers {
aws = { source = "hashicorp/aws", version = "~> 5.0" }
}
}Production apply failed halfway. How do you recover?Expert
I'd read the error and the state — some resources may be created, others pending. Fix the config or cloud issue, re-plan, and apply again; Terraform continues from state. I don't manually delete objects without updating state.
Partial applies are normal under API rate limits or IAM errors. State should reflect what succeeded; the next plan shows remaining creates or tainted replacements. If the real object exists but state missed it — rare crash after create — I import. If state has an object that failed provisioning, I replace carefully. I never hand-edit state JSON. After recovery, a full plan should be empty. Post-incident I'd add retries, smaller blast radius, and pre-apply IAM checks.
terraform plan -out tf.plan terraform apply tf.plan # if object exists but missing from state: # terraform import aws_s3_bucket.logs my-logs-bucket
Interviewer often follows with: How do you practice disaster recovery for the state bucket itself?
What belongs in .gitignore for a Terraform repo?Beginner
I'd ignore .terraform/, crash logs, *.tfstate*, secret tfvars, and plan binaries. Commit the .tf files, modules, lockfile, and example tfvars without secrets.
.terraform/ *.tfstate *.tfstate.* crash.log *.tfplan secret.auto.tfvars
Someone committed a .tfvars with a database password. What do you do?Advanced
I'd rotate the password immediately, purge it from git history if it was pushed, revoke anything derived from it, add secret scanning to CI, and move secrets to a manager injected at apply time — deleting the file alone isn't enough.
Deleting the commit from main doesn't remove clones or CI caches. Rotate first. Use git filter-repo or BFG only with a coordinated force-push policy, then invalidate leaked credentials. Prevention: pre-commit scanners like gitleaks, blocked patterns, and ephemeral secrets from Vault or SSM. sensitive variables still land in state — treat state access like secret access.
# 1) rotate the DB password in the engine # 2) gitleaks detect --source . --verbose # 3) remove *.auto.tfvars secrets from repo; use CI OIDC + Vault
Interviewer often follows with: Does git rm of the tfvars file remove it from history?
What is the difference between terraform destroy and state rm?Intermediate
destroy deletes real infrastructure and removes it from state. state rm only forgets the object in state — the cloud resource keeps running until you delete it another way.
terraform state rm aws_instance.legacy # instance still exists in AWS — manage or delete out of band
How do you pass different values per environment without duplicating modules?Beginner
Keep one module; pass variables via tfvars files, CI env, or env-specific auto.tfvars. I'd rather do directory-per-env with shared modules than copy-paste resource blocks.
terraform plan -var-file=env/prod.tfvars -out tf.plan # env/prod.tfvars: instance_type = "m5.large"
A plan shows replace on an RDS instance because of a ForceNew attribute you did not expect. How do you proceed?Advanced
I'd stop, verify which attribute forces new, check for data loss and outage, and either stage a blue/green migration, use create_before_destroy where it's safe, or revert the config — I never apply a surprise database replace on prod.
Provider schemas mark ForceNew on things like engine version jumps, AZ, or storage encryption toggles. Plan shows -/+ replace. For stateful data stores, replace usually means destroy then create unless create_before_destroy and a cutover plan exist. I'd terraform show the plan JSON for that resource, read the provider docs, take a snapshot, then decide migrate vs undo. Prefer in-place updates or a separate blue instance plus cutover. Document in the PR why replace is acceptable. Unexpected replace on data is an incident gate, not a green apply.
terraform show -json tf.plan | jq '.resource_changes[]
| select(.address|test("aws_db_instance"))
| {address, actions:.change.actions, before:.change.before, after:.change.after}'
# if ForceNew surprise: revert PR or design blue/green cutoverInterviewer often follows with: Which RDS changes are in-place versus ForceNew in the AWS provider?
You must migrate 800 resources from one Terraform state file into three component states with zero downtime. Outline the approach.Expert
I'd invent new state keys, use state mv or moved blocks in coordinated PRs, keep one writer at a time, and prove empty plans on both old and new stacks before deleting the monolith key — never copy state JSON by hand mid-apply.
Split by blast radius — network, data, app. Sequence I'd follow: freeze applies on the monolith, create empty backends for the new keys, state pull the monolith, state mv resources into the new state, remove resources from old config via removed/moved, plan both sides until there are no unexpected destroys, then cut CI over to per-component pipelines. Alternatives like terraformer/import help when addresses never matched. Risks are dual ownership, orphaned resources, and lock contention. Locking strategy, a monolith snapshot for rollback, and proving no destroy of prod data are the expert bar.
terraform state mv -state=old.tfstate -state-out=network.tfstate aws_vpc.main aws_vpc.main # then commit config that only the network stack owns aws_vpc.main
Interviewer often follows with: How do you handle outputs that other stacks still read via remote state during the split?
Two regions must stay in sync for DR, but Terraform only manages the primary today. How do you design multi-region IaC without double-destroy risk?Expert
I'd give each region its own state key and module instantiation, encode promotion and failover outside a single apply, and never point one state at resources in both regions with overlapping lifecycle.
Patterns that work: identical module called twice with provider aliases and separate state or roots; primary Terraform plus DR bootstrapped from AMIs/snapshots with a runbook; or active-active with explicit data replication resources. Avoid one resource address that can destroy both. Prefer region-coded backend keys. Failover is usually DNS/traffic plus data replication health — not terraform destroy on primary. Test DR in non-prod regularly. Interviewers want isolation of state and an honest boundary between IaC and data-plane DR.
provider "aws" { alias = "primary" region = "us-east-1" }
provider "aws" { alias = "dr" region = "us-west-2" }
module "vpc_primary" { source = "./modules/vpc" providers = { aws = aws.primary } }
module "vpc_dr" { source = "./modules/vpc" providers = { aws = aws.dr } }Interviewer often follows with: Would you put Route53 failover records in the primary state or a global DNS stack?
Checkov/OPA passes on the plan, but a security review finds an S3 bucket became public after apply. What did policy miss, and how do you harden the gate?Expert
I'd treat it as a control gap: either the plan JSON didn't include the public ACL/policy change, the rule was too narrow, or we were missing a post-apply AWS guardrail like account Block Public Access — then fix detection and add a preventive account control.
Plan-time policy sees intended Terraform changes, not console clicks or a separate stack. Common gaps: resource created private then a public_access_block omitted; fixture used count=0; rule checked ACL but not policy JSON; apply used a different var-file than CI. Hardening: test policies against real plan fixtures that include public ACL and policy docs, require aws_s3_bucket_public_access_block, enable account-level BPA, and add a post-apply control that pages on Public. Defense in depth beyond a green Conftest run.
# conftest: deny if change after_* has ACL public-read or Policy Principal=* # plus account BPA: # aws s3control put-public-access-block --account-id ... --public-access-block-configuration ...
Interviewer often follows with: How do you regression-test the OPA policy so this class of miss can't come back?
Provider upgrade from major 4→5 rewrites dozens of resources and plans destroys. How do you roll the upgrade safely across environments?Expert
I'd pin and upgrade in a dedicated branch, read the upgrade guide, run plans in dev first, use moved blocks or intermediate module versions, and only promote when the plan noise is understood — never big-bang prod on a Friday.
Major provider bumps often rename resources — think the aws_s3_bucket_* split — or change ForceNew. My path: changelog and upgrade guide → update required_providers → init -upgrade in a sandbox → plan and triage every destroy → apply moved blocks or state replace-provider if needed → terraform test / CI against fixtures → promote env-by-env. Keep the lockfile as the promotion artifact. If destroys are unavoidable, schedule maintenance and backups. Rollback is revert lockfile plus provider constraint and re-init. Treat provider majors like app majors with canaries.
# required_providers aws = { version = "~> 5.0" }
terraform init -upgrade
terraform providers lock -platform=linux_amd64
terraform plan -out tf.plan # triage every destroyInterviewer often follows with: How do you keep app teams unblocked while a platform provider upgrade is in flight?
Atlantis/PR automation applied the wrong workspace to production. How do you prevent recurrence and recover now?Expert
I'd halt automation, assess blast radius from state and plan logs, restore from state history or rebuild, then fix workspace/backend selection so prod applies can't be selected by a mislabeled PR comment.
Usual root causes: shared backend with workspace select driven by free text, missing environment protection, or directory allowlists that are too broad. Recovery: figure out what changed from Atlantis logs and state versions, state pull a previous version if versioning is on, or import/recreate. Prevention: one backend key per env — no workspace roulette — Atlantis project bindings by path, CODEOWNERS plus approvals for prod paths, OIDC roles that can't assume prod from non-prod dirs, and plan-only on forks. I want both incident recovery and structural elimination of workspace ambiguity.
# atlantis.yaml — explicit dir → workspace/backend
projects:
- name: prod-network
dir: envs/prod/network
autoplan: { when_modified: ["**/*.tf"] }
# IAM: role only assumable when claim path matches prodInterviewer often follows with: Why is directory-per-env safer than terraform workspace select in chatops?
CI job died mid-apply; every plan now fails with "Error acquiring the state lock". What do you do?Advanced
I'd confirm no Terraform process still holds the lock, then force-unlock with the ID and immediately re-plan. I never unlock while another apply might still be writing.
Stale locks are common after OOM-killed runners or cancelled pipelines. Check the lock table or TFC UI for LockID and who created it, inspect CI logs for a still-running job, only then force-unlock. After unlock, plan to detect a partial apply — resources created but not in state, or vice versa. Prevention: shorter applies, one writer per state key, runner shutdown hooks that cancel cleanly, and alerts on lock age. Unlock is an incident step with verification, not a reflex.
terraform force-unlock <LOCK_ID> terraform plan -out tf.plan # if partial apply: state list vs cloud inventory; import or rm as needed
Interviewer often follows with: What evidence proves the lock holder is dead before force-unlock?
Security finds plaintext database passwords inside the remote state JSON in S3. How do you contain and redesign?Expert
Rotate every exposed secret, restrict state ACL and KMS, purge old state versions, and stop putting secret values in attributes Terraform must store — use secret references instead.
State stores resource attributes, including sensitive ones, in JSON. Even with sensitive = true on outputs, state and plan files can still hold the value. Containment: rotate DB passwords and tokens, enable bucket encryption plus IAM least privilege, delete historical state object versions that contain the secret, and audit who could read the bucket. Redesign: write secrets to Vault or AWS SM via a separate process, pass ARNs or names into Terraform, or use write-only/ephemeral arguments where the provider supports them. Never commit state; never share plan files with secrets in chat.
# rotate the DB password in the provider/console first # then remove password from .tf — reference aws_secretsmanager_secret by ARN # scrub: delete prior s3://tf-state/... object versions after confirming backups
Interviewer often follows with: Does marking an output sensitive=true remove the value from state?
An engineer applied with the wrong backend key and destroyed staging resources that shared names with prod patterns. How do you recover?Expert
Halt all applies, reconstruct what was destroyed from state history and cloud audit trails, restore data from backups, then hard-separate backend keys and IAM so prod credentials can't touch staging paths — and vice versa.
Wrong-env destroy is usually workspace/backend mis-selection or a copied -var-file. Recovery depends on resource type: re-apply from last known good state if resources still exist; restore DB snapshots for data; import after manual recreate when needed. Forensics: S3 state versions, CloudTrail, CI job parameters. Prevention: directory-per-env, OIDC role binding by path, protected environments, plan artifacts that show the backend key, and deny destroy on prod without break-glass. Don't rely on resource name prefixes alone.
terraform state pull > /tmp/now.json aws s3api list-object-versions --bucket tf-state --prefix envs/staging/ # restore prior version to a file, compare, import/recreate carefully
Interviewer often follows with: Why is IAM path-scoped OIDC stronger than telling engineers to double-check -var-file?
Someone toggled security-group rules in the AWS console; the next Terraform plan wants to undo them. How do you handle the drift?Advanced
I'd treat console edits as unauthorized drift unless there was an incident break-glass. Either revert in AWS by applying Terraform, or intentionally import the change into code — never leave two sources of truth.
I detect drift with plan -refresh-only or regular CI plans. If the console change was emergency access, codify it in a PR quickly or schedule the revert. If it was shadow IT, apply to restore the baseline and page the actor. I avoid lifecycle ignore_changes on security groups unless there's a documented dual-control model — it hides future drift. Longer term: SCPs or Config rules that detect manual SG changes, and make Terraform the only writer via IAM permissions boundaries.
terraform plan -refresh-only terraform plan -out tf.plan # will show SG rule churn # either: apply to enforce code, or PR the console rules into .tf then apply
Interviewer often follows with: When is ignore_changes on an SG rule a reasonable exception?
A module source `version = "~> 3.2"` silently pulled 3.5 into prod CI and the plan shows ForceNew replaces. What went wrong?Advanced
The pessimistic constraint allowed a new minor that changed ForceNew behavior. I'd pin exact module versions — or commit registry checksums — and promote module bumps through environments on purpose.
~> 3.2 allows 3.2.0 through <4.0.0, so a 3.5 module can rewrite resources. .terraform.lock.hcl locks providers, not remote module versions — module pins must be exact in the calling module or managed by a private registry promotion channel. Response: revert the module version pin, re-init, confirm a safe plan, then land 3.5 via a deliberate PR with changelog review. Policy: no floating module versions on prod roots; Dependabot PRs for bumps.
module "vpc" {
source = "app.terraform.io/org/vpc/aws"
version = "3.2.4" # exact — not ~> 3.2
}
terraform init
terraform planInterviewer often follows with: What does the dependency lockfile guarantee for modules vs providers?
You remove an item from the middle of a `count`-based list and Terraform plans to destroy/recreate nearly every instance. How do you explain and fix it?Expert
count indexes shift — resource N becomes N-1 in state addressing, so Terraform sees replaces. I'd migrate to for_each with stable keys, using moved blocks to preserve the real objects.
count addresses resources as foo[0], foo[1], …. Delete index 0 and everything renumbers. for_each with map or set keys — name, AZ, subnet id — keeps addresses stable when one member leaves. Migration: add for_each config, use moved blocks from count indexes to keys, plan until no destroys, then remove count. Hotfix without moved: state mv carefully one by one. Never use count for non-identical sets that shrink or grow in the middle.
moved {
from = aws_instance.web[0]
to = aws_instance.web["a"]
}
# resource "aws_instance" "web" { for_each = toset(["a","b"]) ... }Interviewer often follows with: Why is for_each = toset(var.names) safer than count = length(var.names)?
Import of a brownfield VPC was interrupted; state has half the subnets and the next plan wants to create duplicates. How do you finish safely?Advanced
I'd inventory cloud vs state, import the missing addresses, and only apply when plan shows zero unwanted creates — never apply "to fix" while duplicates are still pending.
Partial import leaves Terraform thinking un-imported objects should be created, which fails on uniqueness or creates shadow resources. I'd terraform state list, compare to AWS CLI inventory, import each missing address with the correct ID, and plan until clean. For large trees, generate import blocks and apply imports without creating. If a create already leaked, delete the duplicate carefully or remove it from config. Lock the state during the repair window so nobody else applies.
terraform state list | grep subnet aws ec2 describe-subnets --filters Name=vpc-id,Values=vpc-… \ --query 'Subnets[].SubnetId' terraform import aws_subnet.a subnet-abc terraform plan # must show no subnet creates
Interviewer often follows with: What risk does applying a plan with create on an existing subnet ID collision create?
Engineer’s laptop plan is empty, but CI plan wants to replace an ALB. Same branch. Why can they disagree?Advanced
Different var-files, backend or workspace, provider versions, or credentials seeing different accounts. I'd diff init lockfiles, -var flags, and terraform version between environments until the plans match.
Common deltas: missing -var-file=prod.tfvars locally, wrong AWS profile or account, stale local state when CI uses remote, provider version drift without committing the lockfile, or CI using -refresh=false vs local refresh. Print terraform version, provider lock hashes, backend config, and the exact plan command in both places. Require CI to upload plan JSON as an artifact. The plan applied to prod must be the plan CI produced — never apply from a laptop against prod state.
terraform version terraform providers git status -- .terraform.lock.hcl # CI and local: same -var-file, same backend key, committed lockfile
Interviewer often follows with: Should prod apply ever use terraform apply without a saved plan file from CI?
After a backend migration to a new S3 key, state looks truncated and resources vanished from state list. How do you respond?Expert
Stop applies, restore the previous state object version, fix the migration procedure, and only then re-point the backend — never keep applying against an empty or partial state.
Migrations go wrong when people init -migrate-state against the wrong source, overwrite with local empty state, or copy JSON by hand. Recovery: S3 versioning or TFC state history to restore the last good blob, verify state list counts, then redo migration with pull → push or the official migrate path. Preflight checklist: resource count before and after, lock during cutover, backup object copied aside. Empty state plus apply is a mass create/destroy disaster.
aws s3api list-object-versions --bucket tf-state --prefix prod/app.tfstate # get-object prior VersionId to ./recovered.tfstate terraform state push ./recovered.tfstate # only after validation
Interviewer often follows with: What safeguard does state push need so you don't overwrite a newer good state?
lifecycle ignore_changes on ami hid that prod instances were months behind the golden image. How do you unwind that safely?Expert
I'd remove ignore_changes in a controlled PR, expect a replace or refresh plan, and roll instance refresh or blue-green so image updates become intentional again — not a surprise mass replace.
ignore_changes is a sharp tool: it silences drift security cares about — AMI, user_data, SG rules. Unwinding: decide the replace strategy — ASG instance refresh, blue/green, create_before_destroy — take backups, remove the ignore, plan, apply in non-prod first. Document why any remaining ignores exist. Prefer a Packer channel plus explicit var.ami_id bumps over permanent ignore. Trade day-2 console stability against patch compliance explicitly.
# remove: lifecycle { ignore_changes = [ami] }
terraform plan -out tf.plan
# prefer ASG instance_refresh / blue-green over in-place shockInterviewer often follows with: What's a safer pattern than ignore_changes when ops sometimes rebakes AMIs out of band?
Changing a for_each key from name to id plans to destroy and recreate every member. How do you rename keys without downtime?Advanced
Use moved blocks — or state mv — from old addresses to new ones so Terraform rewires state without touching real resources, then apply the no-op plan.
for_each keys are part of the resource address. Renaming keys looks like delete old plus create new. Terraform 1.1+ moved blocks express the rename in config; older versions use state mv. Always plan and confirm actions are empty or update-only before apply. Do this in a dedicated PR with no other churn. For modules, moved can reference module addresses too.
moved {
from = aws_subnet.this["public-a"]
to = aws_subnet.this["subnet-abc123"]
}
terraform plan # expect no destroys if mapping is completeInterviewer often follows with: Can you move addresses across different resource types with moved blocks?
Two operators force-unlocked the same state within a minute and both applied. State and cloud now disagree. How do you reconstruct truth?Expert
I'd freeze writers, treat cloud as reality for inventory, rebuild state with import and refresh from a backup point, and fix process so force-unlock needs two-person approval.
Split-brain after dual apply is the nightmare locking exists to prevent. Steps: disable CI apply, snapshot current cloud inventory, recover last pre-incident state version, plan -refresh-only to see divergence, then import missing resources, state rm ghosts, and targeted applies to converge. Don't keep applying hoping it heals. Process controls: break-glass unlock runbook, lock-age alerts, one pipeline per key, and preferably TFC/Enterprise run queuing. You should be able to narrate forensic order without guessing.
terraform state pull > /tmp/suspect.json # compare aws inventory vs state list; import orphans; state rm ghosts terraform plan -refresh-only # re-enable CI only when plan is understood
Interviewer often follows with: Why is cloud inventory usually safer than state file wins after a dual apply?