count, for_each & dynamic blocks

Build many resources from data.

Three subnets. One firewall rule per open port. One audit bucket per team. Real infrastructure repeats itself, and copy-pasting a resource block five times is how a configuration starts drifting away from what you meant it to say. Terraform gives you two ways to stamp out many resources from one block, and a third way to repeat blocks inside a single resource. Pick the wrong one and nothing fails loudly. It quietly destroys things you meant to keep, which matters most when the thing being kept is your audit log.

Two mailrooms hold the same letters. One is a wall of numbered lockers: 0, 1, 2. The other is a wall of pigeonholes with a name card taped under each slot: alice, bob, carol. The day bob leaves the company, the numbered wall shuffles. Everything behind his locker slides down one, so carol's mail now sits in what used to be bob's slot, and the last locker sits empty. The named wall does not shuffle. You peel off one card and empty one slot. count is the numbered wall. for_each is the named pigeonholes. Everything else in this lesson follows from that one difference.

Two Meta-Arguments and One Ledger

Most of what you write inside a resource block is a spec for the thing being built: this much memory, that disk image, this network. A meta-argument is different. It is an instruction to the builder rather than a line on the blueprint, so Terraform itself understands it on any resource, no matter which provider (the plugin that knows how to talk to AWS, Amazon Web Services, or to Azure, or to whatever you are managing) supplied that resource. count and for_each are the two meta-arguments that answer the question of how many.

# count: N copies, addressed by position 0..N-1
resource "aws_instance" "worker" {
  count         = 3
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  tags          = { Name = "worker-${count.index}" }   # worker-0, worker-1, worker-2
}
 
# for_each: one instance per map or set entry, addressed by key
resource "aws_iam_user" "team" {
  for_each = toset(["alice", "bob", "carol"])
  name     = each.key      # for a set, each.key and each.value are the same string
}

Inside a count block you get count.index, the position, counting from zero. Inside a for_each block you get each.key and each.value. Hand for_each a map and the key is the map key with its value sitting underneath. Hand it a set of strings and key and value are that same string, which is why the second block reads name = each.key. Two bits of jargon in that snippet, both worth unpacking now: IAM is Identity and Access Management, Amazon's users-and-permissions service, and an AMI (Amazon Machine Image) is the disk template a virtual machine boots from. Notice also that for_each refuses a plain list outright. The refusal is the point. A list has an order, a set does not, and order is precisely what for_each will not let you depend on, so lists get wrapped in toset() before they go anywhere near it.

terraform state list

aws_iam_user.team["alice"]
aws_iam_user.team["bob"]
aws_iam_user.team["carol"]
aws_instance.worker[0]
aws_instance.worker[1]
aws_instance.worker[2]

State is Terraform's ledger: one JSON (JavaScript Object Notation, a plain-text format for structured data) file listing every real object Terraform believes it created and what that object looked like last time it checked. The strings you just listed are resource addresses, and they are the closest thing Terraform has to a primary key. Brackets around a number mean "whatever happened to be third in a list". Brackets around a quoted string mean "the one called carol". One of those survives an edit to the collection. The other does not.

What Renumbering Costs You

Say every team gets its own bucket in S3 (Simple Storage Service, Amazon's object store) to hold their access logs. One bucket per team, built with count.

variable "teams" {
  type    = list(string)
  default = ["alice", "bob", "carol"]
}
 
resource "aws_s3_bucket" "audit" {
  count  = length(var.teams)
  bucket = "acme-audit-${var.teams[count.index]}"
}

bob leaves. You delete one string from the middle of that list. It looks like the smallest possible change, so you run a plan before touching anything.

terraform plan

aws_s3_bucket.audit[0]: Refreshing state... [id=acme-audit-alice]
aws_s3_bucket.audit[1]: Refreshing state... [id=acme-audit-bob]
aws_s3_bucket.audit[2]: Refreshing state... [id=acme-audit-carol]
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy
-/+ destroy and then create replacement
 
Terraform will perform the following actions:
 
  # aws_s3_bucket.audit[1] must be replaced
-/+ resource "aws_s3_bucket" "audit" {
      ~ arn                = "arn:aws:s3:::acme-audit-bob" -> (known after apply)
      ~ bucket             = "acme-audit-bob" -> "acme-audit-carol" # forces replacement
      ~ bucket_domain_name = "acme-audit-bob.s3.amazonaws.com" -> (known after apply)
      ~ id                 = "acme-audit-bob" -> (known after apply)
        # (9 unchanged attributes hidden)
    }
 
  # aws_s3_bucket.audit[2] will be destroyed
  # (because index [2] is out of range for count)
  - resource "aws_s3_bucket" "audit" {
      - arn    = "arn:aws:s3:::acme-audit-carol" -> null
      - bucket = "acme-audit-carol" -> null
      - id     = "acme-audit-carol" -> null
        # (10 unchanged attributes hidden)
    }
 
Plan: 1 to add, 0 to change, 2 to destroy.

Read the last line first. One created, two destroyed, and all you did was remove a single team. Index 1 used to resolve to bob and now resolves to carol, so Terraform sees the bucket name changing underneath an address it already tracks. A bucket name cannot be edited in place, so it plans a replacement. Index 2 has nothing left to resolve to, so it goes. Carol never appeared in your diff, and carol's bucket is the one being destroyed and rebuilt empty. If those buckets hold the logs you would reach for during an investigation, count has turned a personnel change into evidence loss. In practice the apply often dies partway through, because Terraform will not delete a bucket that still has objects in it unless you set force_destroy = true, leaving you with a half-applied state and a manual cleanup. That is the luckier outcome.

resource "aws_s3_bucket" "audit" {
  for_each = toset(var.teams)          # keyed by team name, not by position
  bucket   = "acme-audit-${each.key}"
}

terraform plan

aws_s3_bucket.audit["alice"]: Refreshing state... [id=acme-audit-alice]
aws_s3_bucket.audit["bob"]: Refreshing state... [id=acme-audit-bob]
aws_s3_bucket.audit["carol"]: Refreshing state... [id=acme-audit-carol]
 
Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  - destroy
 
Terraform will perform the following actions:
 
  # aws_s3_bucket.audit["bob"] will be destroyed
  # (because key ["bob"] is not in for_each map)
  - resource "aws_s3_bucket" "audit" {
      - arn    = "arn:aws:s3:::acme-audit-bob" -> null
      - bucket = "acme-audit-bob" -> null
      - id     = "acme-audit-bob" -> null
        # (10 unchanged attributes hidden)
    }
 
Plan: 0 to add, 0 to change, 1 to destroy.

Same edit, same variable, one destroy. The key "bob" left the set, so the object at that address goes. alice and carol were never addressed by position, so nothing about them moved. The rule to carry around: key by something that means something stable in the real world, a team name, an account ID, a hostname, a port label, and the plan will only ever touch what you actually changed. Both meta-arguments work on module blocks too, and have since Terraform 0.13, so a module with for_each over your teams produces addresses like module.team["payments"].aws_s3_bucket.audit. The same identity guarantee then covers an entire stack per team rather than one resource.

Where count Still Earns Its Place

count still has jobs nothing else does better. The clearest is a switch: build this thing, or do not.

variable "enable_bastion" {
  type    = bool
  default = false
}
 
resource "aws_instance" "bastion" {
  count         = var.enable_bastion ? 1 : 0
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"
  subnet_id     = var.public_subnet_id
}

Zero or one, and the index never carries meaning, so nothing can shuffle. A bastion host is the standard case: the single hardened jump box you log into over SSH (Secure Shell, the encrypted remote-login protocol) before you can reach anything private. On in production, off in every sandbox. The trap sits on the reading side. When count is 0 the resource becomes an empty list, and asking for element zero of an empty list is an error, not a null.

echo 'aws_instance.bastion[0].id' | terraform console
echo 'one(aws_instance.bastion[*].id)' | terraform console
echo 'values(aws_s3_bucket.audit)[*].id' | terraform console

╷
│ Error: Invalid index
│
│   on <console-input> line 1:
│   (source code not available)
│
│ The given key does not identify an element in this collection value: the
│ collection has no elements.
╵
 
null
 
[
  "acme-audit-alice",
  "acme-audit-carol",
]

one() takes a collection of zero or one elements and hands back the element, or null when there is nothing there, which is exactly what you want in an output or a conditional. The [*] is a splat expression, shorthand for "this attribute, from every instance". Splat works directly on count resources because they are lists. for_each resources are maps, and splat on a map does not fail in a way that helps you: it reports that the object has no attribute named id and leaves you guessing. Run them through values() first, as in the third command.

for_each Keys Must Be Known Before Anything Is Built

You cannot print name cards for the pigeonholes if you will not learn the names until after the mail arrives. Terraform lives under the same constraint. Keys become addresses, addresses go into the plan, and the plan is written before a single API (Application Programming Interface, the request-and-response door a cloud opens to programs) call reaches your account. So the keys have to be knowable at plan time. The values hanging off them do not. Here is the version of that mistake that bites people: one flow log per private subnet, keyed by subnet ID. A VPC is a Virtual Private Cloud, your own walled-off network inside Amazon, and a VPC flow log is that network's record of which traffic was accepted and which was rejected, the thing you grep through during an intrusion investigation.

resource "aws_subnet" "private" {
  for_each   = var.private_cidrs   # map: "app" => "10.0.1.0/24", "db" => ..., "mgmt" => ...
  vpc_id     = aws_vpc.main.id
  cidr_block = each.value
}
 
resource "aws_flow_log" "subnet" {
  for_each        = toset([for s in aws_subnet.private : s.id])   # IDs do not exist yet
  subnet_id       = each.key
  traffic_type    = "ALL"
  log_destination = aws_cloudwatch_log_group.flow.arn
  iam_role_arn    = aws_iam_role.flow.arn
}

terraform plan

╷
│ Error: Invalid for_each argument
│
│   on main.tf line 24, in resource "aws_flow_log" "subnet":
│   24:   for_each        = toset([for s in aws_subnet.private : s.id])
│     ├────────────────
│     │ aws_subnet.private is object with 3 attributes
│
│ The "for_each" set includes values derived from resource attributes that
│ cannot be determined until apply, and so Terraform cannot determine the full
│ set of keys that will identify the instances of this resource.
│
│ When working with unknown values in for_each, it's better to use a map value
│ where the keys are defined statically in your configuration and where only
│ the values contain apply-time results.
│
│ Alternatively, you could use the -target planning option to first apply only
│ the resources that the for_each value depends on, and then apply a second
│ time to fully converge.
╵

The fix is spelled out in the message. Key the map with something you wrote yourself, and let the unknown ride along in the value slot. Because aws_subnet.private is itself a for_each resource, it is already a map whose keys you chose, so you can hand the whole resource to for_each and read the ID out of each.value.

resource "aws_flow_log" "subnet" {
  for_each        = aws_subnet.private   # keys are yours: "app", "db", "mgmt"
  subnet_id       = each.value.id        # unknown at plan time, and that is fine
  traffic_type    = "ALL"
  log_destination = aws_cloudwatch_log_group.flow.arn
  iam_role_arn    = aws_iam_role.flow.arn
}

Now the plan names aws_flow_log.subnet["db"] before that subnet exists anywhere but on paper. That property is worth more than it first looks. A reviewer can read the exact set of addresses a change will touch without having to trust the expressions that produced them. The -target escape hatch in the error message does work, and it belongs at the bottom of your list, because a two-step apply leaves your infrastructure in a state that no single plan ever described.

dynamic Blocks Repeat Things Inside One Resource

count and for_each multiply whole resources. Some resources instead carry repeatable blocks nested inside them: ingress rules in a security group, statements in a policy, listeners on a load balancer. You cannot hang for_each on a nested block, so Terraform gives you dynamic, a stamp that presses out copies of a nested block from a collection.

variable "ingress_rules" {
  type = map(object({
    port        = number
    cidr        = string
    description = string
  }))
  default = {}
}
 
resource "aws_security_group" "web" {
  name   = "web"
  vpc_id = var.vpc_id
 
  dynamic "ingress" {
    for_each = var.ingress_rules
    content {
      description = ingress.value.description
      from_port   = ingress.value.port
      to_port     = ingress.value.port
      protocol    = "tcp"
      cidr_blocks = [ingress.value.cidr]
    }
  }
}

The label after dynamic names the block being generated and doubles as the iterator, which is why you read ingress.key and ingress.value inside content. Add iterator = rule to rename it, something you will want the moment you nest one dynamic inside another. Those cidr fields hold CIDR (Classless Inter-Domain Routing) notation, an address followed by how many leading bits are fixed: 10.0.0.0/8 covers a private range, and 0.0.0.0/0 covers every address on the internet. An empty map produces zero ingress blocks, and an AWS security group with no inbound rules refuses all inbound traffic, so it fails closed. Know which way it fails, though. A variable that quietly resolves to {} costs you availability, not safety.

Now the operational cost, and the reason to reach for dynamic sparingly. The rules no longer live in the resource. They live in a values file (.tfvars, the file that carries variable values for one environment). Watch what a change to them looks like in review.

git diff --stat origin/main
git diff origin/main -- envs/prod/terraform.tfvars

 envs/prod/terraform.tfvars | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)
 
diff --git a/envs/prod/terraform.tfvars b/envs/prod/terraform.tfvars
index 3a1f9c2..b7e04d1 100644
--- a/envs/prod/terraform.tfvars
+++ b/envs/prod/terraform.tfvars
@@ -1,4 +1,4 @@
 ingress_rules = {
   https = { port = 443, cidr = "0.0.0.0/0",  description = "public https" }
-  ssh   = { port = 22,  cidr = "10.0.0.0/8", description = "vpn only" }
+  ssh   = { port = 22,  cidr = "0.0.0.0/0",  description = "vpn only" }
 }

One line, one file, and not a single .tf file touched. A reviewer skimming the Terraform code sees nothing at all. The description still reads "vpn only", because nobody updates a comment while opening a hole, and an attacker who has landed commit access certainly will not. The truth shows up in exactly one place, the plan. So read the plan with a machine instead of with your eyes.

terraform plan -out=tf.plan > /dev/null
terraform show -json tf.plan | jq -r '
  .resource_changes[]
  | select(.change.after != null and .type == "aws_security_group")
  | .address as $sg
  | .change.after.ingress[]?
  | select(any(.cidr_blocks[]?; . == "0.0.0.0/0"))
  | "\($sg): port \(.from_port) reachable from the whole internet (\(.description))"'

aws_security_group.web: port 22 reachable from the whole internet (vpn only)
aws_security_group.web: port 443 reachable from the whole internet (public https)

terraform show -json turns a saved plan into a documented, stable data structure, and jq (a command-line JSON query tool, installed with apt install jq on Debian and Ubuntu) reads it. Wire that check into CI (Continuous Integration, the automation that runs on every pull request) as a gate that fails the build, or feed the same JSON to a policy engine such as conftest, which runs Rego rules from the Open Policy Agent project. Either way a machine reads the plan, because humans skim values files.

There is also a structural fix. A dynamic block hides many rules behind one address, so the plan reports that aws_security_group.web changed and leaves you diffing a list of objects by eye. Since version 5 of the AWS provider, each rule can be its own resource, and then for_each gives every rule an address of its own.

resource "aws_vpc_security_group_ingress_rule" "web" {
  for_each          = var.ingress_rules
  security_group_id = aws_security_group.web.id
  description       = each.value.description
  ip_protocol       = "tcp"
  from_port         = each.value.port
  to_port           = each.value.port
  cidr_ipv4         = each.value.cidr
}

terraform plan

Terraform used the selected providers to generate the following execution
plan. Resource actions are indicated with the following symbols:
  ~ update in-place
 
Terraform will perform the following actions:
 
  # aws_vpc_security_group_ingress_rule.web["ssh"] will be updated in-place
  ~ resource "aws_vpc_security_group_ingress_rule" "web" {
      ~ cidr_ipv4   = "10.0.0.0/8" -> "0.0.0.0/0"
        description = "vpn only"
        id          = "sgr-0f3c9a1b2d4e5f607"
        # (7 unchanged attributes hidden)
    }
 
Plan: 0 to add, 1 to change, 0 to destroy.

The address itself now says ssh, and the diff is one line long. A rule per resource costs you a few more objects in state and buys you a plan that a tired reviewer can read correctly at 2am.

moved {
  from = aws_s3_bucket.audit[0]
  to   = aws_s3_bucket.audit["alice"]
}
moved {
  from = aws_s3_bucket.audit[1]
  to   = aws_s3_bucket.audit["bob"]
}
moved {
  from = aws_s3_bucket.audit[2]
  to   = aws_s3_bucket.audit["carol"]
}

terraform plan

Terraform will perform the following actions:
 
  # aws_s3_bucket.audit[0] has moved to aws_s3_bucket.audit["alice"]
    resource "aws_s3_bucket" "audit" {
        bucket = "acme-audit-alice"
        id     = "acme-audit-alice"
        # (11 unchanged attributes hidden)
    }
 
  # aws_s3_bucket.audit[1] has moved to aws_s3_bucket.audit["bob"]
    resource "aws_s3_bucket" "audit" {
        bucket = "acme-audit-bob"
        id     = "acme-audit-bob"
        # (11 unchanged attributes hidden)
    }
 
  # aws_s3_bucket.audit[2] has moved to aws_s3_bucket.audit["carol"]
    resource "aws_s3_bucket" "audit" {
        bucket = "acme-audit-carol"
        id     = "acme-audit-carol"
        # (11 unchanged attributes hidden)
    }
 
Plan: 0 to add, 0 to change, 0 to destroy.

Zero, zero, zero. That is what a pure refactor should always look like, and if your plan says anything else, the rename is wrong somewhere. Keep the moved blocks in the repository for at least one release so anyone applying from an older state catches up, then delete them. The other route skips the plan entirely. It edits shared state the instant you press enter, one address at a time, quoted so the shell does not eat the brackets.

terraform state pull > /tmp/audit-state-backup.json
terraform state mv 'aws_s3_bucket.audit[0]' 'aws_s3_bucket.audit["alice"]'
terraform state list

Move "aws_s3_bucket.audit[0]" to "aws_s3_bucket.audit[\"alice\"]"
Successfully moved 1 object(s).
 
aws_s3_bucket.audit[1]
aws_s3_bucket.audit[2]
aws_s3_bucket.audit["alice"]

Pull the backup first, every time. state mv writes to the shared remote state the moment you press enter, and no plan step stands between a typo and a state file that no longer describes reality. Look at the listing, too. state list sorts numeric keys ahead of quoted ones, so a half-finished migration reads exactly like that, and the two leftovers will not move themselves.

Before you merge any change to a for_each map, or to a values file feeding a dynamic block, save the plan and read two things: the number after "to destroy", and every address in the diff. An address you never edited showing up means you are looking at a reindex rather than a change, and applying it will take out something that had nothing to do with your commit.

Try this

Run terraform state list on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: converting count to for_each rewrites every address. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related