Cleanup & defer destroy

Never leak real resources.

Rent a car, leave it idling in the airport lot with the keys in the ignition, and two things happen while you are on the plane. The fuel burns. Anybody walking past can get in and drive away. A Terratest run that ends without tearing down what it built leaves exactly that behind: a real VPC (virtual private cloud, your own fenced-off network inside the provider), a real load balancer with a real public DNS name (domain name system, the internet's address book), real instances. All billing by the hour. All reachable from anywhere. Go's defer keyword is how you promise to hand the keys back before you ever turn the engine over.

The bill is the boring half of this. Test infrastructure is usually the least hardened thing a company owns. Security groups opened to 0.0.0.0/0 because it was faster than thinking, no patching, a throwaway password sitting in a tfvars file, nobody reading CloudTrail for that account. An orphaned test load balancer is an internet-facing endpoint with no owner, and owners are the reason endpoints ever get fixed. It will be found. The entire IPv4 address space gets port-scanned around the clock by researchers, security vendors and people who are neither. And if your test issued a TLS certificate for a real hostname along the way, that hostname is published within minutes in certificate transparency logs, which are public append-only ledgers every certificate authority is required to write to. People watch those feeds for exactly this. Cleanup is a security control. Treat a leaked resource the way you would treat a leaked credential.

Queue the teardown before you open the door

defer is a note you pin to the door on your way into a room. It says: before I leave, do this. Go collects those notes as the function runs and executes them when the function returns, newest note first. The instinct is to write code in reading order, apply near the top and cleanup at the bottom. That instinct leaks resources.

Here is the mechanism. terraform.InitAndApply checks the error coming back from Terraform and, on any failure, calls require.NoError, which calls t.FailNow(), which calls runtime.Goexit(). Your test body runs in its own goroutine (a goroutine is one of Go's cheap, lightweight threads), and Goexit unwinds that goroutine, running every deferred call that has already been registered. Already registered is the phrase that costs money. If your defer terraform.Destroy sits on the line below a failed apply, that line never executed, so no note was ever pinned to the door. Terraform got as far as the VPC, the subnets and the NAT gateway (network address translation gateway, the box that lets machines in private subnets reach the internet) before the instance failed to launch. All of it stays up. A NAT gateway runs about $0.045 an hour in us-east-1, roughly $33 a month, plus data processing charges on top. Nobody notices one of those.

package test
 
import (
	"strings"
	"testing"
	"time"
 
	http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
)
 
func TestVpcModule(t *testing.T) {
	runID := strings.ToLower(random.UniqueID()) // 6 chars, e.g. "8kq2rw"
 
	opts := &terraform.Options{
		TerraformDir: "../examples/vpc",
		Vars: map[string]interface{}{
			"name": "tt-" + runID,
			"cidr": "10.0.0.0/16",
			// Every resource carries the run id so a sweeper can find
			// orphans later. See the last section of this lesson.
			"tags": map[string]string{"terratest": "true", "run_id": runID},
		},
	}
 
	// Registered BEFORE apply. A failing apply calls t.FailNow, which
	// calls runtime.Goexit, which runs the defers already on the stack.
	// This one is on the stack. One written below apply would not be.
	defer terraform.Destroy(t, opts)
 
	terraform.InitAndApply(t, opts)
 
	url := "http://" + terraform.Output(t, opts, "alb_dns_name")
	http_helper.HttpGetWithRetry(t, url, nil, 200, "ok", 30, 5*time.Second)
}

terraform.Destroy shells out to terraform destroy -auto-approve -input=false inside TerraformDir, and it rebuilds the whole command line from the Options struct you hand it: every entry in Vars is re-rendered as a -var flag, the working directory comes from TerraformDir, the backend settings come from BackendConfig. That is why the destroy has to receive the same Options value the apply used. It has no memory of the apply and no way to look one up. Hand it a freshly built struct pointing somewhere else and it will report a perfectly clean destroy of nothing at all. Terratest also appends -lock=false to both commands, because Options.Lock defaults to false, which is why you see that flag in the logs below. A word on how to read those logs. The file:line at the start of each line points inside Terratest rather than inside your test, and everything Terraform itself prints funnels through one call in Terratest's logger package, which is why so many unrelated messages carry the same prefix. Those numbers are what v1.0.1 produces, and they move when you upgrade.

go test -v -timeout 30m ./test/ -run TestVpcModule

=== RUN   TestVpcModule
TestVpcModule 2026-07-22T09:31:12Z retry.go:159: terraform [init -upgrade=false]
TestVpcModule 2026-07-22T09:31:12Z logger.go:79: Running command terraform with args [init -upgrade=false]
TestVpcModule 2026-07-22T09:31:19Z logger.go:79: Terraform has been successfully initialized!
TestVpcModule 2026-07-22T09:31:19Z retry.go:159: terraform [apply -input=false -auto-approve -var name=tt-8kq2rw -var cidr=10.0.0.0/16 -var tags={"terratest" = "true", "run_id" = "8kq2rw"} -lock=false]
...
TestVpcModule 2026-07-22T09:33:47Z logger.go:79: Apply complete! Resources: 9 added, 0 changed, 0 destroyed.
TestVpcModule 2026-07-22T09:33:47Z logger.go:79: Outputs:
TestVpcModule 2026-07-22T09:33:47Z logger.go:79: alb_dns_name = "tt-8kq2rw-1573881204.us-east-1.elb.amazonaws.com"
TestVpcModule 2026-07-22T09:33:47Z http_helper.go:101: Making an HTTP GET call to URL http://tt-8kq2rw-1573881204.us-east-1.elb.amazonaws.com
...
    http_helper.go:284: 'HTTP GET to URL http://tt-8kq2rw-1573881204.us-east-1.elb.amazonaws.com' unsuccessful after 30 retries
TestVpcModule 2026-07-22T09:36:40Z retry.go:159: terraform [destroy -auto-approve -input=false -var name=tt-8kq2rw -var cidr=10.0.0.0/16 -var tags={"terratest" = "true", "run_id" = "8kq2rw"} -lock=false]
TestVpcModule 2026-07-22T09:38:55Z logger.go:79: Destroy complete! Resources: 9 destroyed.
--- FAIL: TestVpcModule (463.21s)
FAIL
FAIL	github.com/acme/infra/test	463.284s
FAIL

Read the tail of that log. The HTTP assertion failed, the test is red, and Destroy complete! Resources: 9 destroyed. still ran. That is the shape you want out of every failing run. A red test with a clean account is a normal Tuesday. A red test with a live account is an incident nobody has noticed yet.

Build in order, tear down in reverse

Deferred calls come off like a stack of plates: last one on, first one off. The jargon for that is LIFO (last in, first out). It happens to be exactly the ordering real infrastructure needs, and you get it for free. Register each teardown right after the thing it cleans up, in the order you built things, and the unwinding reverses your setup by itself. You never work out the order by hand, which is why it stays correct as the test grows.

func TestPlatformStack(t *testing.T) {
	region := "us-east-1"
	runID := strings.ToLower(random.UniqueID())
 
	// Layer 1: a throwaway bucket to hold this run's Terraform state.
	bucket := "tt-state-" + runID
	aws.CreateS3Bucket(t, region, bucket)
	defer func() {
		// S3 refuses to delete a bucket that still holds objects,
		// and state files are objects. Empty first, then delete.
		aws.EmptyS3Bucket(t, region, bucket)
		aws.DeleteS3Bucket(t, region, bucket)
	}() // unwinds LAST
 
	// Layer 2: the cluster.
	opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../examples/eks",
		BackendConfig: map[string]interface{}{
			"bucket": bucket, "key": runID + ".tfstate", "region": region,
		},
	})
	defer terraform.Destroy(t, opts) // unwinds THIRD
	terraform.InitAndApply(t, opts)
 
	// Layer 3: workloads inside the cluster.
	kubeOpts := k8s.NewKubectlOptions("", terraform.Output(t, opts, "kubeconfig_path"), "tt-app")
	k8s.CreateNamespace(t, kubeOpts, "tt-app")
	defer k8s.DeleteNamespace(t, kubeOpts, "tt-app") // unwinds SECOND
 
	helmOpts := &helm.Options{KubectlOptions: kubeOpts}
	defer helm.Delete(t, helmOpts, "tt-app", true) // unwinds FIRST
	helm.Install(t, helmOpts, "../charts/app", "tt-app")
 
	k8s.WaitUntilAllNodesReady(t, kubeOpts, 30, 10*time.Second)
}

Follow the order that produces. The Helm release goes first, because deleting a namespace while a release still owns objects carrying finalizers can hang for minutes and then time out. A finalizer is a sticky note on an object saying "do not delete me until some controller says it is finished", and if that controller is already gone the note never comes off. The namespace goes next, and this step matters far more than it looks. A Kubernetes Service of type LoadBalancer makes the cloud provider create a real load balancer, with its own security group and its own network interfaces. That load balancer belongs to Kubernetes, not to Terraform. Destroy the cluster first and the load balancer survives forever, because nothing in your Terraform state has ever heard of it. Then the cluster. Then the state bucket, last, because the destroy needed the state stored inside it. One honest wrinkle: the deferred helm.Delete is registered before helm.Install, so an install that never created anything makes the delete fail with release: not found and adds noise to an already-failing test. Swap in helm.DeleteE and log the error if that bothers you. A noisy log beats a leaked load balancer.

When the destroy itself fails

Cloud APIs are eventually consistent on the way down too. Eventually consistent means the answer you get back is a slightly stale photograph, not a live view, so a thing you deleted a second ago can still look present. A subnet will not delete while a network interface is still detaching from a Lambda function that is still draining. A security group will not delete while a second security group still references it. An API throttles you halfway through. Terraform exits non-zero, require.NoError fires, and now you have a failed test and leftovers, which is the worst of both.

terraform.WithDefaultRetryableErrors is the equivalent of knocking a second time when nobody answers the door. It takes your options, returns a deep copy, and loads a set of known-transient error patterns into RetryableTerraformErrors. The keys of that map are regular expressions matched against Terraform's output; the values are the human explanation printed when one matches. On a match, Terratest re-runs the entire command instead of giving up. There is a trap in the ordering, though. The helper hard-sets MaxRetries to 3 and TimeBetweenRetries to 5 seconds on the copy it returns, unconditionally, overwriting whatever you set on the struct you passed in. Note also that the retry loop runs MaxRetries + 1 times, so 3 buys you four attempts in total, and four attempts five seconds apart is nowhere near long enough for an ENI (elastic network interface, the virtual network card attached to an instance) to finish detaching. Set those fields on the returned pointer, after the call.

opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
	TerraformDir: "../examples/vpc",
})
 
// The helper returns a COPY and hard-sets MaxRetries=3 and
// TimeBetweenRetries=5s on it. Tune them AFTER the call; anything
// you set on the struct literal above is silently discarded.
// The loop runs MaxRetries+1 times, so 6 here means 7 attempts.
opts.MaxRetries = 6
opts.TimeBetweenRetries = 30 * time.Second
 
// The map is guaranteed non-nil here, so you can add your own pattern.
// On a hand-built Options you would have to make() it first.
opts.RetryableTerraformErrors["DependencyViolation: The subnet .* has dependencies"] =
	"An ENI detach is still in flight; retry the destroy."
 
defer terraform.Destroy(t, opts)
terraform.InitAndApply(t, opts)

Those assignments work whether you put them above or below the defer, because defer evaluates and captures the pointer, not a snapshot of what it points at. The struct is read at the moment Destroy actually runs. If you would rather inspect a destroy failure than fail the test on it, terraform.DestroyE returns the error to you and leaves the verdict in your hands, which is handy inside a cleanup that wants to try a second approach before giving up. Retrying your assertions is a different problem and belongs to the Assertions and retries lesson.

go test -v -timeout 30m ./test/ -run TestVpcModule 2>&1 | grep -E 'retry|Destroy complete'

TestVpcModule 2026-07-22T10:12:03Z retry.go:159: terraform [destroy -auto-approve -input=false -lock=false]
TestVpcModule 2026-07-22T10:13:31Z retry.go:247: 'terraform [destroy -auto-approve -input=false -lock=false]' failed with the error 'error while running command: exit status 1; Error: deleting EC2 Subnet (subnet-0f31a9c7b2d4e6058): DependencyViolation: The subnet 'subnet-0f31a9c7b2d4e6058' has dependencies and cannot be deleted.' but this error was expected and warrants a retry. Further details: An ENI detach is still in flight; retry the destroy.
TestVpcModule 2026-07-22T10:13:31Z retry.go:173: terraform [destroy -auto-approve -input=false -lock=false] returned an error: error while running command: exit status 1; Error: deleting EC2 Subnet (subnet-0f31a9c7b2d4e6058): DependencyViolation: The subnet 'subnet-0f31a9c7b2d4e6058' has dependencies and cannot be deleted.. Sleeping for 30s and will try again.
TestVpcModule 2026-07-22T10:14:01Z retry.go:159: terraform [destroy -auto-approve -input=false -lock=false]
TestVpcModule 2026-07-22T10:14:52Z logger.go:79: Destroy complete! Resources: 9 destroyed.

Four ways defer quietly does nothing

The most expensive leak is a scoping mistake. You tidy the apply into a setupInfra(t) helper and move the defer terraform.Destroy along with it. Now the destroy fires the instant that helper returns, before a single assertion runs, and your test spends the next two minutes probing infrastructure that is actively being deleted while producing failures that make no sense. defer is bound to the function it is written in, never to the test.

The fix is t.Cleanup, in Go's testing package since 1.14. If defer is a note pinned to one room's door, t.Cleanup is a note pinned to your coat: it goes with you and gets read when you leave the building, no matter which room you wrote it in. It registers a callback on the test object itself instead of on the current stack frame, so it runs when the test finishes, and it unwinds in the same LIFO order defer uses. It has a second advantage worth knowing before you reach the parallelism lesson. t.Cleanup runs after parallel subtests have completed, while a defer in the parent function fires the moment that parent returns, which with t.Parallel() is before the subtests have done any real work.

func setupInfra(t *testing.T) *terraform.Options {
	opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: "../examples/vpc",
	})
 
	// defer terraform.Destroy(t, opts) HERE would tear everything down
	// when setupInfra returns, i.e. before the caller asserts anything.
	// t.Cleanup is scoped to the test, not to this stack frame.
	t.Cleanup(func() { terraform.Destroy(t, opts) })
 
	terraform.InitAndApply(t, opts)
	return opts
}

The second way is the test timeout, and it catches people who have never read the flag. go test kills the whole binary after 10 minutes by default. An EKS control plane (elastic kubernetes service, Amazon's managed Kubernetes) takes 10 to 15 minutes to come up before your assertions have even started. When the deadline passes, the testing package's watchdog panics from the timer goroutine, not from yours. Your test's stack is never unwound, so none of your defers run, and the process dies with a full cluster standing.

# no -timeout flag: the binary gets the 10 minute default
go test -v ./test/ -run TestEksCluster

=== RUN   TestEksCluster
TestEksCluster 2026-07-22T12:00:04Z retry.go:159: terraform [apply -input=false -auto-approve -lock=false]
TestEksCluster 2026-07-22T12:04:11Z logger.go:79: module.eks.aws_eks_cluster.this: Still creating... [4m10s elapsed]
panic: test timed out after 10m0s
	running tests:
		TestEksCluster (10m0s)
 
goroutine 41 [running]:
testing.(*M).startAlarm.func1()
	/usr/local/go/src/testing/testing.go:2802 +0x39d
created by time.goFunc
	/usr/local/go/src/time/sleep.go:215 +0x2d
...
FAIL	github.com/acme/infra/test	600.019s
FAIL

Notice what is missing from that output: any mention of destroy. The cluster is still there. That is why every command in this course carries -timeout 30m. Set it above your worst observed run with real headroom, and remember it is a budget for the whole test binary, not for each test inside it.

The third way is os.Exit, which skips every deferred call by design. log.Fatal calls it for you, and so does the usual TestMain pattern that ends in os.Exit(m.Run()), which makes anything you defer inside TestMain purely decorative. The fourth way is the one you cannot code around: SIGKILL, the one signal a process is not allowed to catch or ignore. A cancelled pipeline, an evicted spot runner, a laptop that suspends on a train. The process is gone before any Go code runs. Nothing in the language saves you there, which is the whole argument for tagging and sweeping.

Skipping teardown on purpose

Waiting fourteen minutes for an apply after every one-line edit is not a workflow. Build the stage set once, rehearse on it as often as you like, strike it at the end. test_structure.RunTestStage gives you that: it splits a test into named stages you can switch off with environment variables, and a stage runs only when SKIP_<name> is unset. Build once, loop on the validate stage, then make one final pass that only tears down. Every go test invocation is a fresh process, so the in-memory terraform.Options from the setup run are long gone by the time the teardown run starts. test_structure.SaveTerraformOptions serializes them to <dir>/.test-data/TerraformOptions.json and LoadTerraformOptions reads them back, and that file is the only reason a teardown-only run knows what to destroy. Add .test-data/ to .gitignore before you accidentally commit a state path and a full variable set to the repository.

func TestEksCluster(t *testing.T) {
	workDir := "../examples/eks"
 
	// Deferred FIRST so it survives a failing validate stage.
	defer test_structure.RunTestStage(t, "teardown", func() {
		opts := test_structure.LoadTerraformOptions(t, workDir)
		terraform.Destroy(t, opts)
	})
 
	test_structure.RunTestStage(t, "setup", func() {
		opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
			TerraformDir: workDir,
		})
		// writes ../examples/eks/.test-data/TerraformOptions.json
		test_structure.SaveTerraformOptions(t, workDir, opts)
		terraform.InitAndApply(t, opts)
	})
 
	test_structure.RunTestStage(t, "validate", func() {
		opts := test_structure.LoadTerraformOptions(t, workDir)
		kubeOpts := k8s.NewKubectlOptions("", terraform.Output(t, opts, "kubeconfig_path"), "default")
		k8s.WaitUntilAllNodesReady(t, kubeOpts, 30, 10*time.Second)
	})
}

# 1. build it once and leave it up
SKIP_teardown=true go test -v -timeout 30m ./test/ -run TestEksCluster
 
# 2. iterate on the checks against the cluster you already have
SKIP_setup=true SKIP_teardown=true go test -v -timeout 30m ./test/ -run TestEksCluster
 
# 3. when you are done, run the teardown stage on its own
SKIP_setup=true SKIP_validate=true go test -v -timeout 30m ./test/ -run TestEksCluster

=== RUN   TestEksCluster
TestEksCluster 2026-07-22T13:41:02Z test_structure.go:47: The 'SKIP_setup' environment variable is set, so skipping stage 'setup'.
TestEksCluster 2026-07-22T13:41:02Z test_structure.go:47: The 'SKIP_validate' environment variable is set, so skipping stage 'validate'.
TestEksCluster 2026-07-22T13:41:02Z test_structure.go:44: The 'SKIP_teardown' environment variable is not set, so executing stage 'teardown'.
TestEksCluster 2026-07-22T13:41:02Z retry.go:159: terraform [destroy -auto-approve -input=false -lock=false]
...
TestEksCluster 2026-07-22T13:53:40Z logger.go:79: Destroy complete! Resources: 47 destroyed.
--- PASS: TestEksCluster (758.11s)
PASS
ok  	github.com/acme/infra/test	758.19s

Prove the account is actually empty

Destroy complete! is a claim, not a verification. Terraform removes what is in its state and nothing else. Objects your test wrote into a bucket, a load balancer created by a Kubernetes controller, a snapshot taken by a backup job the test triggered, a network interface a Lambda left behind: none of it is in state, so none of it gets destroyed. Check the account, not the state file.

terraform -chdir=examples/vpc state list; echo "exit=$?"

exit=0

Nothing listed, exit status 0. Terraform is telling the truth about the resources it owned. Now ask the provider what it still holds with your run's tag on it, which is the reason you set that tags variable in the first test. The bracketed [ResourceARN] in the query is deliberate: it wraps each result in its own row so --output text prints one ARN (amazon resource name, the unique identifier AWS gives every object) per line instead of running them together on a single tab-separated line.

aws resourcegroupstaggingapi get-resources \
  --region us-east-1 \
  --tag-filters Key=terratest,Values=true \
  --query 'ResourceTagMappingList[].[ResourceARN]' \
  --output text

arn:aws:ec2:us-east-1:111122223333:natgateway/nat-07c9d21a4b3ef5580
arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/net/k8s-ttapp-ttapp-1a2b3c4d5e/9f0a1b2c3d4e5f60

Two survivors, and they have different stories. The NAT gateway is left over from a run that hit the 10 minute default timeout before its defer could fire. The load balancer name beginning with k8s- is the tell on the second one. The AWS Load Balancer Controller builds names as k8s-<namespace>-<service>-<hash>, so this NLB (network load balancer) was created by the controller in response to a Service of type LoadBalancer, and was never Terraform's to delete. It only shows up in that tag query because the Service asked for your tags to be copied onto it. Without that annotation the load balancer carries only Kubernetes' own tags and your sweeper walks straight past it.

apiVersion: v1
kind: Service
metadata:
  name: tt-app
  namespace: tt-app
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: external
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: ip
    # Without this line the NLB the controller creates carries only
    # Kubernetes' own tags, so a tag-based sweep never finds it.
    service.beta.kubernetes.io/aws-load-balancer-additional-resource-tags: "terratest=true,run_id=8kq2rw"
spec:
  type: LoadBalancer
  selector:
    app: tt-app
  ports:
    - port: 80
      targetPort: 8080

That query is also your sweeper's query. Run a scheduled job in the test account that deletes anything tagged terratest=true and older than a few hours, using cloud-nuke, aws-nuke, or thirty lines of your own script. The age filter is the whole trade-off. Too aggressive and it deletes a long-running test out from under itself, producing failures that look like flaky cloud APIs and are in fact your own janitor. Too gentle and orphans pile up for a week before anyone looks. Six hours is a sane starting point if your slowest test finishes inside one. Point it only at an account holding nothing you care about, because a sweeper with a broad tag filter and production credentials is an outage generator. Account-level isolation and budget alarms are covered in the Cost, safety and isolation lesson.

Before you merge a new test, run it once with an assertion you have deliberately broken, and watch the last twenty lines. If the log ends with Destroy complete! and the tag query comes back empty, the cleanup path is real and you have tested it rather than assumed it. If it ends any other way, you have found the leak on a Tuesday afternoon instead of in next month's invoice.

Try this

Run go test -v -timeout 30m ./test/ -run TestVpcModule on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the 10 minute default is a resource leak. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related