Why test infrastructure code

The deploy-assert-destroy idea.

You can proofread a recipe all afternoon. Every ingredient is spelled correctly, the oven temperature is a real number, nobody has asked for a cup of concrete. None of that tells you the cake rises. Infrastructure code has the same gap. A linter (a tool that reads your files and complains about bad syntax) confirms your Terraform parses. A policy scanner such as Checkov reads those same files and confirms the storage bucket has encryption switched on. Both will hand you a clean green summary while the thing you deploy is broken, unreachable, or quietly open to the internet. Terratest closes that gap by refusing to read your files at all. It deploys them for real, prods the result with real requests, then tears the whole thing down.

Terratest is a library for Go, maintained by Gruntwork, at github.com/gruntwork-io/terratest. This lesson uses the v1.x release, so every name here is the current one. You write ordinary Go test functions. Underneath, they shell out to the real terraform, kubectl and helm binaries and then assert on whatever actually came up. That makes it integration testing for IaC (infrastructure as code, meaning your networks and servers are described in text files and applied by a tool, instead of being clicked together by hand in a web console). The bugs it catches live in the space between your code and a live cloud provider: an API (application programming interface, the cloud's own service counter that your tool sends orders to) that rejects a name containing an underscore, a subnet with no route out to the internet, a health check pointed at a path your application has never served.

Green Checks, Broken Service

Here is the shape of the problem. Take a small example module that builds a VPC (virtual private cloud, your own fenced-off network inside a cloud provider), an Application Load Balancer, and two instances behind it. A load balancer is a doorman. Every fifteen seconds it walks up to each instance, knocks on one specific door, and only sends customers to the ones that answer. Which door it knocks on is written in one file. The doors the application actually opens are defined somewhere else entirely, in another repository, in another language. Nothing forces the two to agree. Here they are one character apart.

resource "aws_lb_target_group" "app" {
  name        = "${var.name}-tg"
  port        = 8080
  protocol    = "HTTPS"
  target_type = "instance"
  vpc_id      = aws_vpc.this.id
 
  health_check {
    path                = "/healthz"   # the app actually serves /health
    protocol            = "HTTPS"
    matcher             = "200"
    healthy_threshold   = 2
    unhealthy_threshold = 2
    interval            = 15
  }
}

$ terraform -chdir=examples/app-stack init -backend=false > /dev/null
$ terraform -chdir=examples/app-stack validate
$ checkov -d examples/app-stack --quiet --compact

Success! The configuration is valid.
 
terraform scan results:
 
Passed checks: 25, Failed checks: 0, Skipped checks: 1

Valid configuration. Twenty-five passing policy checks, nothing failed, one finding the team triaged and suppressed on purpose. Ship it. Now deploy that same code and ask the load balancer a question instead of asking the file.

$ curl -s -o /dev/null -w '%{http_code}\n' https://tt-8fk2ql.test.example.com/
$ aws elbv2 describe-target-health --target-group-arn "$TG_ARN" \
    --query 'TargetHealthDescriptions[].TargetHealth' --output json

503
[
    {
        "State": "unhealthy",
        "Reason": "Target.ResponseCodeMismatch",
        "Description": "Health checks failed with these codes: [404]"
    },
    {
        "State": "unhealthy",
        "Reason": "Target.ResponseCodeMismatch",
        "Description": "Health checks failed with these codes: [404]"
    }
]

The load balancer is up. The instances are running. The service is down, because two strings in two repositories never agreed, and the doorman keeps knocking on a door that was bricked up months ago. No scanner will catch that, and it is not the scanner's failing. Whether /healthz is the right path is a fact about the application, not about the configuration. The same shape shows up all over security work. A security group rule points at a managed prefix list that quietly carries a vendor's entire /16 (a block of 65,536 addresses). An IAM (identity and access management, the cloud's system of keys and locks) role looks tightly scoped until a permissions boundary somebody added last quarter widens it. A bucket policy denies public reads while a CDN (content delivery network, a fleet of caching servers sitting in front of your storage) happily serves the same objects to anyone who asks. Every one of those is valid, scannable, passing configuration that behaves wrongly the moment it is real.

Deploy, Assert, Destroy

Book a rehearsal hall, run the show for an audience of one, strike the set before the hourly rate bites. That is the whole pattern, and every Terratest test you will ever write is a variation on it. Phase one deploys: apply the Terraform, install the Helm chart, apply the manifest. Phase two asserts: read the outputs and send real traffic at the real endpoint. Phase three destroys everything, including the half of it that failed. In Go, the third phase gets written first, using defer. A deferred call is a note you leave for the cleaner on your way in, not a chore you do on your way out. Go runs it when the surrounding function exits, whichever way it exits, including after an assertion has already failed the test. Registering the cleanup before you create anything is exactly what makes the cleanup survive a test that blows up in the middle.

package test
 
import (
	"strings"
	"testing"
	"time"
 
	http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
)
 
func TestAppStackServesTraffic(t *testing.T) {
	// UniqueID returns 6 base62 characters, mixed case. Lowercase them:
	// plenty of AWS names (S3 buckets, for one) reject capitals outright.
	name := "tt-" + strings.ToLower(random.UniqueID())
 
	opts := &terraform.Options{
		TerraformDir: "../examples/app-stack",
		Vars: map[string]any{
			"name": name,
		},
		EnvVars: map[string]string{
			"AWS_DEFAULT_REGION": "us-east-1",
		},
	}
 
	// Phase 3, written first. defer schedules this to run when the test
	// function exits, pass or fail, so nothing is left behind.
	defer terraform.Destroy(t, opts)
 
	// Phase 1: terraform init, then terraform apply -auto-approve.
	terraform.InitAndApply(t, opts)
 
	// Phase 2: read a real output, then hit the real endpoint.
	// 30 retries 10s apart means 31 attempts across roughly five minutes.
	url := terraform.Output(t, opts, "alb_url")
	http_helper.HttpGetWithRetry(t, url+"/health", nil, 200, "ok", 30, 10*time.Second)
}

terraform.Options carries the same knobs you would type at the command line. TerraformDir is a relative path to a small, cheap example directory, never your production root module. Vars becomes -var flags, so the test drives the module the way a real caller would. terraform.InitAndApply runs terraform init and then terraform apply -auto-approve, failing the test the moment either errors, which means that once it returns the resources genuinely exist. terraform.Output shells out to terraform output -no-color -json and hands you back a string. That random.UniqueID() in the name is not decoration. It stops two concurrent runs colliding on a load balancer name, which starts happening the day two pull requests merge within a minute of each other. HttpGetWithRetry polls instead of asking once, because clouds are eventually consistent: a new DNS (domain name system, the internet's phone book) record exists before every resolver has heard about it, the same way a new phone number takes a while to show up in the directory. Two details of that call catch people out. The retry count is the number of extra attempts, so 30 gives you 31 tries in total. And the expected body is compared for equality after surrounding whitespace is trimmed, so "ok" matches a body of "ok\n" but will never match {"status":"ok"}.

One naming wrinkle to know about. Terratest v1.0 added context-aware twins for every command helper and marked the older short names deprecated. InitAndApply, Destroy, Output and HttpGetWithRetry all still work, and most code you will read in the wild still uses them, so nothing you have written breaks. New tests are better off taking the ctx forms, because a context lets you cap how long a single apply is allowed to hang. Give your cleanup its own fresh context, though. If you hand Destroy an expired one, the teardown fails before it starts and you are back to paying for a stack nobody wanted.

ctx, cancel := context.WithTimeout(context.Background(), 25*time.Minute)
defer cancel()
 
// Cleanup gets a live context of its own, never the one that may have expired.
defer terraform.DestroyContext(t, context.Background(), opts)
 
terraform.InitAndApplyContext(t, ctx, opts)
url := terraform.OutputContext(t, ctx, opts, "alb_url")

$ go test -v -timeout 30m ./test/...

=== RUN   TestAppStackServesTraffic
TestAppStackServesTraffic 2026-07-22T09:14:02Z retry.go:159: terraform [init -upgrade=false]
TestAppStackServesTraffic 2026-07-22T09:14:02Z logger.go:79: Running command terraform with args [init -upgrade=false]
TestAppStackServesTraffic 2026-07-22T09:14:07Z logger.go:79: Terraform has been successfully initialized!
TestAppStackServesTraffic 2026-07-22T09:14:07Z retry.go:159: terraform [apply -input=false -auto-approve -var name=tt-8fk2ql -lock=false]
TestAppStackServesTraffic 2026-07-22T09:14:07Z logger.go:79: Running command terraform with args [apply -input=false -auto-approve -var name=tt-8fk2ql -lock=false]
TestAppStackServesTraffic 2026-07-22T09:14:31Z logger.go:79: aws_vpc.this: Creation complete after 3s [id=vpc-0d4b1a9c7e2f81a33]
TestAppStackServesTraffic 2026-07-22T09:18:44Z logger.go:79: Apply complete! Resources: 23 added, 0 changed, 0 destroyed.
TestAppStackServesTraffic 2026-07-22T09:18:44Z retry.go:159: terraform [output -no-color -json alb_url]
TestAppStackServesTraffic 2026-07-22T09:18:44Z logger.go:79: Running command terraform with args [output -no-color -json alb_url]
TestAppStackServesTraffic 2026-07-22T09:18:45Z retry.go:159: HTTP GET to URL https://tt-8fk2ql.test.example.com/health
TestAppStackServesTraffic 2026-07-22T09:18:45Z http_helper.go:101: Making an HTTP GET call to URL https://tt-8fk2ql.test.example.com/health
TestAppStackServesTraffic 2026-07-22T09:18:45Z retry.go:173: HTTP GET to URL https://tt-8fk2ql.test.example.com/health returned an error: Get "https://tt-8fk2ql.test.example.com/health": dial tcp: lookup tt-8fk2ql.test.example.com on 10.0.0.2:53: no such host. Sleeping for 10s and will try again.
TestAppStackServesTraffic 2026-07-22T09:18:55Z retry.go:159: HTTP GET to URL https://tt-8fk2ql.test.example.com/health
TestAppStackServesTraffic 2026-07-22T09:18:55Z http_helper.go:101: Making an HTTP GET call to URL https://tt-8fk2ql.test.example.com/health
TestAppStackServesTraffic 2026-07-22T09:18:55Z retry.go:173: HTTP GET to URL https://tt-8fk2ql.test.example.com/health returned an error: Validation failed for URL https://tt-8fk2ql.test.example.com/health. Response status: 503. Response body:
<html><head><title>503 Service Temporarily Unavailable</title></head>. Sleeping for 10s and will try again.
TestAppStackServesTraffic 2026-07-22T09:19:06Z retry.go:159: HTTP GET to URL https://tt-8fk2ql.test.example.com/health
TestAppStackServesTraffic 2026-07-22T09:19:06Z http_helper.go:101: Making an HTTP GET call to URL https://tt-8fk2ql.test.example.com/health
TestAppStackServesTraffic 2026-07-22T09:19:07Z retry.go:159: terraform [destroy -auto-approve -input=false -var name=tt-8fk2ql -lock=false]
TestAppStackServesTraffic 2026-07-22T09:19:07Z logger.go:79: Running command terraform with args [destroy -auto-approve -input=false -var name=tt-8fk2ql -lock=false]
TestAppStackServesTraffic 2026-07-22T09:22:58Z logger.go:79: Destroy complete! Resources: 23 destroyed.
--- PASS: TestAppStackServesTraffic (536.44s)
PASS
ok  	github.com/acme/infra/test	536.61s

Read the timestamps, because they are the honest bill for this technique. Four minutes and thirty-seven seconds to create twenty-three resources. Twenty-two seconds of retrying while DNS caught up and the first target passed its health check, which is the eventual consistency that would have flaked a single unretried request. Three minutes and fifty-one seconds to take it all down again. A shade under nine minutes of wall clock, to prove one thing: the endpoint answers 200 with the word ok. That is why Terratest suites are counted in dozens and not in thousands.

What Only a Real Deploy Can Answer

Some questions have no static answer. Does the load balancer route to a backend that is genuinely healthy? Does that IAM role really let the application read that bucket, and does it really stop it reading the other one? Does your module still compose correctly with the networking module it depends on, now that both were bumped in the same week? Does an in-place upgrade from the last released version work, or does it hit an immutable field and force a replacement that takes production down at two in the morning? Those are questions about the behaviour of a running system, and the only way to answer them is to run the system. Terratest answers them in a throwaway environment where being wrong costs you nine minutes and a few cents.

Negative assertions deserve as much attention as positive ones, and they are where the security payoff sits. Reading the specification of a lock tells you it is a good lock. Pulling the handle tells you whether the door is actually locked. A scanner confirms your configuration says deny. A test confirms the request is actually denied, after the bucket policy, the IAM role, the load balancer rules, the reverse proxy and the application code have all had their say. Attackers do not read your HCL (HashiCorp Configuration Language, the syntax Terraform files are written in). They send requests and watch what comes back. So send those requests yourself, in a test, on every merge.

// Called from the assert phase, after InitAndApply has returned.
func assertLeastPrivilege(t *testing.T, opts *terraform.Options) {
	appURL := terraform.Output(t, opts, "alb_url")
 
	// Positive: the app's own health endpoint answers.
	// HttpGet trims surrounding whitespace from the body for you.
	status, body := http_helper.HttpGet(t, appURL+"/health", nil)
	require.Equal(t, 200, status)
	require.Equal(t, "ok", body)
 
	// Negative: /admin must be refused, not merely missing from the nav bar.
	status, body = http_helper.HttpGet(t, appURL+"/admin", nil)
	require.Equal(t, 401, status, "unauthenticated /admin must be refused")
	require.NotContains(t, body, "dashboard")
 
	// Negative: the object must not be readable straight from the bucket,
	// bypassing the load balancer and every rule attached to it.
	objectURL := terraform.Output(t, opts, "private_object_url")
	status, _ = http_helper.HttpGet(t, objectURL, nil)
	require.Equal(t, 403, status, "object must not be publicly readable")
}

The same rhythm works one layer up the stack. For Kubernetes, the k8s module applies your manifests with KubectlApply and then blocks on WaitUntilPodAvailable until the pod genuinely reaches Running. The helm module runs Install, then Upgrade over the top of it, after which you open a port-forward tunnel with NewTunnel and send the Service a real HTTP request through it. Different package, identical loop: deploy, assert, destroy.

The Honest Trade-off

Real deploys are slow and they cost money, and pretending otherwise is how teams abandon a test suite three months in. The run above created a NAT gateway (network address translation, the box that lets private instances reach out to the internet without being reachable from it) at roughly four and a half cents per hour, plus a load balancer at a bit over two cents per hour, for nine minutes. Pennies. Now put forty such tests on every pull request, twenty times a day. The pennies stop being pennies, and the nine-minute wait becomes the reason people start merging without running them. So put each check where it earns its keep. terraform fmt, terraform validate, tflint and Checkov run on every commit in seconds and catch typos, house-style drift and known misconfigurations. Terratest runs on merge to the main branch, or nightly, on the handful of paths where being wrong is expensive.

The failure that will actually bite you is a test that does not clean up. A cancelled pipeline, a killed process, a destroy that errors on a dependency it cannot delete, and now there is infrastructure quietly running in an account nobody opens. Defend against it the way an airline defends against lost luggage, by tagging everything before it leaves your hands. Set default_tags in your provider block so every resource an example creates is stamped terratest = true, then make the question "did we leave anything behind" a single command anyone on the team can run.

$ aws resourcegroupstaggingapi get-resources \
    --tag-filters Key=terratest,Values=true \
    --query 'ResourceTagMappingList[].ResourceARN' --output json

[
    "arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/app/tt-9x2plq-alb/8f2c1d90a4b7e3ee",
    "arn:aws:ec2:us-east-1:111122223333:natgateway/nat-04a91c3f4e77b21d5"
]

Two orphans from a run somebody cancelled yesterday, each identified by its ARN (Amazon Resource Name, the unique address AWS gives every object it owns). Both are billing by the hour. Neither is attached to anything that serves a request. Run that sweep on a schedule and alert when the result comes back non-empty. The tt-cost and tt-cicd lessons turn it into a scheduled reaper that deletes rather than reports. For now, knowing the query exists is what keeps a leak from quietly becoming a line item.

Start smaller than feels worthwhile. Take the one module whose breakage caused your last incident, write a single test that deploys its example, asserts the exact behaviour that failed that day, and destroys it. Run it with go test -v -timeout 30m ./test/... against a sandbox account, then reintroduce the bug and watch it go red. When the nine-minute loop starts to grate while you are iterating, wrap each phase in test_structure.RunTestStage, which skips any stage whose SKIP_ environment variable is set. Export SKIP_teardown=true once and the deployment stays standing between runs, so SKIP_deploy=true SKIP_teardown=true go test -run TestAppStack ./test/... re-runs only your assertions against it. Those nine minutes become about five seconds, right up until you unset both and let the real teardown finish the job.

Try this

Run terraform -chdir=examples/app-stack init -backend=false > /dev/null on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: terraform.Destroy destroys whatever TerraformDir points at. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related