Cost, safety & isolation

Real resources cost money.

A Terratest run rents things. Every terraform apply in the suite creates a real machine, a real load balancer, a real database, and the meter starts the moment each one exists, not when your assertions start passing. It works like a taxi with the engine running. The fare has nothing to do with whether you reach the right address; it counts minutes. That is the trade Terratest makes. It gives up simulation to get the truth, and the truth arrives with a price per hour.

So treat the suite the way a good lab treats a dangerous experiment. You run it in a room with its own fuse box, its own extractor fan, and a cleaner who empties the room every night. Never on the kitchen table. For infrastructure tests, that room is a throwaway cloud account, and the discipline is making sure nothing you create in it can reach anything real or outlive the test that made it.

The Bill Is Resources Times Minutes

Two numbers explain most test-infrastructure bills. Start with a NAT gateway (Network Address Translation gateway, the box that lets machines on a private network reach the internet without being reachable from it). In us-east-1 it costs about $0.045 an hour, plus $0.045 for every gigabyte that passes through it. A twelve-minute test that creates one pays about a cent. The same gateway, forgotten, costs roughly $33 a month and keeps going. Same resource, same code. The only difference is how long it lived.

Fill in the rest of the price list and the picture sharpens. A public IPv4 address (the internet-routable address something needs to be reachable from outside your own network) bills $0.005 an hour whether it is attached to anything or not, so one stranded Elastic IP is $3.65 a month of pure nothing. A db.t3.micro RDS (Relational Database Service) instance, about the smallest managed database AWS sells, runs near $13 a month. An Application Load Balancer starts around $16. An EKS (Elastic Kubernetes Service) control plane is a flat $0.10 an hour, $73 a month, for a cluster nobody is using. Those are us-east-1 list prices at the time of writing and they move, so price your own. The ratio is what matters. The resources that hurt are the ones with no owner and no natural end.

The runs you sit and watch are not where the money goes. You see those finish. The money goes to the run that died at 2am on a CI (continuous integration, the service that builds and tests your code on every push) runner, left a NAT gateway and two disks standing, and told nobody. Cloud billing has no alarm for that. It has an invoice, weeks later. So the first thing to build is not a cleanup script. Build the ability to ask what the suite costs.

# what did the test suite actually cost last month, by service?
aws ce get-cost-and-usage \
  --time-period Start=2026-06-01,End=2026-07-01 \
  --granularity MONTHLY \
  --metrics UnblendedCost \
  --filter '{"Tags":{"Key":"terratest","Values":["true"]}}' \
  --group-by Type=DIMENSION,Key=SERVICE \
  --query 'ResultsByTime[0].Groups[].[Keys[0],Metrics.UnblendedCost.Amount]' \
  --output text

Amazon Elastic Compute Cloud - Compute	1.0142880000
Amazon Elastic Load Balancing	22.9038116000
Amazon Relational Database Service	61.4419820000
Amazon Virtual Private Cloud	9.1541000000
EC2 - Other	214.8734521000

terratest there is a tag your fixtures stamp on every resource, and two things trip people up. A tag you invent does nothing in Cost Explorer until you activate it as a cost allocation tag in the Billing console, and activation is slow: up to a day for the key to even appear in that list, and up to another day for it to switch on. It applies forward from activation, though the StartCostAllocationTagBackfill operation will push the activation status back as far as the previous twelve months if you ask, one request per twenty-four hours, always starting from the first day of some month. The other trap is price. Every paginated Cost Explorer API request (the programmatic interface behind the console) costs a cent, so never put this call in a polling loop.

Now read the numbers. EC2 - Other is the bucket Cost Explorer uses for everything billed under the EC2 product code that is not instance hours: EBS (Elastic Block Store, the virtual hard disks) volumes and snapshots, NAT gateway hours, data processing. The suite ran 312 times last month at about nine minutes each. That is 47 hours of live infrastructure. Price a healthy run against it. Two t3.micro instances for 47 hours is about a dollar, and the compute line agrees almost exactly. One NAT gateway for 47 hours should be $2.11. One load balancer, $1.05. One db.t3.micro, $0.84. Compare those to the invoice. EC2 - Other is a hundred times too big and the database line is seventy times too big, while compute is fine, because instances are the one thing Terraform reliably manages to delete. Add it up and the leak costs more than fifty times what the tests themselves do.

Isolation Is the Master Control

Isolation contains every other mistake, so decide it first. The suite runs in its own cloud account or project: its own credentials, its own bill, and an IAM (Identity and Access Management, the system that decides who is allowed to do what) permission boundary or SCP (Service Control Policy, an account-wide rule that caps what anybody inside the account can do, administrators included) that makes a path from sandbox into production impossible rather than merely discouraged. When that is true, a broken test is an embarrassing invoice. When it is not, a broken test is an incident.

The security case is sharper than the cost case. Terraform runs arbitrary commands through provisioners and external data sources. A pipeline that runs Terratest on pull requests is therefore a service that executes code written by anyone who can open a pull request, using whatever cloud credentials you handed the runner. If those credentials can reach production, you have built a remote code execution path into production and labelled it CI. GitHub Actions gives pull requests from forks no secrets by default, which is the right default. Teams lose it anyway in three familiar ways: pull_request_target, which runs the workflow with the base repository's secrets; self-hosted runners that keep credentials on disk between jobs; and one long-lived administrator key pasted into repository settings two years ago and never rotated.

The account boundary is the wall. The guard is the person checking passes at the door. Terratest ships aws.GetAccountId, which calls STS (Security Token Service) GetCallerIdentity and returns the account the current credentials belong to. Assert on it before a single resource is created. Here it protects TestVpcModule, a test that builds a VPC (Virtual Private Cloud, a private network of your own inside the provider) and everything hanging off it. Two notes on the file. Keep the guard in a _test.go file, because importing Go's testing package from an ordinary source file drags test flags into anything that imports it. And note that Terratest v1.0 renamed this call to aws.GetAccountIDContext(t, ctx) while keeping GetAccountId as a deprecated alias that still compiles; everything below is written against Terratest v1.0.1.

package test
 
import (
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/stretchr/testify/require"
)
 
// The only account this suite is ever allowed to touch.
const sandboxAccountID = "111122223333"
 
// requireSandbox stops the test before anything is created if the
// credentials in the environment point somewhere else.
func requireSandbox(t *testing.T) {
	got := aws.GetAccountId(t) // STS GetCallerIdentity under the hood
	require.Equal(t, sandboxAccountID, got,
		"refusing to run: not the sandbox account (got %s)", got)
}

# who are these credentials, really?
aws sts get-caller-identity
 
# run the suite; the guard fires before the first apply
go test -v -timeout 30m -run TestVpcModule ./test/

{
    "UserId": "AROAY3KX7QW2NLZ4EXAMPL:terratest-ci",
    "Account": "409988776655",
    "Arn": "arn:aws:sts::409988776655:assumed-role/deploy-admin/terratest-ci"
}
=== RUN   TestVpcModule
=== PAUSE TestVpcModule
=== CONT  TestVpcModule
    guard_test.go:17: 
        	Error Trace:	/home/you/infra/test/guard_test.go:17
        	            				/home/you/infra/test/vpc_test.go:24
        	Error:      	Not equal: 
        	            	expected: "111122223333"
        	            	actual  : "409988776655"
        	            	
        	            	Diff:
        	            	--- Expected
        	            	+++ Actual
        	            	@@ -1 +1 @@
        	            	-111122223333
        	            	+409988776655
        	Test:       	TestVpcModule
        	Messages:   	refusing to run: not the sandbox account (got 409988776655)
--- FAIL: TestVpcModule (0.61s)
FAIL
FAIL	github.com/acme/infra/test	0.884s
FAIL

The run stopped in 0.61 seconds and created nothing. Read what it caught. The credentials in that shell belong to 409988776655 under a role called deploy-admin, so somebody was one command away from applying a test fixture into a real account with real permissions. A wrong AWS_PROFILE, a stale AWS_ACCESS_KEY_ID still exported from this morning, a .tfvars file (the file that supplies input values to Terraform) copied out of the production repository. All of them die here instead of after the apply. The === PAUSE and === CONT lines are there because the test calls t.Parallel(). And notice require rather than assert: require calls FailNow and stops the test, while assert records the failure and lets execution carry straight on into the apply, which is the opposite of what you want from a door guard.

Name Every Resource After the Run That Made It

Airlines put a barcode on every bag at check-in, and that one label is the only reason a bag lost in Frankfurt can ever be found. Do the same to your test resources. Give each run a short unique name and stamp it on everything. random.UniqueID() returns a six-character base62 string (digits plus upper and lower case letters, so roughly 56 billion possibilities), and wrapping it in strings.ToLower(...) gives you something like ay3k9x. Lowercase matters for two different reasons: S3 (Simple Storage Service) bucket names reject uppercase outright, and RDS identifiers are silently stored lowercase, which leaves Terraform comparing what you asked for against what AWS returned on every plan, forever. That one string does two jobs. Parallel runs stop colliding on names that must be unique per account, and every resource becomes findable later by a human, a script, or a sweeper.

Tags alone are not enough. Tag search misses resources whose service does not report to the tagging API, and several cleanup tools filter by name rather than by tag. Put the run ID in the name and in the tags. A tt- prefix on the name gives you one pattern that matches everything the suite has ever built.

func TestVpcModule(t *testing.T) {
	t.Parallel()
	requireSandbox(t) // wrong account? nothing gets created
 
	runID := strings.ToLower(random.UniqueID()) // "ay3k9x"
	name := fmt.Sprintf("tt-%s", runID)
 
	// fixed working dir so the stages below can find saved state
	fixture := "../examples/vpc"
 
	opts := &terraform.Options{
		TerraformDir: fixture,
		Vars: map[string]interface{}{
			"name":               name,       // lands in every resource name
			"instance_type":      "t3.micro", // smallest that proves the path
			"az_count":           1,          // one AZ, not three
			"single_nat_gateway": true,       // one NAT gateway, not one per AZ
			"tags": map[string]string{
				"terratest": "true",
				"run":       runID,
				"owner":     "platform-team",
				"expires":   time.Now().Add(2 * time.Hour).UTC().Format(time.RFC3339),
			},
		},
		EnvVars: map[string]string{
			// region only. credentials stay in the test process environment.
			"AWS_DEFAULT_REGION": aws.GetRandomStableRegion(t,
				[]string{"us-east-1", "us-west-2"}, nil),
		},
		NoColor: true,
	}

Three cost levers sit in that block. t3.micro and one Availability Zone (a physically separate datacentre inside a region) shrink the fixture, because you are testing that the module wires things together, not that it survives Black Friday, and the smallest fixture proves the wiring exactly as well as the biggest. single_nat_gateway matters more than the instance size: three Availability Zones means three NAT gateways at $33 a month each if one ever escapes. aws.GetRandomStableRegion picks from a hardcoded list of sixteen regions that have existed for at least a year, which spreads load so parallel runs stop hitting per-region service quotas, and it honours the TERRATEST_REGION environment variable when you need to pin one region for a debugging session.

One more thing about that fixed TerraformDir. If several tests share a folder and run with t.Parallel(), they fight over the same .terraform working directory and the same local state file. test_structure.CopyTerraformFolderToTemp(t, "../", "examples/vpc") solves that by copying the whole tree to a fresh temporary path per test and returning it. People often assume the temp copy breaks the staged workflow in the next section, on the reasoning that a new path each run means yesterday's saved state cannot be found. It does not, and the reason is worth knowing. That function checks first whether any environment variable starting with SKIP_ is set, and if one is, it skips the copy entirely and hands back the original folder. Terratest reads the presence of a skip variable as proof you are iterating by hand rather than running in parallel. Parallel CI runs get an isolated temp copy. Your laptop keeps the fixed directory. You do not have to pick.

Stop Paying to Build What You Already Built

Writing an assertion usually takes a few tries. If every try re-applies the whole stack, an eight-minute apply and a five-minute destroy turn ten attempts into more than two hours of billed infrastructure to fix one line of Go. What you want is a recipe you can restart halfway through. test_structure.RunTestStage splits the test into named phases, and each phase checks an environment variable before running: set SKIP_deploy and the deploy phase does not run. The stack survives between runs because SaveTerraformOptions writes the options to .test-data/TerraformOptions.json inside the working directory and LoadTerraformOptions reads them back.

	// ...continued from above.
	// deferred first so it runs last, after validate
	defer test_structure.RunTestStage(t, "teardown", func() {
		saved := test_structure.LoadTerraformOptions(t, fixture)
		terraform.Destroy(t, saved)
	})
 
	test_structure.RunTestStage(t, "deploy", func() {
		test_structure.SaveTerraformOptions(t, fixture, opts)
		terraform.InitAndApply(t, opts)
	})
 
	test_structure.RunTestStage(t, "validate", func() {
		saved := test_structure.LoadTerraformOptions(t, fixture)
		url := terraform.Output(t, saved, "alb_url")
		http_helper.HttpGetWithRetry(t, url, nil, 200, "ok", 30, 5*time.Second)
	})
}

# first pass: build it and leave it standing
SKIP_teardown=true go test -v -timeout 30m -run TestVpcModule ./test/
 
# now iterate on the assertion against the SAME live stack
SKIP_deploy=true SKIP_teardown=true \
  go test -v -timeout 30m -run TestVpcModule ./test/
 
# finished for the day: run nothing but the destroy
SKIP_deploy=true SKIP_validate=true \
  go test -v -timeout 30m -run TestVpcModule ./test/

=== RUN   TestVpcModule
=== PAUSE TestVpcModule
=== CONT  TestVpcModule
TestVpcModule 2026-07-22T09:41:02+01:00 test_structure.go:47: The 'SKIP_deploy' environment variable is set, so skipping stage 'deploy'.
TestVpcModule 2026-07-22T09:41:02+01:00 test_structure.go:44: The 'SKIP_validate' environment variable is not set, so executing stage 'validate'.
TestVpcModule 2026-07-22T09:41:02+01:00 retry.go:159: HTTP GET to URL http://tt-ay3k9x-alb-1837465920.us-east-1.elb.amazonaws.com
TestVpcModule 2026-07-22T09:41:03+01:00 http_helper.go:101: Making an HTTP GET call to URL http://tt-ay3k9x-alb-1837465920.us-east-1.elb.amazonaws.com
TestVpcModule 2026-07-22T09:41:03+01:00 test_structure.go:47: The 'SKIP_teardown' environment variable is set, so skipping stage 'teardown'.
--- PASS: TestVpcModule (1.24s)
PASS
ok  	github.com/acme/infra/test	1.512s

That is the second command, and it took a little over a second instead of thirteen minutes, with no new resources created. The teardown line appears last because the stage was deferred, so it fires on the way out. Two habits keep this from backfiring. Add .test-data/ to .gitignore, because those saved options are a verbatim dump of your Terraform variables and can hold values you would rather not commit. And never let SKIP_teardown reach CI, where a skipped destroy quietly turns every run into a permanent deployment. Set the skip variables on your own machine, from your own shell, one session at a time.

Plan for Cleanup to Fail

defer terraform.Destroy is the happy path and it is a good one. It is also the path that does not happen when a CI job is cancelled and the runner receives SIGKILL (the one signal a process cannot catch or clean up after), when a spot instance is reclaimed mid-test, when the runner runs out of memory, or when Go's own test timeout fires. Even a destroy that does start can fail halfway. An S3 bucket that is not empty. A security group still held by a network interface. A subnet that will not delete because something else is attached to it. Terraform reports the error and exits. The resources that did delete are gone, and the ones that did not keep billing.

So assume cleanup will miss sometimes, and build the net underneath it. Because the run ID is on every name and every tag, finding what leaked is one query. The Resource Groups Tagging API answers for most services, though not every one, which is the other reason the name prefix earns its place. It is also regional, so ask each region your tests are allowed to use. It returns an ARN (Amazon Resource Name, the unique identifier AWS gives every resource) per match.

# anything the suite created that is still alive, in one region
aws resourcegroupstaggingapi get-resources \
  --region us-east-1 \
  --tag-filters Key=terratest,Values=true \
  --query 'ResourceTagMappingList[].ResourceARN' \
  --output text | tr '\t' '\n'

arn:aws:ec2:us-east-1:111122223333:natgateway/nat-0d3f1c9b7a2e5f480
arn:aws:ec2:us-east-1:111122223333:volume/vol-0a71c4e2f9b3d6851
arn:aws:rds:us-east-1:111122223333:db:tt-ay3k9x-pg
arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/app/tt-ay3k9x/9f2c1d0b7e4a6538

Four resources wearing the same run ID, from a test that finished hours ago. Left alone that is roughly $33 for the NAT gateway, $16 for the load balancer and $13 for the database every month, plus the disk, for a pull request that was merged and forgotten. This is what a scheduled sweep exists to collect. cloud-nuke deletes resources by type, age and tag, and because the account is disposable it can be aggressive without anybody flinching.

#!/usr/bin/env bash
# Runs at 02:00 in the sandbox account. Nowhere else. Ever.
set -euo pipefail
 
EXPECTED=111122223333
ACTUAL=$(aws sts get-caller-identity --query Account --output text)
if [[ "$ACTUAL" != "$EXPECTED" ]]; then
  echo "refusing to sweep: credentials belong to $ACTUAL" >&2
  exit 1
fi
 
# --older-than protects runs still in flight. Its default is 0s, which
#   means no age filter at all, so never leave this flag out.
# --include-tag takes key=value; the value is a case-sensitive regex,
#   so ^true$ pins it exactly instead of matching "truest".
# --force skips the prompt that otherwise makes you type 'nuke'.
cloud-nuke aws \
  --region us-east-1 --region us-west-2 \
  --resource-type ec2 \
  --resource-type nat-gateway \
  --resource-type rds-instance \
  --resource-type elbv2 \
  --resource-type ebs \
  --resource-type eip \
  --include-tag "terratest=^true$" \
  --older-than 2h \
  --force >> /var/log/sandbox-sweep.log 2>&1

Notice the guard again, in the same shape as the one in guard_test.go. That repetition is the point, because this script is the single most dangerous file you own. The identical command with production credentials deletes production, and once --force is set it will not ask twice. Give the sweeper an IAM role that exists only in the sandbox account and cannot assume anything outside it, run it from a host with no other credentials on disk, and keep --older-than comfortably longer than your slowest test so a running suite never gets nuked out from under itself. Run cloud-nuke aws --list-resource-types to get the exact names your version accepts rather than guessing: the RDS type is rds-instance, not rds, and a wrong name gets you Invalid resourceTypes [rds] specified instead of a sweep. One honest limit: --include-tag can only match resources whose tags cloud-nuke can see, so a service that does not surface them slips past the filter. That is another reason the account has to be disposable.

Verify the sweeper before you trust it, and keep verifying. cloud-nuke inspect-aws lists what it would find and touches nothing, which is what you want in a nightly report even on the days it finds nothing. One catch: cloud-nuke writes its own progress logging to standard output, so send the JSON (JavaScript Object Notation, a plain-text data format) somewhere else with --output-file if you want to feed it to jq or a dashboard.

# what WOULD the sweep take right now? nothing is deleted.
cloud-nuke inspect-aws \
  --region us-east-1 \
  --resource-type nat-gateway \
  --resource-type rds-instance \
  --resource-type elbv2 \
  --older-than 2h \
  --output-format json \
  --output-file /var/log/sandbox-leaks.json
 
cat /var/log/sandbox-leaks.json

{
  "timestamp": "2026-07-22T02:00:19.284416Z",
  "command": "aws",
  "query": {
    "regions": [
      "us-east-1"
    ],
    "resource_types": [
      "nat-gateway",
      "rds-instance",
      "elbv2"
    ],
    "exclude_after": "2026-07-22T00:00:19.106552Z",
    "list_unaliased_kms_keys": false
  },
  "resources": [
    {
      "resource_type": "nat-gateway",
      "region": "us-east-1",
      "identifier": "nat-0d3f1c9b7a2e5f480",
      "nukable": true
    },
    {
      "resource_type": "rds-instance",
      "region": "us-east-1",
      "identifier": "tt-ay3k9x-pg",
      "nukable": true
    },
    {
      "resource_type": "elbv2",
      "region": "us-east-1",
      "identifier": "tt-ay3k9x",
      "nukable": true
    }
  ],
  "summary": {
    "total_resources": 3,
    "nukable": 3,
    "non_nukable": 0,
    "general_errors": 0,
    "by_type": {
      "elbv2": 1,
      "nat-gateway": 1,
      "rds-instance": 1
    },
    "by_region": {
      "us-east-1": 3
    }
  }
}

Put a Ceiling on the Account

Everything above is prevention, and prevention has holes: a resource type nobody added to the sweep list, a new service, a region a test wandered into. The last layer is a hard ceiling on the sandbox account itself, working the way a fuse box does. It has no idea which appliance is faulty and it does not need to know. AWS Budgets alerts at a threshold you choose, and a budget action goes further, attaching a deny policy to the test role, applying a Service Control Policy, or stopping EC2 and RDS instances once spending crosses the line. That turns runaway cost into a failing pipeline instead of a quiet invoice. Cost Anomaly Detection catches the other shape you care about: a line item that was flat for a month and suddenly is not.

Be honest about the lag. AWS Budgets refreshes up to three times a day, with roughly eight to twelve hours between updates, so a threshold is a backstop measured in hours, not a control measured in seconds. A leaked EKS cluster gets most of a day at $0.10 an hour before the alert reaches anybody, and the invoice arrives regardless. That is exactly why the budget is the last layer and never the first. If you take one thing from this lesson and put it in your repository this afternoon, make it the guard: one function call and one assertion, in a file you will never think about again, and the only piece here that can save you from something considerably worse than money.

Try this

Run aws sts get-caller-identity on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the guard and Terraform can end up as two different identities. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related