Parallelism & fixtures

Isolated, concurrent tests.

Ten cooks, one cutting board. Everyone is competent, nobody is careless, and dinner is still ruined, because two people reached for the same surface in the same second. Infrastructure tests fail in exactly that way. Running them one at a time works and takes forever, since every terraform apply sits waiting on a cloud API (application programming interface, the remote service you send requests to and wait on for an answer) that replies in minutes rather than milliseconds. Turn on concurrency and the arithmetic looks lovely. Twelve tests at eight minutes each finish in about nine minutes instead of ninety six. Then the collisions start.

Three things every test wants to own

A Terratest run touches three shared surfaces, and each one breaks in its own way. First, the folder on disk. Two tests running terraform init in the same directory fight over the .terraform/ working directory, the downloaded provider binaries, and terraform.tfstate. You get "text file busy" errors, half-written state, or the ugly one: a state file that describes one test's resources while a second test destroys them. Second, the cloud account. S3 (Simple Storage Service, Amazon's file store) bucket names have to be unique across every AWS (Amazon Web Services) account on earth, IAM (Identity and Access Management, the permissions system) role names are unique inside one account, DynamoDB table names are unique per account per region. Two tests both asking for my-app-data means one of them fails, and if the loser's teardown runs anyway it can delete the winner's resources. Third, the operating system process. Every test in a package shares one process, which means one environment block, one working directory, and one clock running one timeout for all of them.

What t.Parallel actually does

Terratest ships no test runner of its own. It is a Go library, so parallelism here is Go's own t.Parallel(), and that function behaves less obviously than its name suggests. Think of a relay race where the runners walk to the start line and then wait for the gun. t.Parallel() does not fork a goroutine (Go's lightweight thread) at the line where you call it. It tells the test runner "hold me here", and the test stops dead. Go then finishes every test that did not ask to be parallel, and only after that serial pass does it fire the gun and release the whole paused group together. Code above the call runs in the serial phase. Code below it runs in the concurrent phase. Put it on the first line of the function and you never have to work out which half you are standing in.

Concurrency has a ceiling, and there are two dials rather than one. The flag -parallel N caps how many parallel tests run at once inside a single test binary, and it defaults to GOMAXPROCS (the number of operating system threads Go will run your code on, normally your CPU core count). That default is wrong for infrastructure tests. Your bottleneck is a cloud API spending four minutes building a NAT gateway (network address translation, the box that lets private servers reach the internet), not your processor, so 8 or 16 on a four-core runner is reasonable. The dial almost nobody notices is -p N, which sets how many test binaries (one per package) run at once, and it also defaults to GOMAXPROCS. Go's own flag documentation is blunt about this: -parallel only applies within a single test binary. Four test packages on an eight-core box therefore give you a worst case of 32 concurrent applies, not 8. Raise -timeout as well, because it defaults to ten minutes, that budget covers the entire binary, and one honest apply can eat all of it. Finally, pass -count=1. Go caches passing test results, a cached result is treated as taking no time at all, and a cached pass on infrastructure code tells you nothing whatsoever about the infrastructure.

# -parallel caps tests inside one binary; -p caps how many binaries run at once
cd ~/src/terraform-aws-baseline
go test -v -timeout 30m -parallel 8 -p 2 -count=1 ./test/...

=== RUN   TestS3BucketIsPrivate
=== PAUSE TestS3BucketIsPrivate
=== RUN   TestDynamoTableEncrypted
=== PAUSE TestDynamoTableEncrypted
=== CONT  TestS3BucketIsPrivate
=== CONT  TestDynamoTableEncrypted
TestS3BucketIsPrivate 2026-07-22T09:14:02Z test_structure.go:149: Copied terraform folder ../examples/s3-bucket to /tmp/TestS3BucketIsPrivate2947103856/examples/s3-bucket
TestDynamoTableEncrypted 2026-07-22T09:14:02Z test_structure.go:149: Copied terraform folder ../examples/dynamodb to /tmp/TestDynamoTableEncrypted1180432877/examples/dynamodb
TestS3BucketIsPrivate 2026-07-22T09:14:02Z region.go:128: Using region eu-west-1
TestDynamoTableEncrypted 2026-07-22T09:14:02Z region.go:128: Using region us-east-2
TestS3BucketIsPrivate 2026-07-22T09:14:03Z command.go:200: Running command terraform with args [init -upgrade=false -reconfigure -backend-config=key=terratest/TestS3BucketIsPrivate/k3f9qz.tfstate]
TestDynamoTableEncrypted 2026-07-22T09:14:03Z command.go:200: Running command terraform with args [init -upgrade=false -reconfigure -backend-config=key=terratest/TestDynamoTableEncrypted/p7v2ma.tfstate]
...
TestDynamoTableEncrypted 2026-07-22T09:15:38Z command.go:200: Running command terraform with args [destroy -auto-approve -input=false -var table_name=terratest-locks-p7v2ma -var aws_region=us-east-2 -lock=true]
--- PASS: TestDynamoTableEncrypted (98.42s)
--- PASS: TestS3BucketIsPrivate (121.07s)
PASS
ok  	github.com/acme/terraform-aws-baseline/test	121.63s

The === PAUSE and === CONT markers are your proof that parallelism is switched on. A test that goes straight from === RUN to its result never paused, which means t.Parallel() is missing or something above it failed first. Notice also that the Terratest lines are not indented the way t.Log output is. Terratest's default logger writes straight to standard output with its own prefix of test name, timestamp and source location, and it does not hold a lock while writing, so with eight tests running the lines braid together and can even split mid-line. The capture above is a tidied two-test run, so read it for the shape rather than as a transcript you will match line for line, and treat the file and line numbers in each prefix as v1.0.1 values that shift whenever the library is rebuilt. That test name at the front of every line is the only thread you have to pull on. One detail in there rewards a close read: terraform init receives -upgrade=false, -reconfigure and the backend key, but no -lock flag at all, because Terratest assembles the init command by hand instead of running it through the shared argument formatter. Locking shows up on plan, apply and destroy.

Give every test its own copy of the module

The fix for disk contention is a photocopy. Instead of pointing eight tests at examples/s3-bucket, hand each one a private copy in a temp directory and let them fight over nothing. test_structure.CopyTerraformFolderToTemp does that for you. You give it a root folder and the module path inside it, it copies the tree to /tmp/<TestName><random>/, and it returns the path to your module inside the copy. Copying the whole root rather than the single module is deliberate, because modules reference their siblings with relative paths like ../../modules/s3, and those paths have to keep resolving after the move.

It does not copy everything, and the details matter. Hidden files and folders are skipped, which is how .terraform/ and .git/ stay behind. Two hidden files are written in as deliberate exceptions and do travel with the copy: .terraform.lock.hcl, the dependency lock file holding the provider checksums somebody reviewed, and .terraform-version. That exception is the entire reason your temp fixture resolves the same provider builds your team approved rather than whatever the registry happens to be serving today, so go and check that the lock file is committed next to your example. If it is missing from the repository it is missing from the copy, and every run then re-resolves providers under a loose constraint like version = "~> 6.0" inside a process holding live cloud credentials. What gets dropped on purpose is state and inputs: terraform.tfstate, terraform.tfstate.backup, terraform.tfvars and terraform.tfvars.json. Pass inputs through Vars in terraform.Options instead of expecting a tfvars file to ride along.

One trapdoor sits underneath all of this. If any environment variable whose name starts with SKIP_ is set, CopyTerraformFolderToTemp quietly stops copying and hands back the original folder instead. That behaviour exists for the stage-skipping workflow in Test stages & speed, where you deliberately want every stage to reuse one directory and its cached state. Export SKIP_teardown in your shell one afternoon, forget about it, then run the full suite in parallel, and all of your tests are back to sharing a single working directory and a single terraform.tfstate. Nothing warns you beyond one log line about using the original examples folder. Check your environment before you trust a parallel run, and keep SKIP_ variables out of CI (continuous integration, the server that runs your tests on every push).

package test
 
import (
	"fmt"
	"strings"
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	test_structure "github.com/gruntwork-io/terratest/modules/test-structure"
	"github.com/stretchr/testify/assert"
)
 
func TestS3BucketIsPrivate(t *testing.T) {
	t.Parallel() // first line: pause here, resume with the rest of the group
 
	// Private copy of the whole repo; returns the module path inside the temp tree.
	exampleDir := test_structure.CopyTerraformFolderToTemp(t, "../", "examples/s3-bucket")
 
	uniqueID := strings.ToLower(random.UniqueID()) // 6 base-62 chars, e.g. "k3f9qz"
	awsRegion := aws.GetRandomStableRegion(t, nil, nil)
	bucketName := fmt.Sprintf("terratest-logs-%s", uniqueID)
 
	opts := terraform.WithDefaultRetryableErrors(t, &terraform.Options{
		TerraformDir: exampleDir,
		Vars: map[string]any{
			"bucket_name": bucketName,
			"aws_region":  awsRegion,
			"tags": map[string]string{
				"Terratest": "true",   // what the sweeper looks for
				"TestName":  t.Name(), // who to go and ask about it
			},
		},
		// Appended to os.Environ() for the terraform child process only.
		// Never os.Setenv: one Go process is shared by every test in the binary.
		EnvVars: map[string]string{"AWS_DEFAULT_REGION": awsRegion},
		// One state key per run, so a shared backend is not shared state.
		BackendConfig: map[string]any{
			"key": fmt.Sprintf("terratest/%s/%s.tfstate", t.Name(), uniqueID),
		},
		Reconfigure: true,
		Lock:        true, // Terratest sends -lock=false unless you ask for locking
		// API throttling is NOT in the default retryable list. Add it yourself;
		// WithDefaultRetryableErrors merges its own patterns in alongside these.
		RetryableTerraformErrors: map[string]string{
			".*RequestLimitExceeded.*": "AWS throttled us under parallel load.",
			".*ThrottlingException.*":  "AWS throttled us under parallel load.",
		},
	})
 
	defer terraform.Destroy(t, opts)
	terraform.InitAndApply(t, opts)
 
	assert.Equal(t, bucketName, terraform.Output(t, opts, "bucket_id"))
	aws.AssertS3BucketVersioningExists(t, awsRegion, bucketName)
	aws.AssertS3BucketPolicyExists(t, awsRegion, bucketName)
}

Four settings there carry most of the weight. BackendConfig gives each run its own state key, because a remote backend with a hardcoded key is one shared file no matter how many temp directories you made. Lock: true matters more than it looks. Options.Lock is a plain boolean whose zero value is false, and it renders directly into -lock=false on plan, apply and destroy, which switches off Terraform's own guard against two applies touching one state. With a unique state key per run there is nothing left to collide with, so locking is cheap insurance. With a shared key you needed it an hour ago. WithDefaultRetryableErrors clones your options, adds Terratest's list of known transient failures, and sets three retries five seconds apart. Read that list before you trust it. It covers provider download and registry flakiness, "connection reset by peer", "transport is closing", and the eventual-consistency case where a provider reports an inconsistent result after apply. It does not cover cloud API throttling, and throttling is precisely what parallelism buys you, so RequestLimitExceeded and ThrottlingException are patterns you add yourself. Your entries survive the clone, so you end up with both sets.

# Go refuses to let a parallel test touch the process-wide environment
go test -v -run TestRegionOverride ./test/

=== RUN   TestRegionOverride
=== PAUSE TestRegionOverride
=== CONT  TestRegionOverride
--- FAIL: TestRegionOverride (0.00s)
panic: testing: test using t.Setenv, t.Chdir, or cryptotest.SetGlobalRandom can not use t.Parallel [recovered, repanicked]
 
goroutine 7 [running]:
testing.tRunner.func1.2({0x6c1a40, 0x77e270})
	/usr/local/go/src/testing/testing.go:1974 +0x239
testing.tRunner.func1()
	/usr/local/go/src/testing/testing.go:1977 +0x349
panic({0x6c1a40?, 0x77e270?})
	/usr/local/go/src/runtime/panic.go:860 +0x13a
testing.(*T).checkParallel(...)
	/usr/local/go/src/testing/testing.go:1838
testing.(*T).Setenv(0xc000103c00, {0x6f9086, 0x12}, {0x6f6282, 0x9})
	/usr/local/go/src/testing/testing.go:1852 +0x65
github.com/acme/terraform-aws-baseline/test.TestRegionOverride(0xc000103c00)
	/home/ci/src/terraform-aws-baseline/test/region_test.go:12 +0x3a
FAIL	github.com/acme/terraform-aws-baseline/test	0.014s

Unique names, spread across regions

On-disk isolation solves half the problem. Cloud namespaces are the other half, and you share those with your colleagues, with every CI branch, and with the run you kicked off ten minutes ago and forgot about. random.UniqueID() returns six base-62 characters (digits plus upper and lower case letters), which is 62 to the sixth power, roughly 57 billion combinations, plenty for one suffix per stack. Two caveats come with it. Lowercase the result before it touches an S3 bucket name or a Kubernetes namespace, because both reject capitals outright. And it comes from Go's math/rand seeded off the wall clock rather than crypto/rand, so treat it as unique-ish, never as unguessable: it is a collision avoider, not a secret. Feed the suffix into the module as a Terraform variable so every resource name is built from it, rather than sprinkling the random string through your test code. Older examples spell the same function with a lowercase d on the end. That spelling still compiles, because Terratest keeps it as a one-line wrapper, but it now carries a deprecation marker pointing at random.UniqueID, so use the capitalised name in anything new.

Region choice is the other lever. aws.GetRandomStableRegion(t, nil, nil) picks from a hardcoded list of sixteen regions old enough to have real capacity, which deliberately leaves out the newest and thinnest ones where capacity errors are common, and it needs no API call to make that choice. Pass an approved list as the second argument and a forbidden list as the third when your account restricts regions. Spreading tests around does two useful things. It dodges the per-region service quotas that a burst of parallel applies would otherwise exhaust (the default allowance is five VPCs, Virtual Private Clouds, per region, and that goes fast), and it flushes out the hardcoded us-east-1 somebody left in the module. When you need one repeatable run, set TERRATEST_REGION=eu-west-1 and every region pick in the run collapses to that value, which is what you want when you are chasing a known failure rather than hunting for new ones.

Shared fixtures and the defer trap

Sometimes one expensive thing should be built once and shared, like an EKS cluster (Elastic Kubernetes Service, AWS's managed Kubernetes) that twenty policy checks all need. Build it in a parent test and run the cheap checks as parallel subtests inside it. This is exactly where defer betrays you. Think of defer as a note pinned to the door saying "lock up on your way out", and t.Cleanup as a note to the caretaker saying "lock up once everybody has gone home". A parallel subtest pauses at its own t.Parallel() and is released only after the parent function has already returned, so a deferred terraform.Destroy in the parent fires while the subtests are still standing outside in the cold. Your cluster is gone before a single check runs, and the failure looks like a broken kubeconfig rather than a scheduling mistake. t.Cleanup is the fix, because Go guarantees that a cleanup function registered on a test runs after that test and all of its subtests have completed, parallel ones included. Twenty lines of Go prove it without touching a cloud account.

mkdir -p demo
cat > demo/order_test.go <<'EOF'
package demo
 
import (
	"fmt"
	"testing"
)
 
func TestParent(t *testing.T) {
	t.Parallel()
	defer fmt.Println(">>> DEFER in parent ran")
	t.Cleanup(func() { fmt.Println(">>> CLEANUP in parent ran") })
 
	for _, n := range []string{"a", "b"} {
		t.Run(n, func(t *testing.T) {
			t.Parallel()
			fmt.Println(">>> subtest", n, "resumed")
		})
	}
	fmt.Println(">>> parent body finished")
}
EOF
go test -v -run TestParent ./demo/

=== RUN   TestParent
=== PAUSE TestParent
=== CONT  TestParent
=== RUN   TestParent/a
=== PAUSE TestParent/a
=== RUN   TestParent/b
=== PAUSE TestParent/b
>>> parent body finished
>>> DEFER in parent ran
=== CONT  TestParent/a
>>> subtest a resumed
=== CONT  TestParent/b
>>> subtest b resumed
>>> CLEANUP in parent ran
--- PASS: TestParent (0.00s)
    --- PASS: TestParent/a (0.00s)
    --- PASS: TestParent/b (0.00s)
PASS
ok  	github.com/acme/terraform-aws-baseline/demo	0.065s

Read the order of those lines carefully, because it is the whole argument. The deferred call fires before either subtest resumes. The cleanup fires after both have finished. Swap in terraform.Destroy for that Println and you have the difference between a cluster that exists while the checks run and one that does not.

// imports: fmt, strings, testing, time, plus the terratest terraform, random,
// k8s and test-structure modules.
 
func TestClusterPolicies(t *testing.T) {
	t.Parallel()
 
	clusterDir := test_structure.CopyTerraformFolderToTemp(t, "../", "examples/eks-cluster")
	opts := &terraform.Options{TerraformDir: clusterDir, Lock: true}
 
	// NOT defer: this function returns the moment the subtests below pause,
	// and defer would destroy the cluster before any of them resume.
	t.Cleanup(func() { terraform.Destroy(t, opts) })
	terraform.InitAndApply(t, opts)
 
	kubeconfig := terraform.Output(t, opts, "kubeconfig_path")
 
	cases := []struct{ name, app string }{
		{"deny-all-egress", "payments"},
		{"allow-dns-only", "checkout"},
	}
	for _, tc := range cases {
		t.Run(tc.name, func(t *testing.T) {
			t.Parallel()
 
			// one namespace per subtest: the same isolation idea, one level down
			ns := tc.app + "-" + strings.ToLower(random.UniqueID())
			kubectlOpts := k8s.NewKubectlOptions("", kubeconfig, ns)
 
			k8s.CreateNamespace(t, kubectlOpts, ns)
			t.Cleanup(func() { k8s.DeleteNamespace(t, kubectlOpts, ns) })
 
			// Manifest paths resolve against the test's working directory, not the
			// temp copy. Each of these files defines a Pod literally named "probe".
			k8s.KubectlApply(t, kubectlOpts, "../manifests/"+tc.name+".yaml")
			k8s.WaitUntilPodAvailable(t, kubectlOpts, "probe", 30, 5*time.Second)
		})
	}
}

The loop variable footgun that used to live in that for loop has gone. Since Go 1.22 loop variables are per-iteration, so each subtest closes over its own tc and the old tc := tc shadowing line is dead weight. The go directive in go.mod selects those semantics, not whichever toolchain you happen to have installed, and Terratest v1.0.1 requires go 1.26 in any case. Sharing a cluster is a cost decision with a security bill attached. A subtest that installs a mutating webhook (a cluster-wide hook that rewrites objects as they are created) or any cluster-scoped policy changes the ground under every other subtest, and a green run then proves nothing about any of them. Keep shared fixtures to things that are read-mostly, and give anything cluster-scoped a cluster of its own.

There is a partial defence, and it is the one piece of Terratest v1.0.1 worth changing your habits for. Every helper now has a Context twin: InitAndApplyContext, DestroyContext, OutputContext, GetRandomStableRegionContext and the rest, with the shorter names kept working but marked deprecated. Those variants take a context.Context, and Terratest hands it to exec.CommandContext, so a per-test deadline genuinely kills the terraform child process instead of letting it grind on until the binary-wide alarm goes off. Give each test its own ctx, cancel := context.WithTimeout(context.Background(), 12*time.Minute), and a stack that hangs fails its own test while its own deferred Destroy still runs. Know what that costs you. The child is killed outright with no chance to write state cleanly, so whatever Destroy finds afterwards is best-effort and you should still expect to sweep.

Prove the isolation, then prove the cleanup

Two checks tell you whether your parallelism is real or theatre. The first is the temp directory list in the log: if every test recorded a different /tmp/Test.../ path, then no two of them shared a state file or a plugin directory. The second is a tag sweep after the run, which answers the only question that matters to the person paying the bill and the person carrying the pager. Interleaved output is the practical tax on all of this, and Terratest ships a parser for it. Pipe the run to a file, feed the file to terratest_log_parser, and you get one clean log per test plus a JUnit XML report (the format CI systems read to show per-test results).

go install github.com/gruntwork-io/terratest/cmd/terratest_log_parser@latest
 
set -o pipefail   # without this, tee hides a failing go test exit code
go test -v -timeout 30m -parallel 8 -count=1 ./test/... 2>&1 | tee out.log
terratest_log_parser --testlog out.log --outputdir test_output
ls test_output/

TestDynamoTableEncrypted.log
TestS3BucketIsPrivate.log
report.xml
summary.log

Now the part that costs money. After a clean run, nothing tagged Terratest should still exist anywhere you ran tests, and because GetRandomStableRegion scattered them you have to look in more than one region. This is a different job from the stage skipping in Test stages & speed, where you leave infrastructure standing on purpose while you iterate. Here, anything still standing is an accident.

for r in eu-west-1 us-east-2 ap-southeast-2; do
  aws resourcegroupstaggingapi get-resources --region "$r" \
    --tag-filters Key=Terratest,Values=true \
    --query 'ResourceTagMappingList[].ResourceARN' --output text
done

arn:aws:s3:::terratest-logs-k3f9qz
arn:aws:dynamodb:us-east-2:111122223333:table/terratest-locks-p7v2ma

Two survivors from the run that timed out, one of them a bucket that will sit there collecting objects and charges until somebody notices. Treat that query as a floor rather than a ceiling, because the Resource Groups Tagging API only reports the resource types it supports, so pair it with a real sweeper. In a sandbox account the backstop is a scheduled cloud-nuke run: cloud-nuke aws --region eu-west-1 --older-than 2h --resource-type s3 --resource-type dynamodb --force, driven from CI on a timer with a config file that restricts it to names matching ^terratest-. The --force flag is not optional in CI, because without it the tool sits waiting on a confirmation prompt nobody is there to answer. Restrict everything else about it hard. A sweeper that can delete anything is a very effective attack tool the moment its credentials leak, so give it a role scoped to the sandbox account and to tag-matched resources, and nothing beyond that.

The real ceiling on all this is not your test code. Sixteen parallel applies means sixteen sets of NAT gateways and load balancers billing by the hour, and cloud APIs start throttling long before your CPU breaks a sweat. Start at -parallel 4, watch the logs for RequestLimitExceeded and for tests that get slower rather than failing outright, and raise the number until one of those shows up. Then drop back by one and write it into the Makefile, next to the -timeout you sized for the slowest stack in the suite and the -p you set to stop four packages quietly multiplying it behind your back.

Try this

Run go test -v -timeout 30m -parallel 8 -p 2 -count=1 ./test/... on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: the environment block is shared, because the process is one process. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related