>_SecOpsLog_
CoursesTerratestStrategy & the test pyramid

Strategy & the test pyramid

What to test, and at what level.

Advanced14 min · lesson 9 of 12

A bridge inspector does not load-test every beam until it snaps. Most of the confidence comes from cheap things: a tape measure, a torque wrench, a second pair of eyes on the drawings. A few full-load tests happen at the end, on the pieces where being wrong would kill someone. Infrastructure testing works the same way. Cheap checks catch the boring majority of mistakes in seconds. Per-module tests catch the ones that only appear when a real cloud API (Application Programming Interface, the machine-to-machine control channel a cloud exposes so that programs can create and query resources) answers back. A small number of full deployments prove the whole stack stands up.

The common mistake is thinking Terratest only lives at the top, running giant end-to-end deploys that take forty minutes and fail for reasons nobody can explain. It lives at every level. Your job is deciding which level a given check belongs to, and the honest tiebreaker is money and minutes.

The Bill Decides the Shape

Terratest is not a simulator. terraform.InitAndApply shells out to the real terraform binary, which calls the real cloud API with real credentials and creates real resources that land on a real invoice. An EKS control plane (Elastic Kubernetes Service, the managed Kubernetes brain that Amazon Web Services runs for you) bills $0.10 an hour whether your test passes or fails. Add a NAT gateway (Network Address Translation, the box that lets private servers reach the internet without being reachable from it) at $0.045 an hour, two load balancers, and three m5.large worker nodes at $0.096 an hour each. A forty-minute run costs you roughly thirty-five cents. That sounds like nothing. Now put fifteen engineers on it, pushing three times a day: forty-five runs, about sixteen dollars a day, north of three hundred dollars a month for one test.

Time is the harsher tax. A plan-level assertion returns in about six seconds. A single S3 (Simple Storage Service, the AWS object store where you dump files) module test takes fifty. An end-to-end run that builds a VPC (Virtual Private Cloud, your own walled-off network inside the cloud), a cluster and an app takes twenty-five to forty minutes, and nobody waits for that on every commit. Whatever you can prove at the bottom of the pyramid, prove it there. Push a check upward only when the layer below genuinely cannot answer the question.

There is a security bill too. Every layer that applies anything needs credentials with the power to create and destroy infrastructure, which is exactly the power an attacker would like to borrow. Point the suite at a dedicated sandbox account with its own budget alarm. Never a shared account, never production. Cost, isolation and CI (continuous integration, the robot that runs your tests on every push) credentials get their own lessons. What belongs here is the idea that the price of a test is part of deciding where it lives.

What lives at each level
Base: nothing is deployed
terraform validate
syntax and types, no provider call, under a second
tflint, Checkov, tfsec
provider anti-patterns and insecure defaults
InitAndPlanAndShowWithStruct
assert on planned values, about 6s, $0
helm.RenderTemplate
chart manifests checked with no cluster anywhere
Middle: one module, real API
InitAndApply + defer Destroy
1 to 5 minutes, cents per run
aws / k8s helper assertions
ask the provider, not the state file
dozens of these
where most of your real confidence comes from
Top: the whole environment
environments/staging
VPC + EKS + app, 25 to 40 minutes, real spend
http_helper.HttpGetWithRetry
the contract a customer actually sees
one or two, no more
critical paths only, run on a schedule
Runtime and dollars climb from left to right while the number of tests falls. Invert that shape, with many end-to-end tests and few module tests, and the suite turns slow, flaky and expensive.

The Base: Checks That Never Call a Cloud API

The cheapest tests need no credentials at all. terraform validate parses your configuration and type-checks it, so a variable declared as a number and handed a string fails here instead of four minutes into an apply. tflint knows provider-specific rules that Terraform itself does not, like an instance type that does not exist in the region you named. Checkov and tfsec are policy scanners: they read the HCL (HashiCorp Configuration Language, the language Terraform files are written in) and flag insecure defaults, such as a bucket with no logging or a security group open to 0.0.0.0/0 (shorthand for every address on the internet).

terminal
# 1. syntax and types. -backend=false means no remote state, no credentials
terraform -chdir=modules/s3-bucket init -backend=false
terraform -chdir=modules/s3-bucket validate
 
# 2. provider anti-patterns
tflint --chdir=modules/s3-bucket --format=compact
 
# 3. insecure defaults, as policy
checkov -d modules/s3-bucket --compact --quiet
output
Initializing provider plugins...
- Finding latest version of hashicorp/aws...
- Installing hashicorp/aws v6.55.0...
- Installed hashicorp/aws v6.55.0 (signed by HashiCorp)
 
Terraform has been successfully initialized!
 
Success! The configuration is valid.
 
main.tf:1:1: Warning - Missing version constraint for provider "aws" in `required_providers` (terraform_required_providers)
 
terraform scan results:
 
Passed checks: 10, Failed checks: 5, Skipped checks: 0
 
Check: CKV_AWS_18: "Ensure the S3 bucket has access logging enabled"
	FAILED for resource: aws_s3_bucket.this
	File: /main.tf:1-8
	Guide: https://docs.prismacloud.io/en/enterprise-edition/policy-reference/aws-policies/s3-policies/s3-13-enable-logging
Check: CKV_AWS_145: "Ensure that S3 buckets are encrypted with KMS by default"
	FAILED for resource: aws_s3_bucket.this
	File: /main.tf:1-8

That whole loop finishes in seconds and costs nothing, which is why it belongs in a pre-commit hook (a small script Git runs before it lets a commit through) where nobody has to remember to run it. Notice that Checkov is opinionated: two of those five findings are about cross-region replication and lifecycle rules, which plenty of teams deliberately skip. Tuning the ruleset gets its own lesson. What matters for strategy is the position: bottom of the pyramid, running constantly, catching the mistakes that are embarrassing rather than interesting.

Plan Tests: Assert on Intent in Seconds

Read the recipe before you turn on the oven and you catch most of what would have gone wrong. terraform plan writes down exactly what Terraform intends to create, change and destroy, and Terratest can read that intention as structured data. terraform.InitAndPlanAndShowWithStruct runs init, then plan -out=<file>, then terraform show -json on the saved plan, and parses the JSON (JavaScript Object Notation, a plain-text way of writing nested data) into a Go struct you can index by resource address.

test/plan_test.go
package test
 
import (
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)
 
func TestPlanS3Bucket(t *testing.T) {
	t.Parallel()
 
	opts := &terraform.Options{
		TerraformDir: "../modules/s3-bucket",
		Vars:         map[string]interface{}{"name": "acme-logs-plan-check"},
 
		// Required. InitAndPlanAndShowWithStruct writes `plan -out=<this>`
		// and reads it back with `terraform show -json`. Leave it empty and
		// the call fails before a single assertion runs.
		PlanFilePath: "plan.out",
		NoColor:      true,
	}
 
	plan := terraform.InitAndPlanAndShowWithStruct(t, opts)
 
	// The guard rail has to exist at all...
	terraform.RequirePlannedValuesMapKeyExists(t, plan,
		"aws_s3_bucket_public_access_block.this")
	pab := plan.ResourcePlannedValuesMap["aws_s3_bucket_public_access_block.this"]
 
	// ...and all four of its switches have to be on.
	for _, flag := range []string{
		"block_public_acls",
		"block_public_policy",
		"ignore_public_acls",
		"restrict_public_buckets",
	} {
		assert.Equal(t, true, pab.AttributeValues[flag], "%s must stay true", flag)
	}
}

Two details bite people here. First, PlanFilePath is not optional. Leave it empty and the call returns you must set PlanFilePath on options struct to use this function before it looks at a single resource. Second, AttributeValues is a map[string]interface{} decoded straight from JSON, so every number arrives as a float64 (a 64-bit floating point number, the only numeric type JSON knows). Write assert.Equal(t, 3, subnets.AttributeValues["count"]) and it fails, though modern testify at least tells you why: expected: int(3) against actual: float64(3). Compare against float64(3) and it passes.

terminal
# -count=1 defeats Go's test result cache (see below)
go test -v -timeout 30m -count=1 -run TestPlanS3Bucket ./test/
output
=== RUN   TestPlanS3Bucket
=== PAUSE TestPlanS3Bucket
=== CONT  TestPlanS3Bucket
TestPlanS3Bucket 2026-07-22T09:41:12+01:00 retry.go:159: terraform [init -upgrade=false -no-color]
TestPlanS3Bucket 2026-07-22T09:41:12+01:00 logger.go:79: Running command terraform with args [init -upgrade=false -no-color]
TestPlanS3Bucket 2026-07-22T09:41:14+01:00 logger.go:79: Terraform has been successfully initialized!
TestPlanS3Bucket 2026-07-22T09:41:14+01:00 retry.go:159: terraform [plan -input=false -lock=false -var name=acme-logs-plan-check -no-color -lock=false -out=plan.out]
TestPlanS3Bucket 2026-07-22T09:41:14+01:00 logger.go:79: Running command terraform with args [plan -input=false -lock=false -var name=acme-logs-plan-check -no-color -lock=false -out=plan.out]
TestPlanS3Bucket 2026-07-22T09:41:17+01:00 logger.go:79: Plan: 4 to add, 0 to change, 0 to destroy.
TestPlanS3Bucket 2026-07-22T09:41:17+01:00 logger.go:79: Saved the plan to: plan.out
TestPlanS3Bucket 2026-07-22T09:41:17+01:00 retry.go:159: terraform [show -no-color -json plan.out]
TestPlanS3Bucket 2026-07-22T09:41:17+01:00 logger.go:79: Running command terraform with args [show -no-color -json plan.out]
TestPlanS3Bucket 2026-07-22T09:41:18+01:00 logger.go:79: {"format_version":"1.2","terraform_version":"1.15.8","variables":{"name":{"value":"acme-logs-plan-check"}},"planned_values":{ ...
--- PASS: TestPlanS3Bucket (6.71s)
PASS
ok  	github.com/acme/infra/test	7.02s

Every command shows up twice in that log, once from the retry wrapper and once from the shell runner. Useful when a run hangs and you want to know which terraform invocation it hung on. The duplicated -lock=false on the plan line is Terratest's own doing and harmless, since Terraform takes the last value it sees.

Now about -count=1. Go caches test results. If the test binary and its flags look unchanged, go test replays a stale (cached) pass in milliseconds and runs nothing at all. The cache only knows about files the test process itself opened, and Terratest does its real work inside a terraform subprocess, so your .tf edits are invisible to it. Watch it happen, and break the module on purpose while you are there, because a test that has never failed is not a test yet, it is a hope.

terminal
go test -timeout 30m -run TestPlanS3Bucket ./test/   # runs for real
go test -timeout 30m -run TestPlanS3Bucket ./test/   # replays the old result
 
# loosen one guard rail, exactly like a rushed pull request would
sed -i 's/block_public_acls       = true/block_public_acls       = false/' \
  modules/s3-bucket/main.tf
 
go test -timeout 30m -run TestPlanS3Bucket ./test/            # still green. still wrong.
go test -timeout 30m -count=1 -run TestPlanS3Bucket ./test/   # honest again
 
git checkout -- modules/s3-bucket/main.tf
output
ok  	github.com/acme/infra/test	6.94s
ok  	github.com/acme/infra/test	(cached)
ok  	github.com/acme/infra/test	(cached)
--- FAIL: TestPlanS3Bucket (6.55s)
    plan_test.go:38: 
        	Error Trace:	/home/you/infra/test/plan_test.go:38
        	Error:      	Not equal: 
        	            	expected: true
        	            	actual  : false
        	Test:       	TestPlanS3Bucket
        	Messages:   	block_public_acls must stay true
FAIL
FAIL	github.com/acme/infra/test	6.88s
FAIL

The third line is the one that should scare you. The guard rail is off, the plan would say so, and Go reports a green pass without starting Terraform at all. A green light that checked nothing is worse than a red one. Put -count=1 in every Terratest command you write. The fourth run then does its job in six seconds for zero dollars, catching the bug that turns up in breach write-ups over and over: a public access block someone loosened to unblock a demo and never put back.

Helm charts get the same treatment. helm.RenderTemplate runs helm template and hands you the rendered YAML (a plain-text format for structured configuration) as a string, and helm.UnmarshalK8SYaml parses that string into a real Kubernetes struct. You can then assert that runAsNonRoot is true, or that no container asks for privileged, without a cluster existing anywhere.

A green plan is not a green apply
Plan tests are cheap and seductive, and it is tempting to make them your whole safety net. Do not. During a plan the provider checks your arguments against its own schema, offline, on your machine. It never asks AWS whether the request would actually succeed. So a plan will not catch an S3 bucket name someone already took globally, an IAM (Identity and Access Management, the cloud's permission system) policy the API rejects with MalformedPolicyDocument, a service quota you have exhausted, an instance type not offered in that availability zone, or the eventual-consistency gaps that only surface during a real apply. A plan can be perfectly green while the apply dies four minutes later. Your floor for the claim 'this module actually deploys' is one real apply and destroy at the middle layer, not a plan.

The Middle: One Module, One Real API

Plan tests prove intent. They cannot prove the cloud agrees. The middle layer is Terratest's home ground: apply a single module in isolation, ask the provider what really happened, then tear it down. Test the appliance on the bench, not the whole kitchen. A focused s3-bucket or vpc test is quick enough to run on every pull request and small enough that a failure names the culprit instead of pointing at a forty-resource environment and shrugging.

test/s3_bucket_test.go
package test
 
import (
	"strings"
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/aws"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/gruntwork-io/terratest/modules/terraform"
	"github.com/stretchr/testify/assert"
)
 
func TestS3BucketModule(t *testing.T) {
	t.Parallel()
 
	region := "us-east-1"
	// random.UniqueID() returns 6 base-62 characters, so two engineers (or two
	// CI jobs) running this at the same moment do not fight over one name.
	name := "acme-logs-" + strings.ToLower(random.UniqueID())
 
	opts := &terraform.Options{
		TerraformDir: "../modules/s3-bucket",
		Vars:         map[string]interface{}{"name": name, "region": region},
		EnvVars:      map[string]string{"AWS_DEFAULT_REGION": region},
	}
 
	// Registered BEFORE the apply on purpose: a half-finished apply still
	// gets cleaned up when the assertions below blow up.
	defer terraform.Destroy(t, opts)
 
	terraform.InitAndApply(t, opts)
 
	bucketID := terraform.Output(t, opts, "bucket_id")
	assert.Equal(t, name, bucketID)
 
	// Ask AWS what is true. State only records what Terraform believes.
	aws.AssertS3BucketVersioningExists(t, region, bucketID)
}

The rule that earns this layer its keep is in that last comment. Assert against the provider's API, never against Terraform's own state file. State is Terraform's memory of what it did last time. If a colleague turned versioning off in the console, or a bucket policy quietly overrides what your module set, state still reports everything as fine while the account sits exposed. aws.AssertS3BucketVersioningExists makes a live call to AWS and answers the question a defender actually asked. When Terratest has no helper for what you need, aws.NewS3Client(t, region) and its siblings (NewEc2Client, NewIamClient, NewKmsClient and about a dozen more) hand you a configured SDK (Software Development Kit, the vendor's own client library) client so you can call the API yourself.

One naming note, because Terratest reached v1.0 in 2026. Every helper in this lesson now has a ...Context twin that takes a context.Context for cancellation and deadlines: InitAndApplyContext, DestroyContext, OutputContext, AssertS3BucketVersioningExistsContext, WaitUntilDeploymentAvailableContext. The short forms still compile and still work, because each one is a wrapper that passes context.Background(), but they carry a deprecation marker and your linter will say so. Existing suites are fine. New ones should reach for the Context versions.

terminal
# real credentials, sandbox account, real resources for about 50 seconds
go test -v -timeout 30m -count=1 -run TestS3BucketModule ./test/
output
=== RUN   TestS3BucketModule
=== PAUSE TestS3BucketModule
=== CONT  TestS3BucketModule
TestS3BucketModule 2026-07-22T09:52:03+01:00 retry.go:159: terraform [init -upgrade=false]
TestS3BucketModule 2026-07-22T09:52:03+01:00 logger.go:79: Running command terraform with args [init -upgrade=false]
TestS3BucketModule 2026-07-22T09:52:07+01:00 retry.go:159: terraform [apply -input=false -auto-approve -var name=acme-logs-2f9qk1 -var region=us-east-1 -lock=false]
TestS3BucketModule 2026-07-22T09:52:07+01:00 logger.go:79: Running command terraform with args [apply -input=false -auto-approve -var name=acme-logs-2f9qk1 -var region=us-east-1 -lock=false]
TestS3BucketModule 2026-07-22T09:52:11+01:00 logger.go:79: aws_s3_bucket.this: Creating...
TestS3BucketModule 2026-07-22T09:52:14+01:00 logger.go:79: aws_s3_bucket.this: Creation complete after 3s [id=acme-logs-2f9qk1]
TestS3BucketModule 2026-07-22T09:52:22+01:00 logger.go:79: Apply complete! Resources: 4 added, 0 changed, 0 destroyed.
TestS3BucketModule 2026-07-22T09:52:23+01:00 retry.go:159: terraform [output -no-color -json bucket_id]
TestS3BucketModule 2026-07-22T09:52:23+01:00 logger.go:79: Running command terraform with args [output -no-color -json bucket_id]
TestS3BucketModule 2026-07-22T09:52:26+01:00 retry.go:159: terraform [destroy -auto-approve -input=false -var name=acme-logs-2f9qk1 -var region=us-east-1 -lock=false]
TestS3BucketModule 2026-07-22T09:52:26+01:00 logger.go:79: Running command terraform with args [destroy -auto-approve -input=false -var name=acme-logs-2f9qk1 -var region=us-east-1 -lock=false]
TestS3BucketModule 2026-07-22T09:52:49+01:00 logger.go:79: Destroy complete! Resources: 4 destroyed.
--- PASS: TestS3BucketModule (49.31s)
PASS
ok  	github.com/acme/infra/test	49.88s

Forty-nine seconds and a fraction of a cent, for proof that the module deploys, that AWS accepted every argument, and that versioning is genuinely on. Dozens of tests like this carry most of your confidence. They are also the layer people under-invest in, because writing one means thinking about unique names, teardown, and which API call actually proves the point. Writing another end-to-end test feels like more coverage for less thought.

The Top: End to End, Kept Rare

At the peak sit tests that wire several modules into a working environment and check the promise a customer cares about, like an HTTPS endpoint (HyperText Transfer Protocol Secure, the encrypted version of the web's request-and-response protocol) returning 200, the status code that means 'here you go'. They are worth having and they are expensive: half an hour, real spend, and dozens of moving parts that can each flake. Keep them to a handful covering critical paths, not one per combination of variables. Because they are slow, this is where test_structure.RunTestStage earns its place, letting you re-enter a run without rebuilding the cluster.

test/platform_e2e_test.go
package test
 
import (
	"testing"
	"time"
 
	http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/k8s"
	"github.com/gruntwork-io/terratest/modules/terraform"
	test_structure "github.com/gruntwork-io/terratest/modules/test-structure"
)
 
func TestPlatformEndToEnd(t *testing.T) {
	workingDir := "../environments/staging"
 
	// Any stage is skipped when SKIP_<name> is set to anything non-empty.
	defer test_structure.RunTestStage(t, "teardown", func() {
		opts := test_structure.LoadTerraformOptions(t, workingDir)
		terraform.Destroy(t, opts)
	})
 
	test_structure.RunTestStage(t, "deploy", func() {
		opts := &terraform.Options{TerraformDir: workingDir}
		test_structure.SaveTerraformOptions(t, workingDir, opts)
		terraform.InitAndApply(t, opts) // VPC + EKS + app: 25 to 40 minutes
	})
 
	test_structure.RunTestStage(t, "validate", func() {
		opts := test_structure.LoadTerraformOptions(t, workingDir)
 
		// args: context name (empty = current), kubeconfig path, namespace
		kube := k8s.NewKubectlOptions("",
			terraform.Output(t, opts, "kubeconfig_path"), "shop")
		k8s.WaitUntilDeploymentAvailable(t, kube, "storefront", 30, 10*time.Second)
 
		// args: url, *tls.Config, expected status, expected body, retries, sleep.
		// The body check is an EXACT match after whitespace trimming.
		url := terraform.Output(t, opts, "app_url")
		http_helper.HttpGetWithRetry(t, url, nil, 200, "ok", 30, 10*time.Second)
	})
}

WaitUntilDeploymentAvailable polls thirty times, ten seconds apart, until Kubernetes reports the deployment healthy. That covers the gap between Terraform saying 'done' and the workload actually serving traffic. HttpGetWithRetry does the same for the front door, and it has a sharp edge worth knowing about. The body check is exact, not a substring search. Terratest trims leading and trailing whitespace and then compares the entire body to your string, so a health endpoint answering ok followed by a newline passes against "ok", while one answering {"status":"ok"} fails. Reach for http_helper.HttpGetWithRetryWithCustomValidation when you need a looser check. Passing nil for the *tls.Config argument means normal certificate verification, which is what you want; a test that skips verification would happily miss a broken certificate chain.

terminal
# full run, once, on a schedule
go test -v -timeout 45m -count=1 -run TestPlatformEndToEnd ./test/
 
# iterating on the checks only: keep the cluster, skip the slow parts
SKIP_deploy=true SKIP_teardown=true \
  go test -v -timeout 30m -count=1 -run TestPlatformEndToEnd ./test/
output
=== RUN   TestPlatformEndToEnd
TestPlatformEndToEnd 2026-07-22T11:04:02+01:00 logger.go:79: The 'SKIP_deploy' environment variable is set, so skipping stage 'deploy'.
TestPlatformEndToEnd 2026-07-22T11:04:02+01:00 logger.go:79: The 'SKIP_validate' environment variable is not set, so executing stage 'validate'.
TestPlatformEndToEnd 2026-07-22T11:04:02+01:00 retry.go:159: terraform [output -no-color -json kubeconfig_path]
TestPlatformEndToEnd 2026-07-22T11:04:02+01:00 logger.go:79: Running command terraform with args [output -no-color -json kubeconfig_path]
TestPlatformEndToEnd 2026-07-22T11:04:04+01:00 logger.go:79: Wait for deployment storefront to be provisioned.
TestPlatformEndToEnd 2026-07-22T11:04:04+01:00 logger.go:79: Wait for deployment storefront to be provisioned. returned an error: Deployment storefront is not available as 'Progressing' condition indicates that the Deployment is not complete, status: True, reason: ReplicaSetUpdated. Sleeping for 10s and will try again.
TestPlatformEndToEnd 2026-07-22T11:04:14+01:00 logger.go:79: Wait for deployment storefront to be provisioned.
TestPlatformEndToEnd 2026-07-22T11:04:15+01:00 retry.go:159: terraform [output -no-color -json app_url]
TestPlatformEndToEnd 2026-07-22T11:04:15+01:00 logger.go:79: Running command terraform with args [output -no-color -json app_url]
TestPlatformEndToEnd 2026-07-22T11:04:16+01:00 logger.go:79: HTTP GET to URL https://staging.shop.example.com/healthz
TestPlatformEndToEnd 2026-07-22T11:04:16+01:00 logger.go:79: Making an HTTP GET call to URL https://staging.shop.example.com/healthz
TestPlatformEndToEnd 2026-07-22T11:04:16+01:00 logger.go:79: The 'SKIP_teardown' environment variable is set, so skipping stage 'teardown'.
--- PASS: TestPlatformEndToEnd (14.22s)
PASS
ok  	github.com/acme/infra/test	14.67s

Fourteen seconds instead of forty minutes, because the cluster from the previous run is still up. Which is also the trap: that cluster is still billing you, and it will keep billing you all weekend if you forget. Skipping teardown is a loan, not a saving.

A timed-out test leaves the cloud running
go test defaults to a ten-minute timeout. When it fires, the testing package panics from a timer goroutine (a lightweight thread inside the Go process) that it started itself, dumps every stack and kills the process. Your defer terraform.Destroy sits on the test goroutine, so it never runs. A half-built EKS cluster, its NAT gateway and three load balancers keep charging you until somebody notices. That is why every command here carries an explicit -timeout 30m or -timeout 45m: set it longer than the slowest apply you expect, not longer than you feel like waiting. Back it with two habits. Tag every resource a test creates (Owner=terratest, TTL=2h, short for time to live) and run a sweeper such as cloud-nuke against the sandbox account on a schedule, because a CI job cancelled mid-apply leaks in exactly the same way.

Where a New Test Goes

Three questions, in order. Can I prove this from the configuration alone, before anything is created? Then it belongs at the base, as a validate rule, a policy check or a plan assertion. Does proving it require a provider to accept the request and answer back, like a bucket name being available or versioning really being enabled? Then it belongs in the middle, as one module applied and destroyed. Does it only break when two modules disagree about a contract, like the cluster's security group refusing to let the load balancer through? Only then does it belong at the top.

Most things people push to the top belong in the middle, and most things people push to the middle belong at the base. Before you write a fourth end-to-end test, name the module test that would have caught the same bug in fifty seconds for a fraction of a cent. There almost always is one.

Quick check
01Why do plan-level Terratest checks belong at the wide base of the pyramid rather than being an afterthought?
Incorrect — during a plan the provider only validates arguments against its own schema, offline, which is exactly why an apply can still fail after a green plan.
Correct — the base is wide because each check is nearly free in both time and money.
Incorrect — terraform.Output shells out to terraform output against applied state, which is a middle-layer and top-layer activity.
Incorrect — any test that applies still owns its own teardown, no matter what the plan layer proved.
02Terratest suites almost always run go test with -count=1. What does that flag prevent?
Incorrect — the default is a single run; -count sets how many, and 1 is already that number.
Incorrect — parallelism comes from t.Parallel() and the -parallel flag, not -count.
Incorrect — locking is controlled by terraform's own -lock flag and has nothing to do with the Go test runner.
Correct — -count=1 is the documented way to defeat the test cache, which cannot see files the terraform subprocess read or changed.
03A nightly job runs go test -v -run TestPlatformEndToEnd ./test/ with no -timeout flag. After ten minutes the log ends with panic: test timed out after 10m0s and a wall of goroutine stacks. What is the state of the AWS account, and what do you do first?
Correct — the panic comes from the timer goroutine the testing package starts, so the deferred cleanup registered on the test goroutine never gets a chance to run.
Incorrect — Terraform has no rollback, and every resource created before the panic is still there and still billing.
Incorrect — a panic on a goroutine Terratest does not own cannot be recovered by Terratest, and the whole process dies with it.
Incorrect — there is no teardown in the log at all, because the process died before reaching it.

Try this

Run terraform -chdir=modules/s3-bucket init -backend=false on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a green plan is not a green apply. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Parallelism & fixturesCost, safety & isolation

Related

What InSpec is: compliance as codeInSpecThe secrets-in-Git problem & SOPSSOPS