Terratest in CI/CD

Credentials, cleanup, gating.

A CI runner is a borrowed workshop. CI/CD (continuous integration and continuous delivery, the automation that builds, tests and ships your code whenever someone pushes) hands your test a machine it has never seen before, gives it a temporary badge at the door, lets it build real things with live tools, and expects the room back exactly as it was found before the next tenant walks in. Three duties fall out of that arrangement. Prove who you are without leaving a key under the mat. Guarantee the room is empty when you leave, even when the job gets shot mid-swing. Decide whose work is allowed through the door at all.

Your test code from the earlier lessons barely changes. What changes is who holds the cleanup responsibility, how the runner gets cloud credentials, and what happens when the process dies before your deferred destroy ever fires. Get those three wrong and the bill arrives before the bug report. A stranger assumes your deploy role. A NAT gateway hums along for six weeks because somebody cancelled a job at an unlucky second.

Credentials: A Badge At The Door

The instinct is to paste a long-lived access key into the CI secret store and move on. Resist it. A static key is a spare key under the doormat. It works for anyone who finds it. It works at three in the morning. It keeps working long after the person who hid it there has left the company. It gets copied into a Dockerfile, echoed by a debugging step somebody added in a hurry, and read by every workflow in the repository, including one a contributor added last Tuesday.

The replacement is OIDC (OpenID Connect, a standard way for one system to vouch for an identity to another system) federation. Think of it as a receptionist who phones your employer to confirm you really work there, then prints you a badge that stops working at lunchtime. Your CI provider signs a short-lived JWT (JSON Web Token, a small tamper-evident blob of statements about who is asking) describing the exact run: which repository, which branch, which environment. AWS is configured to trust GitHub as an identity issuer, checks the signature, checks the statements against a condition you wrote, and hands back temporary credentials for one IAM (identity and access management) role through STS (the security token service, the part of AWS that mints temporary credentials). Nothing is stored. Nothing needs rotating. There is nothing in the secret store for an attacker to steal, because the secret store is empty.

name: terratest-e2e
 
on:
  schedule:
    - cron: '0 3 * * *'          # nightly, when a 40 minute run is acceptable
  workflow_dispatch:            # and on demand, before a merge you feel nervous about
 
permissions:
  id-token: write               # lets the job ask GitHub for an OIDC token
  contents: read                # everything else stays read-only
 
jobs:
  e2e:
    runs-on: ubuntu-latest
    timeout-minutes: 90         # hard wall-clock stop for the whole job
    environment: terratest-e2e  # protected env: human approval + its own OIDC subject
    concurrency:
      group: terratest-e2e      # two runs would fight over the same resource names
      cancel-in-progress: false # cancelling mid-apply is exactly how you leak a VPC
    steps:
      - uses: actions/checkout@v5
      - uses: actions/setup-go@v5
        with:
          go-version-file: go.mod   # one source of truth (Go 1.26), no drift in CI
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/terratest-ci
          role-session-name: gha-${{ github.run_id }}
          role-duration-seconds: 7200   # must be <= the role's MaxSessionDuration
          aws-region: us-east-1

That YAML (the indented text format GitHub uses for workflow files) is the polite half. The half that actually decides anything lives in AWS, on the role, in its trust policy. The trust policy is the lock. The workflow is only the person knocking. It names GitHub's issuer, then adds conditions on the statements inside the token. Two statements matter. The aud (audience) claim says who the token was minted for. The sub (subject) claim is the one that names your repository, and your branch or environment.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": [
          "repo:acme/infra:environment:terratest-e2e",
          "repo:acme/infra:ref:refs/heads/main"
        ]
      }
    }
  }]
}

Never trust the file in your repository to tell you what the live role accepts. Terraform can apply a template with an empty variable. Somebody can hand-edit the role in the console during an incident and forget to say so. Both leave your committed JSON looking perfectly correct. Read the lock itself. The same call also shows the role's session ceiling, which turns out to be the other half of this story.

# Who may assume this role, and for how long?
aws iam get-role --role-name terratest-ci \
  --query '{MaxSession: Role.MaxSessionDuration,
            Cond: Role.AssumeRolePolicyDocument.Statement[0].Condition}'

{
    "MaxSession": 3600,
    "Cond": {
        "StringEquals": {
            "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
            "token.actions.githubusercontent.com:sub": [
                "repo:acme/infra:environment:terratest-e2e",
                "repo:acme/infra:ref:refs/heads/main"
            ]
        }
    }
}

The condition is tight. The ceiling is wrong. The workflow asks for role-duration-seconds: 7200 while the role tops out at the default 3600, so the very first step of the job dies with ValidationError: The requested DurationSeconds exceeds the MaxSessionDuration set for this role. Raise it with aws iam update-role --role-name terratest-ci --max-session-duration 7200. Care about this beyond the error message, because a badge that expires mid-run expires during the part of the run that happens last. That part is your destroy.

Cleanup That Survives A Killed Job

defer terraform.Destroy(t, opts) is a promise made by a running process. Go keeps it when the function returns, and it keeps it when an assertion calls t.Fatal, because t.Fatal unwinds the goroutine (Go's word for a lightweight thread of execution) rather than killing the program outright. Go cannot keep it when the process is killed from outside. CI kills processes all day long. Somebody clicks Cancel. The job hits timeout-minutes. A self-hosted runner sitting on cheap reclaimable capacity gets taken back by the cloud. A machine reboots for patching. In every one of those cases the test binary gets no chance to unwind, and whatever the apply built stays built, billing by the hour, with nobody watching.

The fix is to stop keeping the whole lifecycle inside one process. test_structure.RunTestStage wraps each phase in a named stage that a SKIP_<stage> environment variable can switch off, which you already met as a local iteration trick. In CI it earns its keep differently. Run each stage as its own step, and mark the teardown step if: always(), which GitHub evaluates as true on failure and on cancellation alike. Two honest limits on that. GitHub gives the runner a grace period after a cancel rather than unlimited time, so a ten minute destroy can still get cut in half. And if the machine itself disappears, no step runs at all. The tag sweep in the next section is the backstop for both.

//go:build e2e
 
// Compiled only when -tags=e2e is passed, so the fast PR lane never sees it.
package test
 
import (
	"fmt"
	"os"
	"testing"
	"time"
 
	http_helper "github.com/gruntwork-io/terratest/modules/http-helper"
	"github.com/gruntwork-io/terratest/modules/terraform"
	test_structure "github.com/gruntwork-io/terratest/modules/test-structure"
)
 
func TestVpcE2E(t *testing.T) {
	// A FIXED path, not CopyTerraformFolderToTemp: every CI step is its own
	// process, and all of them must find the same saved options file.
	dir := "../examples/vpc"
 
	// Registered first, so it is already scheduled if the deploy blows up.
	defer test_structure.RunTestStage(t, "teardown", func() {
		terraform.Destroy(t, test_structure.LoadTerraformOptions(t, dir))
	})
 
	test_structure.RunTestStage(t, "deploy", func() {
		runID := os.Getenv("GITHUB_RUN_ID") // identical in every step of one run
		if runID == "" {
			runID = fmt.Sprintf("local%d", time.Now().Unix())
		}
		opts := &terraform.Options{
			TerraformDir: dir,
			Vars: map[string]interface{}{
				"name":   fmt.Sprintf("tt-%s", runID),
				"run_id": runID, // lands as a tag on every resource
			},
			NoColor: true, // CI logs are plain text; drop the escape codes
		}
		// Writes ../examples/vpc/.test-data/TerraformOptions.json
		test_structure.SaveTerraformOptions(t, dir, opts)
		terraform.InitAndApply(t, opts)
	})
 
	test_structure.RunTestStage(t, "validate", func() {
		opts := test_structure.LoadTerraformOptions(t, dir)
		url := fmt.Sprintf("http://%s", terraform.Output(t, opts, "alb_dns_name"))
		http_helper.HttpGetWithRetry(t, url, nil, 200, "ok", 30, 10*time.Second)
	})
}

      - name: deploy
        env: { SKIP_validate: "true", SKIP_teardown: "true" }
        run: go test -v -timeout 30m -count=1 -tags=e2e -run TestVpcE2E ./test/...
 
      - name: validate
        env: { SKIP_deploy: "true", SKIP_teardown: "true" }
        run: go test -v -timeout 30m -count=1 -tags=e2e -run TestVpcE2E ./test/...
 
      - name: teardown
        if: always()        # true on failure AND on cancellation
        env: { SKIP_deploy: "true", SKIP_validate: "true" }
        run: go test -v -timeout 30m -count=1 -tags=e2e -run TestVpcE2E ./test/...

If the apply fails, that step exits non-zero, GitHub skips validate because its default condition is success, and teardown still runs. That is the whole point of the split. -count=1 is on all three steps because Go caches passing test results: without it, a nightly run whose Go inputs have not changed can replay a cached pass in milliseconds, deploying nothing and destroying nothing while every step reports green. Run the teardown step by hand before you trust it, though, because a cleanup path nobody has ever executed is a guess wearing a uniform. It is the same command with two skips set, and the log tells you exactly which stages it decided to run.

SKIP_deploy=true SKIP_validate=true \
  go test -v -timeout 30m -count=1 -tags=e2e -run TestVpcE2E ./test/...

=== RUN   TestVpcE2E
TestVpcE2E 2026-07-22T04:12:11Z test_structure.go:38: The 'SKIP_deploy' environment variable is set, so skipping stage 'deploy'.
TestVpcE2E 2026-07-22T04:12:11Z test_structure.go:38: The 'SKIP_validate' environment variable is set, so skipping stage 'validate'.
TestVpcE2E 2026-07-22T04:12:11Z test_structure.go:38: The 'SKIP_teardown' environment variable is not set, so executing stage 'teardown'.
TestVpcE2E 2026-07-22T04:12:11Z logger.go:66: Running command terraform with args [destroy -auto-approve -input=false -no-color -var name=tt-16493822107 -var run_id=16493822107 -lock=false]
TestVpcE2E 2026-07-22T04:15:44Z logger.go:66: Destroy complete! Resources: 14 destroyed.
--- PASS: TestVpcE2E (213.09s)
PASS
ok  	github.com/acme/infra/test	213.11s

Two details in that command line are worth a second look. -no-color is there because you set NoColor: true, and Terratest adds the flag for every Terraform command that supports it. -lock=false is there because Terratest's Options.Lock field defaults to false, so state locking is off unless you turn it on. Harmless for a throwaway example with local state. A genuine hazard the day two runs share a remote state file, so set Lock: true if yours do. One more constraint: this teardown works because the deploy step already ran init in the same workspace on the same runner. Split the stages into separate GitHub *jobs* and you get a clean machine each time, which means shipping .test-data between them as an artifact and calling terraform.Init before the destroy.

Tag Everything, Then Sweep Behind Yourself

Hotels do not rely on guests returning keys. They also employ a housekeeper who opens every room at eleven and clears out whatever got left behind. Your reaper is that housekeeper: a scheduled job, running completely outside the test, that finds infrastructure the tests created and deletes anything too old to belong to a live run. Separate is the entire point. Anything running inside the test process shares the test process's fate. So stamp every resource at apply time, and let the housekeeper recognise your rooms on sight. The AWS provider will do the stamping for you.

provider "aws" {
  region = "us-east-1"
 
  # Stamps every resource this module creates, with no edits to any
  # resource block. Needs AWS provider >= 3.38, so anything current.
  # Do NOT put timestamp() in here: it produces a perpetual diff. The
  # sweep reads each resource's real creation time from the API instead.
  default_tags {
    tags = {
      Owner          = "terratest"
      TerratestRunId = var.run_id
    }
  }
}

After a job finishes, ask the tagging API (application programming interface, the machine-facing door into a service) whether anything still answers to that run id. A clean run returns nothing at all. A cancelled one returns a bill.

# ARNs come back tab separated on one line, so split them for reading.
aws resourcegroupstaggingapi get-resources \
  --region us-east-1 \
  --tag-filters Key=TerratestRunId,Values=16493822107 \
  --query 'ResourceTagMappingList[].ResourceARN' --output text | tr '\t' '\n'

arn:aws:ec2:us-east-1:111122223333:vpc/vpc-0f3a19c47be2d5610
arn:aws:ec2:us-east-1:111122223333:natgateway/nat-04b7c8e19f2a3d6b0
arn:aws:elasticloadbalancing:us-east-1:111122223333:loadbalancer/app/tt-16493822107/9d1c8f0a2b3e4d5f

Three lines, roughly two dollars a day. Most of that is the NAT gateway (network address translation, the box that lets machines on a private network reach the internet) at about four and a half cents an hour, plus the load balancer at a bit over two cents, both charged whether or not a single packet moves. Two things narrow what this query can see. It is per region, so the sweep runs once for every region you test in. And default_tags never reaches resources AWS creates on your behalf, such as the network interfaces an EKS cluster (Elastic Kubernetes Service, Amazon's managed Kubernetes) attaches or the disks an autoscaling group launches, which is why a good sweep also matches on the tt- name prefix and on age. Gruntwork's cloud-nuke is built for this exact chore, and cloud-nuke aws --region us-east-1 --older-than 24h --force makes a reasonable nightly cron. Run it once with --dry-run first and read every line. Point it only at the dedicated sandbox account from the cost lesson, because its default behaviour is to delete every resource it can reach, and a wrong AWS_PROFILE in that cron turns a housekeeper into a demolition crew.

Never Hand The Badge To A Stranger

Here is the attack that has emptied more cloud accounts than any leaked key. GitHub deliberately withholds secrets, and refuses to grant id-token: write, for workflows triggered by a pull_request from a fork. That is correct behaviour and you want it. What people do next is reach for pull_request_target, because it runs in the context of the base repository with full permissions and the secrets flow again. Then they add a checkout of the pull request's head commit so the tests actually run against the proposed change. Those two decisions together mean a stranger's code, from a fork nobody has read, executing on a runner that holds your deploy role.

# The "pwn request". Every line is individually reasonable.
on: pull_request_target          # runs with the BASE repo's secrets
permissions:
  id-token: write                # ...and can mint your OIDC token
jobs:
  test:
    steps:
      - uses: actions/checkout@v5
        with:
          ref: ${{ github.event.pull_request.head.sha }}   # a stranger's code
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/terratest-ci
      - run: go test -v -timeout 30m -tags=e2e ./test/...  # ...now runs it

The attacker never has to touch your Terraform. A one-line TestMain, or a package-level init() function in any file under test/, runs before a single assertion of yours does, and it inherits the whole environment: credentials, tokens, the lot. It can create an IAM user with a long-lived key and be gone before the job reaches your first test. A new entry in go.mod does the same trick from further away. The rule is short. Fork pull requests never get cloud credentials, ever. Real applies run from branches inside the repository, or from a protected environment where a human read the exact commit and approved it, which is the second reason to pin the OIDC subject to environment: rather than to a branch.

Two settings deserve a look while you are in there. Under Actions, set fork pull request approval to require it for all external contributors, not the default of first-time contributors only, otherwise someone who landed a typo fix last year runs unattended forever. And on private repositories there are toggles to send secrets and write tokens to workflows from forks. Leave both off. Separately, pin third-party actions to a full commit SHA (the forty-character fingerprint of one exact commit) rather than a moving tag like @v4. Tags can be repointed at new code by whoever owns the action, and that has already been used in the wild to dump runner memory, secrets included, into public build logs.

One Fast Lane, One Slow Lane

A test gates a merge only when the branch is protected and the job is listed as a required status check. Anything else is decoration, and people click past decoration. But you cannot block every pull request for forty minutes while real infrastructure gets built and torn down. Contributors will find a way around a gate that eats their afternoon, and the way around it is usually a direct push. So split the suite into two lanes that make different promises.

The fast lane is the required check: terraform validate, tflint, a policy scan, and the plan-level Terratest assertions from the strategy lesson. It touches no cloud API and finishes in about three minutes. Kubernetes work belongs here too, because a kind cluster (Kubernetes in Docker, a complete cluster running as containers on the runner) comes up in roughly thirty seconds, which means tests written against Terratest's k8s and helm modules run for free on every pull request. The slow lane is everything that really applies, and it runs nightly or on workflow_dispatch. Go build tags are the switch between them, and you can prove which tests each lane compiles without running any of them.

# What does the PR lane actually compile?
go test -list '.*' ./test/...
 
# And with the e2e tag switched on?
go test -tags=e2e -list '.*' ./test/...

TestPlanVpcIsPrivate
TestNamingConventions
TestHelmChartOnKind
ok  	github.com/acme/infra/test	0.019s
 
TestPlanVpcIsPrivate
TestNamingConventions
TestHelmChartOnKind
TestVpcE2E
TestEksNodeGroup
ok  	github.com/acme/infra/test	0.021s

Keep Secrets Out Of The Job Log

go test -v streams everything, and Terratest logs each line of Terraform's output as it arrives. That is exactly what you want at two in the morning when a destroy has failed, and exactly what you do not want the moment a test reads a generated database password. terraform.Output runs terraform output -no-color -json <name>, and the -json form prints sensitive values in full rather than replacing them with a placeholder. So the password lands in the job log as plain text. GitHub masks only strings it already knows are secrets, and a value born inside this run was never registered. On a public repository that log is world readable, and deleting it afterwards does not reach the caches, the forks, or the notification emails already sent. The only real remedy is rotating the value, which is a bad afternoon. Keep it out of the log in the first place.

package test
 
import (
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/logger"
	"github.com/gruntwork-io/terratest/modules/terraform"
	test_structure "github.com/gruntwork-io/terratest/modules/test-structure"
)
 
func TestDbAcceptsGeneratedPassword(t *testing.T) {
	opts := test_structure.LoadTerraformOptions(t, "../examples/rds")
 
	// Copy the options and swap in a silent logger for this one read.
	// Vars and other fields come along; only the logging changes.
	quiet := *opts
	quiet.Logger = logger.Discard
 
	host := terraform.Output(t, opts, "db_endpoint")        // fine to log
	password := terraform.Output(t, &quiet, "db_password")  // never hits the log
 
	// Assert on what the secret can do. Never print the secret.
	assertCanConnect(t, host, password)
}

If you truly cannot avoid a runtime secret passing through the runner, register it with GitHub's add-mask workflow command so later log lines get starred out. Treat that as a seatbelt, not a plan: masking matches literal strings, so a base64 or JSON-escaped copy of the same value sails straight through. While you are in there, leave TF_LOG unset in CI. Setting it to DEBUG prints request and response bodies from every provider call, which is a fine way to debug on your laptop and a fast way to publish a token you did not know was in flight.

The honest trade-off in this whole design is the delay before you hear bad news. The fast lane keeps contributors moving, and the price is that a change which breaks a real apply can sit on main for up to a day before the nightly run notices. Choose that deliberately rather than by accident: have the nightly job open an issue and page the module owner on failure, and let anyone trigger the slow lane on demand before a merge that worries them. Then break the reaper on purpose, once. Start an e2e run, cancel it halfway through the deploy stage, wait for the sweep, and query the tagging API for that run id. If something still comes back, you found out for two dollars instead of on the invoice.

Try this

Run go test -v -timeout 30m -count=1 -tags=e2e -run TestVpcE2E ./test/... on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: an aud-only trust policy trusts all of GitHub. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related