Your test calls k8s.WaitUntilServiceAvailable on a Service of type ClusterIP and it returns on the first attempt. What has it proved?

The Service object exists in the namespace, and nothing more. For ClusterIP, NodePort and ExternalName the availability check returns true as soon as the object can be fetched.

You point http_helper.HttpGetWithRetry(t, url, nil, 200, "Welcome to nginx!", 30, 5*time.Second) at the stock nginx welcome page through a working tunnel. The page returns 200 in a browser, but the test burns all 30 retries and fails. Why?

HttpGetWithRetry compares the entire response body to the string you pass, and the welcome page is a full HTML document. That argument is an exact whole-body match, so anything but a byte-for-byte identical body keeps failing.

A test installs chart v1 with no --wait flags, probes it green, calls helm.Upgrade to v2, then calls k8s.WaitUntilPodAvailable with a pod name captured before the upgrade, then probes again and gets 200. The v2 pod crash-loops the entire time, yet the test passes. What happened, and what do you change?

The captured name still belongs to a healthy v1 pod that is Ready and still a Service endpoint, so both the wait and the probe hit v1; switch to WaitUntilDeploymentAvailable and re-list pods after the upgrade. A rolling upgrade keeps old pods serving, a stale pod name resolves to one of them, and only the Deployment's Progressing condition notices that the new ReplicaSet never became available.

Courses TerratestTesting Kubernetes & Helm

Testing Kubernetes & Helm

Deploy and probe workloads.

Advanced12 min · lesson 6 of 12

A health inspector does not read the menu and go home. They walk into the kitchen, open the fridge, and order a plate off the line. Terratest is that inspector for Kubernetes. It applies your manifests or installs your Helm chart against a live cluster, waits for the workload to come up, then sends it a real request over a real socket. If the pod crash-loops, or the Service answers with a 502, the test fails. YAML (the indented text format Kubernetes reads) that merely parses gets no credit here.

Two packages do the work. modules/k8s wraps kubectl and the Kubernetes API (application programming interface, the one control desk every change to the cluster has to walk up to). modules/helm wraps the helm binary, so install, upgrade, delete and template behave exactly as they do in your shell. Everything else is ordinary Go: a test is a function, an assertion is testify's require, cleanup is defer. The retry loops inside every Wait helper are covered in tt-assert. What matters here is the live loop, and knowing precisely how much each step proves. Most flaky Kubernetes tests are tests that trusted a wait which never checked the thing the author assumed it checked.

A Cluster You Are Allowed to Break

Start with a cluster you would not mind destroying, because these tests create and delete real objects. kind (Kubernetes in Docker) runs a whole control plane inside Docker containers on your laptop or a CI runner (CI is continuous integration, the machine that builds and tests every push). It starts in about thirty seconds, costs nothing, and dies on command, which makes it the right default for chart and manifest tests. A managed cluster (Amazon EKS, Google GKE, Azure AKS) bills by the hour instead. Worse, any Service you create there with type LoadBalancer provisions a real cloud load balancer with a real public address. That is money on the meter, and for the whole life of the test it is also an internet-facing door into a workload nobody is watching. Keep test Services on ClusterIP and reach them through a port-forward.

terminal

# a throwaway cluster on the local Docker daemon: free, about 30 seconds
kind create cluster --name terratest --image kindest/node:v1.34.0
 
kubectl config get-contexts

output

Creating cluster "terratest" ...
 ✓ Ensuring node image (kindest/node:v1.34.0)
 ✓ Preparing nodes
 ✓ Writing configuration
 ✓ Starting control-plane
 ✓ Installing CNI
 ✓ Installing StorageClass
Set kubectl context to "kind-terratest"
You can now use your cluster with:
 
kubectl cluster-info --context kind-terratest
 
Thanks for using kind!
 
CURRENT   NAME             CLUSTER          AUTHINFO         NAMESPACE
*         kind-terratest   kind-terratest   kind-terratest

KubectlOptions is the address label on an envelope: which building, which set of keys, which room. The three arguments to k8s.NewKubectlOptions are the kubeconfig context name (the building), the path to the kubeconfig file (the keys, where an empty string means $KUBECONFIG, falling back to ~/.kube/config), and the namespace (the room). Naming the context explicitly matters more than it looks. An empty context string means whatever kubectl happens to be pointed at right now, and that is exactly how a test suite ends up applying manifests, and deleting namespaces, on a shared staging cluster because someone ran kubectl config use-context an hour earlier. Terratest stamps those three values onto every kubectl command it runs, which is why the log lines below open with --context and --namespace. Read the context name from an environment variable, because in CI the cluster is named after the job and the context name changes with it.

manifests/nginx.yaml

# one Deployment, one ClusterIP Service. Nothing here is reachable from outside.
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx
spec:
  replicas: 2
  selector:
    matchLabels: { app: nginx }
  template:
    metadata:
      labels: { app: nginx }
    spec:
      automountServiceAccountToken: false
      containers:
        - name: nginx
          image: nginxinc/nginx-unprivileged:1.28-alpine
          ports: [{ containerPort: 8080 }]
          readinessProbe:
            httpGet: { path: /, port: 8080 }
            periodSeconds: 2
          securityContext:
            runAsNonRoot: true
            allowPrivilegeEscalation: false
            capabilities: { drop: ["ALL"] }
---
apiVersion: v1
kind: Service
metadata:
  name: nginx
spec:
  type: ClusterIP
  selector: { app: nginx }
  ports:
    - port: 80
      targetPort: 8080

test/nginx_manifest_test.go

package test
 
import (
	"os"
	"strings"
	"testing"
	"time"
 
	"github.com/gruntwork-io/terratest/modules/k8s"
	"github.com/gruntwork-io/terratest/modules/random"
	"github.com/stretchr/testify/require"
)
 
// The cluster is named per job in CI, so the context name cannot be hardcoded.
func kubeContext() string {
	if ctx := os.Getenv("KUBE_CONTEXT"); ctx != "" {
		return ctx
	}
	return "kind-terratest"
}
 
func TestNginxManifest(t *testing.T) {
	t.Parallel()
 
	// one namespace per run, e.g. "tt-a1b2c3"
	namespace := "tt-" + strings.ToLower(random.UniqueID())
 
	// context, kubeconfig path ("" = $KUBECONFIG or ~/.kube/config), namespace
	options := k8s.NewKubectlOptions(kubeContext(), "", namespace)
 
	k8s.CreateNamespace(t, options, namespace)
	defer k8s.DeleteNamespace(t, options, namespace)
 
	k8s.KubectlApply(t, options, "../manifests/nginx.yaml") // kubectl -n <ns> apply -f
 
	// structural: the Service object exists
	k8s.WaitUntilServiceAvailable(t, options, "nginx", 30, 5*time.Second)
	// stronger: the rollout finished and the new pods went Ready
	k8s.WaitUntilDeploymentAvailable(t, options, "nginx", 30, 5*time.Second)
 
	svc := k8s.GetService(t, options, "nginx")
	require.Equal(t, "ClusterIP", string(svc.Spec.Type))
}

terminal

go test -v -timeout 30m -run TestNginxManifest ./test/

output

=== RUN   TestNginxManifest
=== PAUSE TestNginxManifest
=== CONT  TestNginxManifest
TestNginxManifest 2026-07-22T09:12:03Z logger.go:79: Configuring Kubernetes client using config file /home/dev/.kube/config with context kind-terratest
TestNginxManifest 2026-07-22T09:12:03Z logger.go:79: Running command kubectl with args [--context kind-terratest --namespace tt-a1b2c3 apply -f ../manifests/nginx.yaml]
TestNginxManifest 2026-07-22T09:12:04Z logger.go:79: deployment.apps/nginx created
TestNginxManifest 2026-07-22T09:12:04Z logger.go:79: service/nginx created
TestNginxManifest 2026-07-22T09:12:04Z retry.go:159: Wait for service nginx to be provisioned.
TestNginxManifest 2026-07-22T09:12:04Z logger.go:79: Service is now available
TestNginxManifest 2026-07-22T09:12:04Z retry.go:159: Wait for deployment nginx to be provisioned.
TestNginxManifest 2026-07-22T09:12:04Z retry.go:173: Wait for deployment nginx to be provisioned. returned an error: Deployment nginx is not available as 'Progressing' condition indicates that the Deployment is not complete, status: True, reason: ReplicaSetUpdated, message: ReplicaSet "nginx-7d9c5b46f8" is progressing.. Sleeping for 5s and will try again.
TestNginxManifest 2026-07-22T09:12:09Z retry.go:159: Wait for deployment nginx to be provisioned.
TestNginxManifest 2026-07-22T09:12:09Z logger.go:79: Deployment is now available
--- PASS: TestNginxManifest (6.44s)
PASS
ok      github.com/acme/infra/test      6.71s

One note on reading those lines. The file and line number in each one points inside Terratest, at the code that printed the message, not at anything in your test. k8s.NewKubectlOptions leaves the Logger field unset, so nearly everything Terratest logs for you falls through to the same line of logger.go, which is why so many different messages here carry logger.go:79. The retry loop logs by a different route and keeps its own position. These numbers come from Terratest v1.0.1 and they move on every upgrade, so treat them as a hint about which package is talking, and never assert on them.

What Each Wait Actually Proves

Look at the timestamps. The Service wait returned instantly. The Deployment wait needed a retry, and told you why: the Progressing condition still said ReplicaSetUpdated, not NewReplicaSetAvailable. That gap is the whole lesson. Applying a manifest returns as soon as the API server has written the object down. Nothing has been scheduled, no image has been pulled, no process has started. So every apply needs a wait, and the wait you pick decides what your test actually verifies.

WaitUntilServiceAvailable polls until the Service object can be fetched, and for a ClusterIP, NodePort or ExternalName Service that is all it ever checks. It does not look at Endpoints. It does not look at pods. Only for type LoadBalancer does it wait for something with weight behind it, namely a non-empty ingress list on the Service status, which is the cloud saying the load balancer finally has an address you could dial. On a ClusterIP Service, that call is a spelling check on the name.

WaitUntilPodAvailable is stronger than its reputation, and plenty of blog posts get this backwards. It checks that the pod has a status entry for every container in its spec, that each of those containers reports both Ready and Started, and that the pod phase is Running. A container stuck failing its readiness probe forever will not pass it. The real catch is the argument: you have to hand it a pod name. Pod names are generated fresh on every rollout, so a name you captured before an upgrade points at an old pod that is perfectly healthy and tells you nothing about the version you were trying to ship.

WaitUntilDeploymentAvailable is the one to reach for, and the reason is boring and practical: you name the Deployment, and the Deployment name does not change when pods do. It reads the Deployment's Progressing condition and returns only when that condition is True with reason NewReplicaSetAvailable, which is Kubernetes' own phrasing for "the new ReplicaSet rolled out and its pods went Ready". That covers readiness probes, minReadySeconds, and the rollout as a whole. It still cannot tell you the app returns the right bytes, which is why the probe later in this lesson exists.

Four Rungs of Evidence, Cheapest First

Free, no cluster

helm.RenderTemplate

runs helm template locally

helm.UnmarshalK8SYaml

assert typed fields, not text

The object exists

WaitUntilServiceAvailable

ClusterIP: true the moment it exists

k8s.GetService / GetDeployment

fetch the object, assert its fields

The rollout finished

WaitUntilDeploymentAvailable

Progressing reason NewReplicaSetAvailable

WaitUntilPodAvailable

every container Ready, but you must name the pod

It actually serves

NewTunnel + custom-validation GET

real status, real body, real socket

RunKubectlAndGetOutputE + require.Error

the deny path still denies

Each zone costs more and proves more than the one to its left. Run them in that order, and never stop before the last one.

Render the Chart Before It Touches the Cluster

A chart is a recipe, not dinner. The thing your cluster admits is the cooked output, and the two drift apart every time somebody edits a values file or bumps a dependency. helm.RenderTemplate runs helm template locally with your values and hands back that rendered YAML as a string. No cluster, no credentials, no bill, under a second. Each entry in the templateFiles argument becomes one --show-only flag, so you get exactly the document you asked for instead of every manifest in the chart glued together. Terratest resolves the chart directory to an absolute path before handing it to helm, which is why the log shows a full path where your code passed a relative one. helm.UnmarshalK8SYaml then parses the result into a real Kubernetes Go struct, so you assert on typed fields rather than grepping for a string.

This is where your security assertions belong. Charts pull subcharts from remote repositories, and a version bump you did not read can change defaults you never inspected. Nothing stops a values edit, or a friendly-looking pull request, from setting hostNetwork: true, quietly dropping a securityContext block, remounting the ServiceAccount token (the credential Kubernetes drops into a pod so the pod can call the API), or adding a ClusterRoleBinding to cluster-admin. A rendered-manifest test catches all of that on the pull request, in the same second the change is proposed, before any cluster has been asked to accept it.

test/chart_render_test.go

package test
 
import (
	"testing"
 
	"github.com/gruntwork-io/terratest/modules/helm"
	"github.com/stretchr/testify/require"
	appsv1 "k8s.io/api/apps/v1"
)
 
func TestChartRendersSafely(t *testing.T) {
	t.Parallel()
 
	// no KubectlOptions at all: this never speaks to a cluster
	opts := &helm.Options{
		SetValues: map[string]string{
			"replicaCount": "2",
			"image.tag":    "1.28-alpine",
		},
	}
 
	// each entry in templateFiles becomes `--show-only <file>`
	out := helm.RenderTemplate(t, opts, "../charts/nginx", "render-check",
		[]string{"templates/deployment.yaml"})
 
	var deployment appsv1.Deployment
	helm.UnmarshalK8SYaml(t, out, &deployment)
 
	require.Equal(t, int32(2), *deployment.Spec.Replicas)
 
	pod := deployment.Spec.Template.Spec
	require.False(t, pod.HostNetwork, "pod asks for the host network")
	require.NotNil(t, pod.AutomountServiceAccountToken)
	require.False(t, *pod.AutomountServiceAccountToken, "API token mounted into the pod")
 
	sc := pod.Containers[0].SecurityContext
	require.NotNil(t, sc, "container has no securityContext block")
	require.True(t, sc.RunAsNonRoot != nil && *sc.RunAsNonRoot, "container may run as root")
	require.True(t, sc.AllowPrivilegeEscalation != nil && !*sc.AllowPrivilegeEscalation,
		"privilege escalation is allowed")
}

terminal

# no cluster, no cloud credentials: safe to run on every pull request
go test -v -timeout 30m -run TestChartRendersSafely ./test/

output

=== RUN   TestChartRendersSafely
=== PAUSE TestChartRendersSafely
=== CONT  TestChartRendersSafely
TestChartRendersSafely 2026-07-22T09:04:11Z logger.go:79: Running command helm with args [template --set image.tag=1.28-alpine --set replicaCount=2 --show-only templates/deployment.yaml render-check /home/dev/infra/charts/nginx]
    chart_render_test.go:34: 
        	Error Trace:	/home/dev/infra/test/chart_render_test.go:34
        	Error:      	Expected value not to be nil.
        	Messages:   	container has no securityContext block
        	Test:       	TestChartRendersSafely
--- FAIL: TestChartRendersSafely (0.31s)
FAIL
FAIL    github.com/acme/infra/test      0.42s
FAIL

Note the two --set flags came out alphabetically sorted, because Terratest sorts the keys before building the command. That is deliberate, and it means the same options produce the same command every time, which makes these logs diffable. Note the bigger thing too: that failure took a third of a second and cost nothing. The same mistake found by the live test would have cost a cluster, four minutes of pipeline time, and a container running as UID 0 (the root user id) with a mounted API token for as long as the test ran. Render first. Every time.

Install, Upgrade, and the Flags That Matter

helm.Options is your command line as a struct. SetValues is --set. SetStrValues is --set-string, which you want for anything that looks like a number but is not: an image tag of 1.30 passed through --set arrives as 1.3, and your pod pulls a tag that does not exist. SetFiles is --set-file, and it exists so a certificate or token lands in the release from a file instead of being printed into your test log. ExtraArgs is the escape hatch, a map keyed by helm subcommand (install, upgrade, delete, rollback) whose values are appended to that exact command. Two entries earn their place on day one. ExtraArgs["install"] carrying --wait and --timeout 5m makes helm itself block until the release's pods report Ready, which turns "installed" into something much closer to "working". And note that Terratest passes --namespace but never --create-namespace, so either create the namespace yourself with k8s.CreateNamespace or add that flag here.

Test the upgrade, not only the clean install. Install the version that is live in production today, then upgrade in place to the chart in your branch and probe again. That sequence is where immutable-field errors, hook ordering, and data migrations show themselves; a first-time install hides all three. It is also where tests lie to you. During a rolling upgrade the old pods keep serving while the new ones start, so a probe can happily return 200 from the version you were trying to replace. Wait on the rollout, and re-list pods after the upgrade rather than reusing a name you captured before it. One flag deserves a second thought. --atomic tells helm to roll back a failed upgrade. Tidy cluster, dead crime scene. helm.Upgrade still fails the test, but by the time your diagnostics run the crash-looping pods are gone and you are reading logs from the old version.

One naming detail bites everybody once. The objects a chart creates are named by its fullname template, which is normally the release name plus the chart name, unless the release name already contains the chart name, in which case it collapses to the release name alone. Call the release app-a1b2c3 and the Deployment is app-a1b2c3-nginx. Call it nginx-a1b2c3 and the Deployment is nginx-a1b2c3, with no suffix. Guess wrong and your wait times out against a resource that exists perfectly well under a different name.

test/nginx_chart_test.go

// package test; imports as before, plus helm, http_helper and
// metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 
func TestNginxChart(t *testing.T) {
	t.Parallel()
 
	uniq := strings.ToLower(random.UniqueID()) // 6 chars, e.g. "a1b2c3"
	namespace := "tt-" + uniq
	release := "app-" + uniq       // NOT "nginx-...": see the fullname note above
	fullname := release + "-nginx" // what the chart names the Deployment and Service
 
	kubectlOptions := k8s.NewKubectlOptions(kubeContext(), "", namespace)
	k8s.CreateNamespace(t, kubectlOptions, namespace)
	defer k8s.DeleteNamespace(t, kubectlOptions, namespace) // registered 1st, runs last
 
	helmOptions := &helm.Options{
		KubectlOptions: kubectlOptions, // the release lands in OUR namespace
		SetValues: map[string]string{
			"replicaCount": "2",
			"service.type": "ClusterIP", // never LoadBalancer in a test
		},
		SetStrValues: map[string]string{"image.tag": "1.28-alpine"},
		ExtraArgs: map[string][]string{
			"install": {"--wait", "--timeout", "5m"},
			"upgrade": {"--wait", "--timeout", "5m"}, // no --atomic: keep the evidence
		},
	}
 
	// 2nd. DeleteE, not Delete: if the install never happened, a "release not
	// found" from cleanup must not become the error you spend an hour reading.
	defer func() { _ = helm.DeleteE(t, helmOptions, release, true) }() // true = purge history
 
	// 3rd, so it runs BEFORE the two deletes above: dump evidence on failure.
	// Every call here is an E variant, so one broken dump command cannot
	// abort the rest of the dump.
	defer func() {
		if !t.Failed() {
			return
		}
		_ = k8s.RunKubectlE(t, kubectlOptions, "get", "pods", "-o", "wide")
		_ = k8s.RunKubectlE(t, kubectlOptions, "get", "events", "--sort-by=.lastTimestamp")
		for _, pod := range k8s.ListPods(t, kubectlOptions,
			metav1.ListOptions{LabelSelector: "app.kubernetes.io/instance=" + release}) {
			logs, _ := k8s.GetPodLogsE(t, kubectlOptions, &pod, "") // "" = default container
			t.Log(logs)
		}
	}()
 
	// 1) the version running in production today
	helm.AddRepo(t, helmOptions, "acme", "https://charts.acme.internal")
	helmOptions.Version = "1.4.0"
	helm.Install(t, helmOptions, "acme/nginx", release)
 
	// 2) upgrade in place to the chart in this branch
	helmOptions.Version = ""
	helm.Upgrade(t, helmOptions, "../charts/nginx", release)
 
	// wait on the rollout, not on a pod name captured before the upgrade
	k8s.WaitUntilDeploymentAvailable(t, kubectlOptions, fullname, 30, 5*time.Second)
}

terminal

go test -v -timeout 30m -run TestNginxChart ./test/

output

=== RUN   TestNginxChart
=== PAUSE TestNginxChart
=== CONT  TestNginxChart
TestNginxChart 2026-07-22T09:30:58Z logger.go:79: Running command helm with args [repo add acme https://charts.acme.internal]
TestNginxChart 2026-07-22T09:30:59Z logger.go:79: "acme" has been added to your repositories
TestNginxChart 2026-07-22T09:30:59Z logger.go:79: Running command helm with args [install --kube-context kind-terratest --namespace tt-a1b2c3 --wait --timeout 5m --version 1.4.0 --set replicaCount=2 --set service.type=ClusterIP --set-string image.tag=1.28-alpine app-a1b2c3 acme/nginx]
TestNginxChart 2026-07-22T09:31:24Z logger.go:79: NAME: app-a1b2c3
TestNginxChart 2026-07-22T09:31:24Z logger.go:79: NAMESPACE: tt-a1b2c3
TestNginxChart 2026-07-22T09:31:24Z logger.go:79: STATUS: deployed
TestNginxChart 2026-07-22T09:31:24Z logger.go:79: REVISION: 1
TestNginxChart 2026-07-22T09:31:24Z logger.go:79: Running command helm with args [upgrade --kube-context kind-terratest --namespace tt-a1b2c3 --wait --timeout 5m --set replicaCount=2 --set service.type=ClusterIP --set-string image.tag=1.28-alpine app-a1b2c3 ../charts/nginx]
TestNginxChart 2026-07-22T09:36:26Z logger.go:79: Error: UPGRADE FAILED: timed out waiting for the condition
    nginx_chart_test.go:60: 
        	Error Trace:	/home/dev/infra/test/nginx_chart_test.go:60
        	Error:      	Received unexpected error:
        	            	error while running command: exit status 1; Error: UPGRADE FAILED: timed out waiting for the condition
        	Test:       	TestNginxChart
TestNginxChart 2026-07-22T09:36:26Z logger.go:79: Running command kubectl with args [--context kind-terratest --namespace tt-a1b2c3 get pods -o wide]
TestNginxChart 2026-07-22T09:36:26Z logger.go:79: NAME                               READY   STATUS             RESTARTS      AGE
TestNginxChart 2026-07-22T09:36:26Z logger.go:79: app-a1b2c3-nginx-6d8f4c9b7-2xk9p   1/1     Running            0             5m27s
TestNginxChart 2026-07-22T09:36:26Z logger.go:79: app-a1b2c3-nginx-7c5b96d84-q7wzt   0/1     CrashLoopBackOff   5 (39s ago)   5m2s
TestNginxChart 2026-07-22T09:36:26Z logger.go:79: Running command kubectl with args [--context kind-terratest --namespace tt-a1b2c3 get events --sort-by=.lastTimestamp]
TestNginxChart 2026-07-22T09:36:27Z logger.go:79: 39s   Warning   BackOff   pod/app-a1b2c3-nginx-7c5b96d84-q7wzt   Back-off restarting failed container nginx
TestNginxChart 2026-07-22T09:36:27Z logger.go:79: Running command kubectl with args [--context kind-terratest --namespace tt-a1b2c3 logs app-a1b2c3-nginx-7c5b96d84-q7wzt]
    nginx_chart_test.go:47: 2026/07/22 09:36:21 [emerg] 1#1: mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)
        nginx: [emerg] mkdir() "/var/cache/nginx/client_temp" failed (30: Read-only file system)
--- FAIL: TestNginxChart (329.44s)
FAIL
FAIL    github.com/acme/infra/test      329.91s
FAIL

Read that output backwards and the story finishes itself in three lines. One old pod still Ready, one new pod in CrashLoopBackOff, and an nginx log saying it cannot create its cache directory because the filesystem is read-only. Somebody on the branch added readOnlyRootFilesystem: true to the chart, which is a good change, and forgot the writable emptyDir volume that nginx needs at /var/cache/nginx to go with it. Without the deferred dump you would have had a helm timeout and nothing else, on a namespace that was already deleted by the time you opened the build log.

Tunnels Bind a Real Local Port, and defer Runs Backwards

Give k8s.NewTunnel a local port of 0 so Terratest asks the operating system for a free one and writes it back into the tunnel. The log prints the tunnel line before it resolves the port, so you will see "routing local port 0" followed by "Selected port 41235" a couple of lines later. That is normal. Hardcode 8080 instead and the second concurrent test greets you with "bind: address already in use". Then mind the order. defer is last in, first out, like plates stacked on a counter: the call you register last comes off first. Register the namespace delete first, the release delete second, your failure dump third, and tunnel.Close() last, so the execution order comes out as close the tunnel, dump the evidence, delete the release, delete the namespace. Get it backwards and you tear the Service out from under an open port-forward, which leaks the forwarding goroutine and buries the real failure under a connection-refused error that explains nothing.

The Probe Is the Only Honest Witness

A ClusterIP Service has no address your test process can reach. k8s.GetServiceEndpoint hands back the cluster IP and port for one, which is routable only from inside the cluster, so dialling it from the test host hangs until something times out. A port-forward is the service hatch cut through that wall. k8s.NewTunnel opens the same tube kubectl port-forward opens: a local port on the test machine, through the API server, down to a port on a pod behind the resource you name. It takes a resource type (k8s.ResourceTypePod, ResourceTypeDeployment or ResourceTypeService), the resource name, the local port, and the remote port.

Then send a real request, and read the fine print on the helper you pick. http_helper.HttpGetWithRetry takes a URL, an optional TLS config (transport layer security, the encryption behind HTTPS), the status code you expect, and the body you expect. That last argument is an exact, whole-body comparison, not a substring search. Hand it "Welcome to nginx!" and the test burns every retry and fails, because the real welcome page is a full HTML document with a stylesheet in it. For anything longer than a short fixed string, reach for http_helper.HttpGetWithRetryWithCustomValidation and supply your own function over the status code and body. Both variants retry on connection errors and on a wrong answer alike, so a slow first image pull does not turn into a red build at 3am. This call is the only step in the whole test that proves the workload serves traffic. Everything before it proves that Kubernetes stored some objects and started some processes.

test/nginx_chart_test.go

	// ...continuing TestNginxChart, after WaitUntilDeploymentAvailable
 
	// local port 0: the OS picks a free one, which is what parallel tests need
	tunnel := k8s.NewTunnel(kubectlOptions, k8s.ResourceTypeService, fullname, 0, 80)
	defer tunnel.Close() // registered last, so it runs first
	tunnel.ForwardPort(t)
 
	// A real GET over a real socket. HttpGetWithRetry would compare the WHOLE
	// body to the string you pass, so use custom validation for a real page.
	http_helper.HttpGetWithRetryWithCustomValidation(t, "http://"+tunnel.Endpoint(), nil,
		30, 5*time.Second,
		func(status int, body string) bool {
			return status == 200 && strings.Contains(body, "Welcome to nginx!")
		})

terminal

# after fixing the chart: same test, green path
go test -v -timeout 30m -run TestNginxChart ./test/

output

TestNginxChart 2026-07-22T10:03:26Z retry.go:159: Wait for deployment app-a1b2c3-nginx to be provisioned.
TestNginxChart 2026-07-22T10:03:31Z logger.go:79: Deployment is now available
TestNginxChart 2026-07-22T10:03:31Z tunnel.go:189: Creating a port forwarding tunnel for resource service/app-a1b2c3-nginx routing local port 0 to remote port 80
TestNginxChart 2026-07-22T10:03:31Z tunnel.go:297: Requested local port is 0. Selecting an open port on host system
TestNginxChart 2026-07-22T10:03:31Z tunnel.go:305: Selected port 41235
TestNginxChart 2026-07-22T10:03:32Z tunnel.go:333: Successfully created port forwarding tunnel
TestNginxChart 2026-07-22T10:03:32Z http_helper.go:101: Making an HTTP GET call to URL http://localhost:41235
TestNginxChart 2026-07-22T10:03:32Z logger.go:79: Running command helm with args [delete --kube-context kind-terratest --namespace tt-a1b2c3 app-a1b2c3]
TestNginxChart 2026-07-22T10:03:34Z logger.go:79: release "app-a1b2c3" uninstalled
--- PASS: TestNginxChart (58.20s)
PASS
ok      github.com/acme/infra/test      58.63s

Prove the Deny Path, Not Only the Allow Path

Everything so far asks "does it work". The security question is usually the reverse. Is the thing that should be blocked actually blocked? A lock you never try is a lock you are guessing about. A NetworkPolicy meant to stop other namespaces from reaching your Service is exactly that sort of lock, because when it breaks, everything still works and nobody notices. Same for a chart that should refuse to mount the ServiceAccount token, or an Ingress that should reject plain HTTP. So write the test that expects failure. Run a throwaway curl pod in a different namespace, point it at the Service's DNS name (domain name system, the cluster's internal phone book), and require an error. Every Terratest helper whose name ends in E hands the error back instead of failing the test on the spot, which is what lets you assert that something went wrong.

test/networkpolicy_test.go

// package test; imports as before
 
func TestNginxRejectsOtherNamespaces(t *testing.T) {
	t.Parallel()
 
	// installNginx is the shared helper the chart test uses: it installs the
	// chart, waits for the rollout, and registers its own cleanup. It returns
	// the chart's fullname and the namespace it landed in.
	fullname, namespace := installNginx(t)
 
	outsideNS := "tt-outside-" + strings.ToLower(random.UniqueID())
	outside := k8s.NewKubectlOptions(kubeContext(), "", outsideNS)
	k8s.CreateNamespace(t, outside, outsideNS)
	defer k8s.DeleteNamespace(t, outside, outsideNS)
 
	// the E variant hands back the error instead of failing the test here
	out, err := k8s.RunKubectlAndGetOutputE(t, outside,
		"run", "probe", "--rm", "-i", "--restart=Never",
		"--image=curlimages/curl:8.11.1", "--",
		"curl", "-sS", "--max-time", "5",
		"http://"+fullname+"."+namespace+".svc.cluster.local")
 
	require.Error(t, err, "NetworkPolicy did not block cross-namespace traffic")
	// blocked, not misspelled: an unresolvable name gives curl exit 6 instead
	require.Contains(t, out, "Connection timed out")
}

terminal

go test -v -timeout 30m -run TestNginxRejectsOtherNamespaces ./test/

output

=== RUN   TestNginxRejectsOtherNamespaces
=== PAUSE TestNginxRejectsOtherNamespaces
=== CONT  TestNginxRejectsOtherNamespaces
TestNginxRejectsOtherNamespaces 2026-07-22T09:41:12Z logger.go:79: Running command kubectl with args [--context kind-terratest --namespace tt-outside-9f2k1a run probe --rm -i --restart=Never --image=curlimages/curl:8.11.1 -- curl -sS --max-time 5 http://app-a1b2c3-nginx.tt-a1b2c3.svc.cluster.local]
TestNginxRejectsOtherNamespaces 2026-07-22T09:41:19Z logger.go:79: curl: (28) Connection timed out after 5001 milliseconds
TestNginxRejectsOtherNamespaces 2026-07-22T09:41:19Z logger.go:79: pod "probe" deleted
TestNginxRejectsOtherNamespaces 2026-07-22T09:41:19Z logger.go:79: pod tt-outside-9f2k1a/probe terminated (Error)
--- PASS: TestNginxRejectsOtherNamespaces (7.31s)
PASS
ok      github.com/acme/infra/test      7.58s

Negative tests pass for the wrong reason very easily. A typo in the DNS name, an image that fails to pull, a namespace deleted a line too early: all three produce an error and a green test, and none of them tested your NetworkPolicy. That require.Contains on "Connection timed out" is the guard, because a name that does not resolve gives you curl exit code 6 and a completely different message. There is a larger trap underneath it. NetworkPolicy is enforced by your CNI plugin (container network interface, the component that wires up pod networking), and kind's built-in plugin has not always enforced it at all. On a cluster where nothing enforces policy, your deny test goes green because nothing is listening, which is the most expensive shade of green there is. Build the test cluster with disableDefaultCNI: true, install the plugin you actually run in production (Calico or Cilium, usually), and pair the deny test with a matching allow test from a permitted namespace in the same file. Then a broken URL turns both of them red at once and the contradiction is impossible to miss.

A Namespace Does Not Contain a Chart

k8s.DeleteNamespace removes everything namespaced: Deployments, Services, ConfigMaps, Secrets. It does not touch cluster-scoped objects, and plenty of charts create those. A ClusterRole, a ClusterRoleBinding, a CustomResourceDefinition (CRD, a new object type registered across the whole cluster), a ValidatingWebhookConfiguration, a PersistentVolume, an IngressClass: each one survives the namespace delete, each one has a single global name, and two parallel runs of the same chart will fight over that name. This is why the deferred release delete matters even when the namespace is about to vanish anyway. CRDs are the exception even then, because helm never deletes a CRD it installed. And a leftover webhook is worse than clutter. It is a cluster-wide admission hook still pointing at a Service that no longer exists, and with failurePolicy: Fail it will reject every matching create in the entire cluster until somebody goes looking for it.

Wiring It Into CI

The pipeline job is four commands. Create a kind cluster named after the job, load the image you built in the previous stage straight into the cluster nodes, run the suite, delete the cluster. kind load skips a registry push entirely, so nothing half-tested is ever published where a human could pull it by accident. Export the context name so the tests find the right cluster. Put that last command in whatever your CI calls an always-runs step: a runner killed mid-test leaves a container holding a control plane, and the next job on that runner will collide with it. Keep -timeout 30m as well. Go's default is ten minutes, and when a test binary hits that global timeout the process panics and dies on the spot, so not one of your deferred deletes runs, and you go on paying for whatever the test created.

terminal

export KUBE_CONTEXT=kind-ci-$CI_JOB_ID
 
kind create cluster --name ci-$CI_JOB_ID --image kindest/node:v1.34.0 --wait 90s
kind load docker-image acme/nginx:$CI_COMMIT_SHORT_SHA --name ci-$CI_JOB_ID
go test -v -timeout 30m -parallel 4 ./test/...
kind delete cluster --name ci-$CI_JOB_ID   # always-runs step, never conditional

output

Creating cluster "ci-91847" ...
 ✓ Ensuring node image (kindest/node:v1.34.0)
 ✓ Preparing nodes
 ✓ Writing configuration
 ✓ Starting control-plane
 ✓ Installing CNI
 ✓ Installing StorageClass
 ✓ Waiting ≤ 1m30s for control-plane = Ready
 • Ready after 22s
Set kubectl context to "kind-ci-91847"
Image: "acme/nginx:3f9a2c1" with ID "sha256:6f0a1c7b9e42" not yet present on node "ci-91847-control-plane", loading...
=== RUN   TestChartRendersSafely
=== RUN   TestNginxChart
=== RUN   TestNginxRejectsOtherNamespaces
--- PASS: TestChartRendersSafely (0.29s)
--- PASS: TestNginxRejectsOtherNamespaces (7.31s)
--- PASS: TestNginxChart (58.20s)
PASS
ok      github.com/acme/infra/test      58.63s
Deleting cluster "ci-91847" ...
Deleted nodes: ["ci-91847-control-plane"]

Try this

Run kind create cluster --name terratest --image kindest/node:v1.34.0 on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: tunnels Bind a Real Local Port, and defer Runs Backwards. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Assertions & retriesTest stages & speed

What InSpec is: compliance as codeInSpec The secrets-in-Git problem & SOPSSOPS

Testing Kubernetes & Helm

A Cluster You Are Allowed to Break

What Each Wait Actually Proves

Render the Chart Before It Touches the Cluster

Install, Upgrade, and the Flags That Matter

The Probe Is the Only Honest Witness

Prove the Deny Path, Not Only the Allow Path

Wiring It Into CI

Try this

Takeaway

Related