Assertions & retries

Eventual consistency in tests.

Advanced14 min · lesson 5 of 12

Real infrastructure is eventually consistent — a load balancer is created but not yet serving, DNS has not propagated, a pod is scheduled but not ready — so a naive assert immediately after apply often fails not because the code is wrong but because reality has not caught up. Terratest handles this with retry: helpers that poll a condition repeatedly until it holds or a timeout, rather than checking once. This is the core skill of writing stable infra tests.

RETRY, DON'T SLEEP: POLL UNTIL CONVERGED

1Apply deploys resource

eventually consistent — not serving yet

2Poll the condition

DoWithRetry runs the check once

3Not ready? wait & retry

up to N attempts, interval between

4Condition holds -> pass

returns the instant it's true (fast)

5Budget exhausted -> fail

clear timeout error, not a flake

Polling succeeds as soon as reality catches up; a fixed time.Sleep is both slower and flakier.

retry example

import "github.com/gruntwork-io/terratest/modules/retry"
retry.DoWithRetry(t, "wait for API", 30, 10*time.Second, func() (string, error) {
  resp, err := http.Get(url)
  if err != nil || resp.StatusCode != 200 {
    return "", fmt.Errorf("not ready yet")
  }
  return "ok", nil
})
// http_helper.HttpGetWithRetry wraps this exact pattern for HTTP checks

Retry, not sleep

The wrong fix is a fixed time.Sleep — it is either too short (flaky) or too long (slow), and it is a guess. Retry polls: it returns as soon as the condition is true (fast in the common case) and only waits the full budget when something is genuinely wrong (giving a clear timeout error). Size the retry count and interval to your slowest realistic convergence. Flaky infra tests almost always trace back to a missing or under-budgeted retry.

Fixed sleeps make tests flaky and slow at once

time.Sleep(30 * time.Second) is the classic infra-test smell: it slows every run by 30s even when the resource was ready in 3, and still flakes when something takes 31. Replace sleeps with retry/poll helpers that succeed as soon as the condition holds and fail with a useful message on timeout. A test suite full of sleeps is both slow and unreliable — the worst combination.

Cleanup & defer destroyTesting Kubernetes & Helm