Testing infrastructure

Unit and integration tests for infra.

There are two honest ways to check a lock. You can read the blueprint and confirm somebody specified a deadbolt, which takes seconds and needs no building. Or you can walk to the finished door and try the handle, which takes a van, a key, and an afternoon. Both answers are true. Neither one replaces the other.

A Pulumi program is ordinary code in an ordinary language, so ordinary test tools work on it. The awkward part is what sits at the far end of the wire. The thing under test is a live cloud account, and you cannot hand every commit a fresh one. So the work splits by altitude. Unit tests swap the Pulumi engine (the part that normally talks to your cloud) for in-memory fakes: no credentials, no network, milliseconds per run, safe on every push. Integration tests do the opposite. They deploy a throwaway stack (one named, deployed copy of your program, the way dev and prod are two copies of the same code) to a real provider, poke at it, and tear it down. Pulumi ships a purpose-built harness for each.

Here is why a security or operations person should care more than anyone. The change that opens SSH (Secure Shell, the encrypted remote-login service that normally listens on port 22) to the entire internet is a one-line diff. A list of allowed addresses reading ["10.0.0.0/8"] becomes ["0.0.0.0/0"] because somebody was debugging from a hotel on a Friday night. A tired reviewer scrolls past it. A test never scrolls past it. Ten lines of assertion turn a rule you keep repeating in code review into a wall the pipeline holds up on your behalf.

Swap The Cloud For A Stunt Double

A film crew does not push the lead actor off a real roof. They hire a double who looks right on camera and lands on a mat. pulumi.runtime.setMocks hires that double for your cloud. Call it and the engine stops dialing a provider (the plugin that normally speaks to Amazon Web Services, Azure, or Google Cloud for Pulumi) and starts calling two functions you wrote. newResource fires once for every resource your program declares, and you hand back a fake physical id plus the resource's output state. call fires for every provider function, meaning the read-only lookups like aws.getAmi, which normally asks Amazon which disk image matches a name pattern. That is the entire substitution. Nothing leaves your laptop.

One rule outranks all the others: install the mocks before you import the program under test. The import is what actually executes your infrastructure code. If the double is not already standing on the mark at that moment, the real engine answers, and your unit test starts building things. The example below runs under mocha, a JavaScript test runner that has been around long enough that most CI setups already know what to do with it.

import * as pulumi from "@pulumi/pulumi";
import * as assert from "assert";
 
// Install the fakes BEFORE importing the program under test.
pulumi.runtime.setMocks(
  {
    // Once per resource. Return a fake physical id + the output state.
    newResource: (args: pulumi.runtime.MockResourceArgs) => ({
      id: `${args.name}_id`,
      state: args.inputs,        // your inputs, echoed back as outputs
    }),
    // Once per provider function / data source (aws.getAmi, getCallerIdentity...).
    call: (args: pulumi.runtime.MockCallArgs) => args.inputs,
  },
  "webserver",   // project name baked into every URN
  "test",        // stack name baked into every URN
  false,         // preview? false = outputs resolve as known values
);
 
// Three lines that let mocha await an Output instead of juggling done().
const promiseOf = <T>(o: pulumi.Output<T>): Promise<T> =>
  new Promise((resolve) => o.apply(resolve));
 
describe("network", () => {
  let infra: typeof import("../index");
  before(async () => { infra = await import("../index"); });
 
  it("never opens SSH to the whole internet", async () => {
    const [urn, ingress] = await promiseOf(
      pulumi.all([infra.group.urn, infra.group.ingress]),
    );
 
    const wideOpen = (ingress ?? []).filter((r) =>
      (r.cidrBlocks ?? []).includes("0.0.0.0/0") &&
      (r.protocol === "-1" ||                                  // "-1" = every protocol
        ((r.fromPort ?? 0) <= 22 && (r.toPort ?? 0) >= 22)));  // range swallows 22
 
    assert.strictEqual(wideOpen.length, 0,
      `${urn} exposes port 22 to 0.0.0.0/0`);
  });
});

Two details in that filter are the ones people get wrong in real reviews. A rule with protocol: "-1" is Amazon's shorthand for every protocol, so it reaches port 22 without ever naming a port. And a range written as 20 to 8080 covers 22 while looking innocent. The failure message carries the URN (Uniform Resource Name, Pulumi's globally unique address for one resource inside one stack), which reads urn:pulumi:<stack>::<project>::<type>::<name>. That is why the project and stack strings you pass to setMocks matter. They are what makes the URN in a red build point at something a human can go and open.

# No cloud credentials, no network, no state file. Safe on every push.
npx mocha --require ts-node/register 'tests/**/*.spec.ts'

 
  network
    1) never opens SSH to the whole internet
 
  0 passing (24ms)
  1 failing
 
  1) network
       never opens SSH to the whole internet:
 
      AssertionError [ERR_ASSERTION]: urn:pulumi:test::webserver::aws:ec2/securityGroup:SecurityGroup::web-sg exposes port 22 to 0.0.0.0/0
      + expected - actual
 
      -1
      +0
 
      at Context.<anonymous> (tests/infra.spec.ts:36:12)
      at processTicksAndRejections (node:internal/process/task_queues:95:5)
 

Twenty-four milliseconds, no credentials, and a named resource. Now tighten the rule so the only allowed source is the bastion host (the one hardened machine everyone is supposed to hop through) at a single address: a /32 in CIDR notation, which is Classless Inter-Domain Routing, the standard shorthand for a block of addresses. A /32 means exactly one address. Run it again to prove the fix landed.

npx mocha --require ts-node/register 'tests/**/*.spec.ts'

 
  network
    ✔ never opens SSH to the whole internet
 
  1 passing (21ms)
 

What The Stunt Double Cannot See

The fake hands your inputs straight back as the outputs. That is the deal, and it is where careless tests quietly rot. Anything the provider would have computed, and that you did not explicitly return from your mock, comes back undefined. Not wrong-but-plausible. Empty. An ARN (Amazon Resource Name, the unique identifier AWS assigns to something it created), a provider-generated bucket suffix, the AMI id (Amazon Machine Image, the disk template a virtual machine boots from) that a getAmi lookup would have found: all absent. Print them once so you believe it.

  it("cannot see anything AWS would compute", async () => {
    const arn = await promiseOf(infra.bucket.arn);   // AWS computes this
    const id  = await promiseOf(infra.bucket.id);    // your mock computed this
    console.log("arn:", arn, "| id:", id);
  });

npx mocha --require ts-node/register 'tests/**/*.spec.ts'

 
  network
    ✔ never opens SSH to the whole internet
arn: undefined | id: logs-bucket_id
    ✔ cannot see anything AWS would compute
 
  2 passing (26ms)
 

Sit with why that is dangerous. A check written as "the bucket ARN must not contain the word public" passes on every run. It passes because there is no ARN to contain anything. Negative assertions aimed at absent values are the quietest way to ship a green build over broken infrastructure. So assert on inputs you wrote yourself, or on values you deliberately returned from the mock, and hand everything else to the tier that uses a real account.

The Same Test In Python

Python subclasses instead of passing an object literal, and the callbacks go snake_case. new_resource returns a two-item list, the id and the state. call returns a dict, and note that Python names its payload args.args rather than args.inputs. Then comes the piece nobody may skip. @pulumi.runtime.test waits for the Output your test function returns. Leave the decorator off and a test that returns an Output finishes instantly, asserts nothing at all, and reports success.

# tests/__init__.py must exist (empty is fine) or unittest discovery
# refuses to import this directory as a package.
import unittest
import pulumi
 
 
class Mocks(pulumi.runtime.Mocks):
    def new_resource(self, args: pulumi.runtime.MockResourceArgs):
        return [args.name + "_id", args.inputs]   # (physical id, output state)
 
    def call(self, args: pulumi.runtime.MockCallArgs):
        return {}                                 # payload is args.args in Python
 
 
pulumi.runtime.set_mocks(Mocks(), project="webserver", stack="test", preview=False)
 
import infra  # noqa: E402  -> import AFTER set_mocks, never before
 
 
class TestTagging(unittest.TestCase):
    @pulumi.runtime.test          # waits for the Output you return
    def test_owner_tag(self):
        def check(args):
            urn, tags = args
            self.assertIn("Owner", tags or {}, f"{urn} has no Owner tag")
 
        return pulumi.Output.all(infra.server.urn, infra.server.tags).apply(check)

# -t . puts the project root on sys.path so `import infra` resolves.
python3 -m unittest discover -s tests -t . -v

test_owner_tag (tests.test_infra.TestTagging.test_owner_tag) ... FAIL
 
======================================================================
FAIL: test_owner_tag (tests.test_infra.TestTagging.test_owner_tag)
----------------------------------------------------------------------
Traceback (most recent call last):
  ... frames through pulumi/output.py and asyncio trimmed ...
  File "/home/dev/webserver/tests/test_infra.py", line 26, in check
    self.assertIn("Owner", tags or {}, f"{urn} has no Owner tag")
AssertionError: 'Owner' not found in {'Name': 'web-01'} : urn:pulumi:test::webserver::aws:ec2/instance:Instance::web-01 has no Owner tag
 
----------------------------------------------------------------------
Ran 1 test in 0.042s
 
FAILED (failures=1)

An ownership tag is a name written on the side of a crate in a warehouse. It looks like paperwork until the day you need to find the crate. When an alert fires at 3am on an instance nobody recognizes, the difference between a five-minute page and a two-hour hunt is whether that tag exists. Enforcing it at unit-test speed costs nothing per commit.

Deploy It For Real, Then Kill It

Some questions have no mock. Does the load balancer actually answer with 200 (the HTTP status code that means "here is the page")? Does the database refuse a connection from outside its subnet? Does the name you published in DNS (the internet's address book, which turns names into addresses) really point at the load balancer and not at last month's one? Does an IAM policy (Identity and Access Management, the AWS permission system) that looks fine in code fall over the first time something calls it? Does the response carry the security headers you configured three layers up in a module you did not write? You have to build the thing. Pulumi's Go harness, integration.ProgramTest, runs that errand around any real Pulumi project, whatever language the project itself is written in. It is a Go test that drives the real pulumi CLI (command-line interface, the same binary you type at a prompt).

The order it runs in is worth learning, because it explains every option you will reach for. It copies your project into a temporary directory, so your working tree is never touched. It runs pulumi stack init with a generated name shaped p-it-<host>-<testdir>-<8 hex chars>, all lowercase, with the host and directory names clipped to ten characters each. It applies Config and Secrets. It installs the project's dependencies. It previews and updates. It exports and re-imports the state to prove the state file survives a round trip. It runs an empty preview and update to prove a second run changes nothing, which is the idempotence check. It calls your ExtraRuntimeValidation. It refreshes. Then pulumi destroy and pulumi stack rm, even when an assertion has already failed. Quick: true skips the standalone preview, the export/import round trip, and the empty-update pass. SkipRefresh: true drops the refresh.

package tests
 
import (
	"net/http"
	"path/filepath"
	"testing"
 
	"github.com/pulumi/pulumi/pkg/v3/testing/integration"
	"github.com/stretchr/testify/assert"
	"github.com/stretchr/testify/require"
)
 
func TestWebserver(t *testing.T) {
	integration.ProgramTest(t, &integration.ProgramTestOptions{
		Dir:   filepath.Join("..", "webserver"), // a real project, any language
		Quick: true,                             // skip preview + export/import + empty update
		Config: map[string]string{
			"aws:region": "eu-west-1",
		},
		Secrets: map[string]string{
			"webserver:dbPassword": "throwaway-for-this-run",
		},
		ExtraRuntimeValidation: func(t *testing.T, stack integration.RuntimeValidationStackInfo) {
			url := stack.Outputs["url"].(string)
 
			resp, err := http.Get(url)
			require.NoError(t, err)   // require, not assert: resp is nil on error
			defer resp.Body.Close()
 
			// Questions no mock can answer:
			assert.Equal(t, 200, resp.StatusCode)
			assert.Equal(t, "DENY", resp.Header.Get("X-Frame-Options"))
			assert.NotEmpty(t, resp.Header.Get("Strict-Transport-Security"))
		},
	})
}

# The harness shells out to the real CLI, so give it a backend and credentials.
pulumi login --local                                    # or PULUMI_ACCESS_TOKEN in CI
export PULUMI_CONFIG_PASSPHRASE='ci-only-passphrase'    # required by Secrets{}
go test -v -timeout 30m ./tests/...

=== RUN   TestWebserver
    program.go:1264: Initializing project (dir /tmp/p-it-runner01-webserver-3f9ac21d-1846203715; stack p-it-runner01-webserver-3f9ac21d)
    ... pulumi up trimmed: 11 created (2m41s) ...
    ... validation trimmed: GET http://web-alb-1042.eu-west-1.elb.amazonaws.com -> 200 ...
    ... pulumi destroy trimmed: 11 deleted (1m12s) ...
--- PASS: TestWebserver (247.83s)
PASS
ok      github.com/acme/webserver/tests 248.06s

ExpectFailure: true is the option most people never notice, and for defensive work it may be the best one in the struct. It inverts the harness: the test passes when pulumi up fails. Point it at a small project whose config asks for an unencrypted bucket or a publicly reachable database, and a green test now means your guardrail actually bit. Proving a control blocks something is a different claim from proving the happy path works, and it is the claim an auditor will ask you for.

If writing Go to test a TypeScript program feels like the wrong shape, the Automation API is the alternative: a library that exposes the CLI's actions as ordinary function calls in whatever language you already write. LocalWorkspace.createOrSelectStack({ stackName, workDir }), then stack.up(), then stack.destroy() and stack.workspace.removeStack(stackName). You get the same safety only if you write that teardown inside a finally block yourself, which is precisely the part ProgramTest refuses to let you forget.

Leaked Test Stacks Are Attack Surface

A test that dies hard leaves a stack alive, the way a film set left standing after the crew goes home is still a building with a door. The runner gets killed mid-destroy, the network drops, someone cancels the job. What remains is not only a bill. It is unowned, unmonitored, unpatched infrastructure still carrying whatever permissive settings the test needed in order to run. Every stack the harness creates starts with the same prefix, so hunting them is one command. That prefix only holds if nobody set StackName by hand, which is a good reason not to.

# jq slices the JSON that Pulumi prints when you ask for machine-readable output.
pulumi stack ls --all --json \
  | jq -r '.[] | select(.name | test("p-it-")) | "\(.name)\t\(.resourceCount)\t\(.lastUpdate)"'

acme/webserver/p-it-runner01-webserver-3f9ac21d	11	2026-07-19T02:14:07.000Z
acme/webserver/p-it-runner02-webserver-c40b8e17	9	2026-07-11T23:51:44.000Z

Nine resources have been running for ten days behind a test-grade security group. Run that query on a schedule, alert on anything older than a few hours, and clean up with pulumi destroy --stack <name> --yes followed by pulumi stack rm <name> --yes. Setting DestroyOnCleanup: true in the options also moves teardown into Go's cleanup phase, which survives some failure modes the inline path does not.

Is Production Still What You Wrote

Unit tests read the blueprint. Integration tests inspect a building you put up and then demolished. Neither one walks the building people are actually using. That gap is where console clicks live. Somebody with a legitimate login opens 3389 (RDP, Remote Desktop Protocol, the Windows remote-desktop service) for "ten minutes" during an outage and forgets. Your code never changed, so nothing in CI (continuous integration, the automation that runs on every commit) ever goes red. The infrastructure has quietly stopped matching the reviewed, approved, version-controlled description of it. That gap has a name: drift.

pulumi refresh reads every managed resource back from the provider and compares it against state. Add --expect-no-changes and it stops being a maintenance chore and becomes a detector.

pulumi refresh --stack acme/webserver/prod \
  --diff --expect-no-changes --yes --non-interactive

Previewing refresh (acme/prod):
 
    ~ aws:ec2/securityGroup:SecurityGroup: (update)
        [id=sg-0a1b2c3d4e5f60718]
        [urn=urn:pulumi:prod::webserver::aws:ec2/securityGroup:SecurityGroup::web-sg]
      ~ ingress: [
          + [1]: {
                  + cidrBlocks : [
                  +     [0]: "0.0.0.0/0"
                    ]
                  + description: "temp - incident 4821"
                  + fromPort   : 3389
                  + protocol   : "tcp"
                  + toPort     : 3389
                }
        ]
 
Resources:
    ~ 1 to update
    13 unchanged
 
Refreshing (acme/prod):
     Type                      Name            Status
     pulumi:pulumi:Stack       webserver-prod
 ~   └─ aws:ec2:SecurityGroup  web-sg          updated
 
Resources:
    ~ 1 updated
    13 unchanged
 
Duration: 11s
 
error: no changes were expected but changes occurred

That diff names the resource, the property, the port, and even the excuse somebody typed into the description field. Read the flag's fine print before you schedule it, though. On refresh the check runs after the refresh has already been written, so the drift is in your state file by the time the command errors. If you want a detector that changes nothing at all, run pulumi preview --refresh --diff --expect-no-changes instead, because a preview never persists state. Either way the exit code is non-zero, the nightly job goes red, and a human gets a diff instead of an audit-log line nobody was reading.

Try this

Run npx mocha --require ts-node/register 'tests/**/*.spec.ts' on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a green test that checked nothing. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related