Automation API & dynamic providers

Pulumi as a library; custom resources.

Every pulumi up you have run so far had a person behind it. One hand, one lever, one stack at a time. The Automation API (a library version of the Pulumi engine, so your own program drives the engine directly instead of a human typing at a prompt) takes that lever off the wall and bolts it into your code. API is short for application programming interface: the machine-to-machine way one program controls another. preview, up, refresh and destroy stop being things you type and become functions you call, handing back structured results instead of text scrolling past a terminal.

That opens doors worth walking through. A signup form that builds each new customer their own isolated environment. An integration test that stands up real infrastructure, asserts against it, and tears it down. A nightly job that checks all four hundred stacks you own for changes nobody wrote down. The second half of this lesson covers the other extension point, dynamic providers, which is how you teach Pulumi to manage something no published provider has ever heard of. Both features take deployment decisions out of human hands. That is exactly why you want to know where the trust boundaries land.

The Library Still Drives The Binary

The name makes it sound like the CLI (command line interface, the pulumi program you type at a shell prompt) has been retired. It has not. Treat the Automation API as a remote control rather than a replacement engine: it runs that same binary underneath as a child process. So the machine running your service needs the Pulumi CLI installed and findable on PATH (the list of directories your shell searches when you name a program). Miss that and your program dies on its very first call, complaining that pulumi is not a command.

command -v pulumi && pulumi version

/usr/local/bin/pulumi
v3.189.0

That one fact changes how you package the service. Your container image now ships a large third-party binary that will execute with your cloud credentials sitting in its environment. Pin the version, verify the checksum at install time, and treat a Pulumi CLI bump the way you would treat any other dependency bump in software that can delete production. Everything else behaves exactly as it does from a terminal. Same engine, same state backend, same encryption of secrets. Different steering wheel.

Driving A Stack From Code

The entry point is a workspace: the desk an operation works at, holding the project folder and its settings. LocalWorkspace.createOrSelectStack hands you back a Stack object. Point it at a normal project directory with workDir, or hand it an inline program, which is an ordinary function that declares resources. With an inline program there is no Pulumi.yaml and no project folder on disk at all, because the workspace writes what it needs into a temporary directory.

import * as fs from "fs";
import { LocalWorkspace } from "@pulumi/pulumi/automation";
import * as aws from "@pulumi/aws";
 
const audit = fs.createWriteStream("/var/log/pulumi/tenant-42.jsonl", { flags: "a" });
 
// The Pulumi program itself: a plain function. No Pulumi.yaml on disk.
const program = async () => {
  const bucket = new aws.s3.BucketV2("tenant-data", {});
  return { bucketName: bucket.id };            // becomes a stack output
};
 
async function main() {
  const stack = await LocalWorkspace.createOrSelectStack(
    { stackName: "tenant-42", projectName: "tenant", program },
    // envVars land in the CLI child process, readable via /proc on this host
    { envVars: { PULUMI_ACCESS_TOKEN: process.env.PULUMI_ACCESS_TOKEN ?? "" } },
  );
 
  await stack.setConfig("aws:region", { value: "eu-west-1" });
 
  const res = await stack.up({
    policyPacks: ["/opt/infra/policies"],                    // CrossGuard runs here too
    onOutput: (s) => process.stdout.write(s),                // human-readable log
    onEvent: (e) => audit.write(JSON.stringify(e) + "\n"),   // machine-readable log
  });
 
  console.log("result :", res.summary.result);
  console.log("changes:", res.summary.resourceChanges);
  console.log("bucket :", res.outputs.bucketName.value);
}
 
main();

npx tsx /opt/infra/provision.ts

Updating (tenant-42)
 
View Live: https://app.pulumi.com/acme/tenant/tenant-42/updates/1
 
     Type                 Name              Status
 +   pulumi:pulumi:Stack  tenant-tenant-42  created (7s)
 +   └─ aws:s3:BucketV2   tenant-data       created (3s)
 
Outputs:
    bucketName: "tenant-data-4b1e07c9"
 
Resources:
    + 2 created
 
Duration: 8s
 
result : succeeded
changes: { create: 2 }
bucket : tenant-data-4b1e07c9

Three lines in that program earn their keep. setConfig does the same job as pulumi config set, and it takes { value, secret: true } when the value should be encrypted rather than parked in the clear. policyPacks runs your CrossGuard rules (CrossGuard is Pulumi's policy engine, the code that can refuse a deployment outright) against this run, which matters enormously once an HTTP request (hypertext transfer protocol, the language browsers and web services speak to each other) can trigger an up. Guardrails you wrote for the pipeline do not follow you into a library call unless you name them here. And res.outputs is a map where every entry carries a value and a secret flag. Dump that whole object into your logs and you print decrypted secrets, so read the fields you need by name.

The Audit Trail You Build Yourself

onOutput gives you the pretty log a human would watch scroll by. onEvent gives you the same deployment as a stream of structured engine events: one JavaScript object per thing that happens. Configuration read. Resource about to change. Resource finished. Diagnostic printed. Run summarised. Write them one per line and you have JSONL (JSON Lines, one JavaScript Object Notation record per line of the file), which is evidence you can grep, or slice with jq (a small command line tool for querying JSON).

jq -c 'select(.resourcePreEvent) | {op: .resourcePreEvent.metadata.op, urn: .resourcePreEvent.metadata.urn}' \
  /var/log/pulumi/tenant-42.jsonl

{"op":"create","urn":"urn:pulumi:tenant-42::tenant::pulumi:pulumi:Stack::tenant-tenant-42"}
{"op":"create","urn":"urn:pulumi:tenant-42::tenant::aws:s3/bucketV2:BucketV2::tenant-data"}

A URN is Pulumi's unique postal address for one resource: stack, then project, then type, then name. You want these lines because of what your cloud's own audit log now shows. When a person ran pulumi up, CloudTrail (Amazon's record of who called which API, and the equivalent exists on every cloud) recorded that person's role creating a bucket. When a web service runs it, CloudTrail records one shared service role doing everything for everybody, forever. The engine event stream is the only place left where you can tie a resource change back to the tenant and the request that caused it. Tag each line with your request identifier and ship it wherever you already keep logs.

One Stack, One Operation At A Time

A stack is a single-occupancy room with a latch on the door. Pulumi allows one operation against a given stack at a time, and the backend enforces that latch rather than trusting your code to behave itself. Two requests that both touch tenant-42 at once produce this.

npx tsx /opt/infra/provision.ts   # started while the first run is still going

error: [409] Conflict: Another update is currently in progress.
To learn more about possible reasons and resolution, visit https://www.pulumi.com/docs/troubleshooting/#conflict

The Node automation package hands that back as a ConcurrentUpdateError, and exports isConcurrentUpdateError() so your handler can requeue the work instead of returning a 500 to the caller. Two designs dodge the collision entirely: serialise work per stack behind a queue keyed on the stack name, or give every tenant its own stack so parallelism comes free. If a process is killed mid-update the latch can stay shut, and pulumi cancel releases it. Be careful with that one. Cancelling frees the lock, but it cannot reach inside a dead process and stop cloud API calls it already had in flight, so a cancel followed by an immediate retry can put two engines to work on one state file.

Two mistakes show up again and again in services built this way. The first is blast radius. stack.destroy() is one line of code away from stack.up() in any router, so run this service under a role scoped to the resources it actually owns, and keep the same approvals you would put on a pipeline apply. The second is stack names. Build stackName out of caller-supplied input, as in tenant-${req.body.id}, and anyone who sends another customer's identifier selects that customer's stack and reads its outputs, because createOrSelectStack is perfectly happy to select a stack it did not create. An identifier matching nothing is worse in a quieter way: it creates a brand new stack on demand, which is free storage and free noise for anyone who wants to fill your backend with junk. Check that identifier against your own customer records before it goes anywhere near createOrSelectStack.

A Watchdog That Notices Drift

Here is the defensive payoff, and it costs about forty lines. preview on its own compares your program against state, which is Pulumi's written memory of what it built. It never asks the cloud what is actually out there. Somebody who widens a security group by hand in the console is therefore invisible to a plain preview: the recipe and the ledger still agree, and nobody walked into the kitchen to look. Add refresh: true and the engine reads the live resources first, then compares. Any gap between your program and reality turns up as pending work.

import { LocalWorkspace } from "@pulumi/pulumi/automation";
 
const WORK_DIR = "/opt/infra/platform";
 
(async () => {
  const ws = await LocalWorkspace.create({ workDir: WORK_DIR });
 
  for (const s of await ws.listStacks()) {
    const stack = await LocalWorkspace.selectStack({ stackName: s.name, workDir: WORK_DIR });
 
    // refresh reads the live cloud first, then the preview compares it to your program
    const pre = await stack.preview({ refresh: true });
 
    const moved = Object.entries(pre.changeSummary).filter(([op]) => op !== "same");
    if (moved.length > 0) {
      console.error(`DRIFT ${s.name} ${JSON.stringify(Object.fromEntries(moved))}`);
      process.exitCode = 1;                    // a non-zero exit is the alert
    }
  }
})();

preview never writes to your cloud and never writes a new checkpoint, so hand this job read-only credentials and it still does its work. Then let systemd (the program modern Linux distributions use to start and supervise services) run it on a timer, because a failed unit is something your alerting already watches. Note the sandboxing block at the bottom: a drift checker that can rewrite files on its own host is a drift checker an attacker can quietly silence.

[Unit]
Description=Pulumi drift check across all stacks
Wants=network-online.target
After=network-online.target
 
[Service]
Type=oneshot
User=pulumi
Group=pulumi
WorkingDirectory=/opt/infra/platform
Environment=PULUMI_HOME=/var/lib/pulumi
# read-only cloud credentials live here, owned by pulumi, mode 0600
# systemd has no end-of-line comments, so keep them on their own lines
EnvironmentFile=/etc/pulumi/drift.env
# drift.js is the compiled output of drift.ts
ExecStart=/usr/bin/node /opt/infra/drift.js
 
# this unit must never be able to change anything, locally or in the cloud
NoNewPrivileges=true
ProtectSystem=strict
ProtectHome=true
PrivateTmp=true
ReadWritePaths=/var/lib/pulumi

[Unit]
Description=Nightly Pulumi drift check
 
[Timer]
OnCalendar=*-*-* 02:30:00
RandomizedDelaySec=15m
Persistent=true
 
[Install]
WantedBy=timers.target

systemctl daemon-reload
systemctl enable --now pulumi-drift.timer

Created symlink /etc/systemd/system/timers.target.wants/pulumi-drift.timer → /etc/systemd/system/pulumi-drift.timer.

RandomizedDelaySec=15m smears the start time so four hundred hosts do not hammer the same API at 02:30 exactly. The next morning, the timer has fired once.

systemctl list-timers pulumi-drift.timer --no-pager

NEXT                        LEFT     LAST                        PASSED UNIT               ACTIVATES
Thu 2026-07-23 02:38:02 UTC 16h left Wed 2026-07-22 02:31:44 UTC 8h ago pulumi-drift.timer pulumi-drift.service
 
1 timers listed.

journalctl -u pulumi-drift.service -n 3 --no-pager

Jul 22 02:31:44 ops-01 node[41883]: DRIFT payments-prod {"update":1}
Jul 22 02:31:44 ops-01 systemd[1]: pulumi-drift.service: Main process exited, code=exited, status=1/FAILURE
Jul 22 02:31:44 ops-01 systemd[1]: pulumi-drift.service: Failed with result 'exit-code'.

An attacker who edits a firewall rule in the console leaves nothing in Git and nothing in your merge history. Your program still says port 443 only. The refresh reads what is really there, the preview shows one update waiting to put it back, and the unit fails at 02:31. That is your detection, and the remediation is already written. It is the program.

Teaching Pulumi A Resource It Has Never Heard Of

Pulumi can create an S3 bucket (Simple Storage Service, Amazon's object storage) because somebody wrote a translator sitting between the engine and Amazon's API. When the thing you need has no translator, an internal service, a small vendor with only a REST interface (representational state transfer, a plain HTTP style of API), a row in a database you want lifecycle-managed, you write the translator yourself, inside your own program. That is a dynamic provider. Do not confuse it with a component resource. A component groups resources that already exist under one logical parent, the way a folder groups files. A dynamic provider invents a genuinely new resource type with its own create, update and delete. Only two language runtimes can do this: Node.js, meaning JavaScript and TypeScript, and Python. Go and .NET cannot.

You implement an object with up to six methods and wrap it in a subclass of dynamic.Resource. check validates inputs before anything happens. create returns an id plus outs, and those outs become the resource's recorded state, read back on the next deployment. diff decides whether a change is an in-place update or a full rebuild. update and delete do what their names say. The sixth is read, the method pulumi refresh calls to ask the real system what exists right now. Only create is required. The rest are optional, and that word does a lot of quiet damage later in this lesson.

import * as pulumi from "@pulumi/pulumi";
 
interface ApiKeyArgs { owner: pulumi.Input<string>; scope: pulumi.Input<string>; }
 
const provider: pulumi.dynamic.ResourceProvider = {
  // runs before anything else: validate inputs, fill in defaults
  async check(_olds, news) {
    const failures: pulumi.dynamic.CheckFailure[] = [];
    if (!["read", "write"].includes(news.scope)) {
      failures.push({ property: "scope", reason: "scope must be read or write" });
    }
    return { inputs: news, failures };
  },
 
  async create(inputs) {
    const { request } = require("undici");            // require INSIDE the body
    const r = await request("https://keys.corp.internal/v1/keys", {
      method: "POST",
      headers: {
        authorization: `Bearer ${process.env.KEYS_TOKEN}`,   // read at run time
        "content-type": "application/json",
      },
      body: JSON.stringify({ owner: inputs.owner, scope: inputs.scope }),
    });
    const key = await r.body.json();
    // outs ARE the state. Whatever you leave out is forgotten.
    return { id: key.id, outs: { owner: inputs.owner, scope: inputs.scope, secret: key.secret } };
  },
 
  async diff(_id, olds, news) {
    return {
      changes: olds.scope !== news.scope || olds.owner !== news.owner,
      replaces: olds.owner !== news.owner ? ["owner"] : [],  // owner is immutable upstream
      deleteBeforeReplace: false,                            // old key works until cutover
    };
  },
 
  async update(_id, olds, news) {
    // PATCH https://keys.corp.internal/v1/keys/<id>  {"scope": news.scope}
    return { outs: { ...olds, scope: news.scope } };         // returned outs REPLACE the state
  },
 
  async delete(_id, _props) {
    // DELETE https://keys.corp.internal/v1/keys/<id>   (runs on pulumi destroy)
  },
};
 
export class ApiKey extends pulumi.dynamic.Resource {
  public readonly secret!: pulumi.Output<string>;
  constructor(name: string, args: ApiKeyArgs, opts?: pulumi.CustomResourceOptions) {
    super(provider, name, { ...args, secret: undefined },
          { ...opts, additionalSecretOutputs: ["secret"] },   // encrypt it in state
          "corp", "ApiKey");                                  // module, type
  }
}

pulumi up --yes --skip-preview

Updating (prod)
 
View Live: https://app.pulumi.com/acme/billing/prod/updates/7
 
     Type                                  Name          Status
 +   pulumi:pulumi:Stack                   billing-prod  created (3s)
 +   └─ pulumi-nodejs:dynamic/corp:ApiKey  deploy-key    created (0.9s)
 
Outputs:
    keyId: "k-8a3f21"
 
Resources:
    + 2 created
 
Duration: 4s

The engine treats this like any other resource. It shows up in the plan, honours protect and dependsOn, and gets deleted on pulumi destroy. Look hard at the Type column, though. Every Node.js dynamic resource reports pulumi-nodejs:dynamic:Resource by default, and every Python one reports pulumi-python:dynamic:Resource. Policy rules match on resource type, so a CrossGuard pack cannot tell your API key apart from anybody else's dynamic resource unless it starts sniffing property names and guessing. Those two extra constructor arguments, "corp" and "ApiKey", buy you the distinct type string in the output above and make the resource policeable. Use them every time.

additionalSecretOutputs is the other line worth copying. Your outs get written to state exactly as returned, so a credential the remote API handed back sits there in plaintext unless you name it as a secret. Naming it means Pulumi encrypts that one property in the state file and masks it in CLI output.

What Happens When You Skip A Method

Say you shipped check, create and diff, and left update for next sprint. Nothing warns you, because only create is required. Someone changes scope from read to write, your diff reports a change with no replacement, and the engine runs an update step against a provider that has no update handler. That step calls nothing. It returns nothing. The engine faithfully records that nothing as the resource's new state.

pulumi stack export | jq -c '.deployment.resources[]
  | select(.type | test("dynamic"))
  | {inputs: .inputs.scope, state: .outputs.scope}'
curl -s -H "authorization: Bearer $KEYS_TOKEN" \
  https://keys.corp.internal/v1/keys/k-8a3f21 | jq -r .scope

{"inputs":"write","state":null}
read

Three different answers to one question. Your program says the key is write. The recorded state says nothing at all, because the missing handler returned an empty object and that emptiness overwrote the outputs. The live key still grants read, and read only. Pulumi printed a green success, and the next preview is clean, so nobody goes looking. This is the failure mode to keep in your head: a dynamic provider is exactly as honest as the methods you actually wrote, and the engine will never tell you which ones are missing.

Now the sting in the tail of that optional read. Refresh works by asking each provider to go and look at the real thing. Skip read and there is nobody home to ask, so the refresh hands back the state Pulumi already had and calls it current. The drift watchdog you built earlier is blind to every dynamic resource you own. Whatever the last create or update wrote into state is what Pulumi believes, indefinitely. If a dynamic resource guards anything that matters, write read, or write a separate reconciliation job that queries the real API and compares, because the engine will not do it for you.

Your State File Now Holds Runnable Code

Here is the part almost nobody knows. Your create, update and delete functions have to run in a separate process from your program, so Pulumi serialises the whole provider object into JavaScript source text and stores it as an ordinary input property named __provider. The next time anything touches that stack, the engine pulls that text back out of state and evaluates it, roughly requireFromString(text).handler(). State stops being a ledger of what was built. It becomes a page of instructions that the next run reads out loud and obeys.

pulumi stack export \
  | jq -r '.deployment.resources[] | select(.type | startswith("pulumi-nodejs:dynamic")) | .inputs.__provider' \
  | head -c 240

exports.handler = __f0;
 
function __f0() {
  return (function() {
    with({ provider: __obj0 }) {
 
return function () { return provider; };
 
    }
  }).apply(undefined, undefined).apply(this, argu

That with({ ... }) block is the interesting part. Pulumi's serialiser walks the free variables your functions reference and writes their values straight into the text, so anything captured from the surrounding module gets baked in at deploy time. An earlier version of this file had const KEYS_TOKEN = process.env.KEYS_TOKEN at the top and used the constant inside create. Here is what that costs you.

pulumi stack export \
  | jq -r '.deployment.resources[] | select(.type | startswith("pulumi-nodejs:dynamic")) | .inputs.__provider' \
  | grep -oE 'sk_live_[A-Za-z0-9]+'

sk_live_7Qd2f9a1c4

Reading process.env.KEYS_TOKEN inside the function body, as the version above does, leaves nothing behind, because process is a global the runtime resolves when the provider actually executes. Same rule for modules. require("undici") inside the body serialises as a require and is resolved on the deploy host, while an HTTP client object built at module scope has to be serialised whole and usually explodes on the way. Keep the provider self-contained, require inside the body, capture nothing live and nothing large.

If you truly cannot avoid capturing something sensitive, add __provider to additionalSecretOutputs and Pulumi encrypts the entire serialised blob in the state file. Encrypted at rest is not the same as hidden, though. pulumi stack export --show-secrets prints it in the clear for anyone holding the passphrase or the encryption key.

One habit to build before you ship any of this. Your create, update and delete functions run inside the deployment process, on whatever host executes it, so the target API sees calls from one shared service credential and your cloud audit log records nothing whatsoever about them. Pass the stack name into the resource as an input property, then log it from inside each handler alongside the resource id and the operation, and send those lines somewhere you already watch. Skip that, and the only surviving record that a key was created, rescoped, or revoked is a diff in a state file written by the same process that did the deed.

Try this

Run command -v pulumi && pulumi version on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: state becomes an execution path. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related