State, stacks & the up/preview loop

How Pulumi tracks reality.

Pulumi never walks into your cloud account and counts what is there before a run. It reads a ledger it wrote for itself the last time a deployment finished. That ledger is a file called the checkpoint, written in JSON (JavaScript Object Notation, a plain-text format for structured data), and it lists every resource Pulumi created, the properties each one ended up with, and which resource depends on which. A warehouse runs the same way. The manager does not recount ten thousand crates every morning. She keeps a book, and the book is only as honest as the last person who wrote in it.

So there are always three descriptions of your infrastructure in play. Your program says what should exist: the desired state. The checkpoint says what Pulumi believes it built: the last-known state. The provider's API (Application Programming Interface, the machine-readable control surface your cloud exposes to tools) holds what actually exists right now: the actual state. Nearly every confusing Pulumi moment clears up the second you work out which two of those three the command in front of you is comparing.

The Checkpoint Is the Ledger, Not the Warehouse

Where the ledger lives is your choice. By default it goes to Pulumi Cloud, a hosted backend that also handles locking and update history. You can keep it on storage you own instead, which Pulumi calls a self-managed (or DIY) backend, addressed by a URL (Uniform Resource Locator, the same kind of address a browser takes): a local directory (file://), an Amazon S3 bucket (Simple Storage Service, s3://), Azure Blob Storage (azblob://), or Google Cloud Storage (gs://). Local disk is the easiest way to look at the thing with your own eyes, so start there. Sort one thing out before you type: a self-managed backend encrypts secrets with a passphrase you supply, so point PULUMI_CONFIG_PASSPHRASE_FILE at a file holding that passphrase or the CLI (command-line interface) stops and prompts you.

# keep the passphrase in a root-owned file, not in your shell history
export PULUMI_CONFIG_PASSPHRASE_FILE=/etc/pulumi/dev.passphrase
 
# put the ledger on local disk so we can read it
pulumi login file://~
pulumi stack init dev
pulumi up --yes
ls -l ~/.pulumi/stacks/acme-net/

Logged in to ops-runner as sachin (file://~)
Created stack 'dev'
 
Updating (dev)
 
     Type                      Name          Status
 +   pulumi:pulumi:Stack       acme-net-dev  created (18s)
 +   ├─ aws:ec2:SecurityGroup  web-sg        created (4s)
 +   └─ aws:ec2:Instance       web           created (13s)
 
Outputs:
    sgId: "sg-0a1b2c3d4e5f67890"
 
Resources:
    + 3 created
 
Duration: 21s
 
total 16
-rw------- 1 sachin sachin 8914 Jul 21 10:12 dev.json
-rw------- 1 sachin sachin  341 Jul 21 10:12 dev.json.bak

The program behind that run is small on purpose: one security group and one EC2 (Elastic Compute Cloud, Amazon's virtual machines) instance on AWS (Amazon Web Services). dev.json is the entire memory of the stack, and Pulumi keeps the previous copy next to it as dev.json.bak every time it writes. Trimmed down to a single resource, the parts that matter look like this.

{
  "version": 3,
  "deployment": {
    "manifest": { "time": "2026-07-21T10:12:44.118+05:30", "version": "v3.148.0" },
    "secrets_providers": { "type": "passphrase", "state": { "salt": "v1:9pQ2xK1c/2A=:v1:..." } },
    "resources": [
      {
        "urn": "urn:pulumi:dev::acme-net::aws:ec2/securityGroup:SecurityGroup::web-sg",
        "custom": true,
        "id": "sg-0a1b2c3d4e5f67890",
        "type": "aws:ec2/securityGroup:SecurityGroup",
        "inputs":  { "name": "web-sg", "vpcId": "vpc-0c9d8e7f", "ingress": [ ... ] },
        "outputs": { "arn": "arn:aws:ec2:ap-south-1:...", "ingress": [ ... ] },
        "parent": "urn:pulumi:dev::acme-net::pulumi:pulumi:Stack::acme-net-dev",
        "provider": "urn:pulumi:dev::acme-net::pulumi:providers:aws::default_6_66_2::04da6b54-80e4-46f7-96ec-b56ff0331ba9",
        "dependencies": [],
        "created":  "2026-07-21T10:12:41.000Z",
        "modified": "2026-07-21T10:12:44.000Z"
      }
    ]
  }
}

Two things in there matter for security. The first is the URN (Uniform Resource Name), the long identifier at the top of every entry. Read it left to right: stack, project, resource type, then the logical name you gave the object in your code. That string is how Pulumi matches a live object in your running program to a row in the ledger, run after run. The second is that everything you did not mark secret sits there in plain text. Resource identifiers, network ranges, ARNs (Amazon Resource Names, the unique identifier AWS stamps on every object), and every output are all readable by anyone who can read the file.

A value you did mark secret is stored as ciphertext instead, wrapped in a small envelope. Pulumi tags special values with the signature key 4dabf18193072939515e22adb298388d, and that key covers several kinds of special value: file assets, archives, references to other resources, and secrets. The one that means secret is the value 1b47061264138c4ac30d75fd1eb44270 sitting under that key, next to a ciphertext field. Grep for that pair when you want proof a password really was encrypted rather than merely believed to be.

"outputs": {
  "dbPassword": {
    "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
    "ciphertext": "v1:kM1t7bF9r0Yb2Xn/:Cw8n1cGq9Yy4Rr2pF6sT0aZq..."
  }
}

Everything else in the file is cleartext, so treat the state store the way you treat a production database. Private bucket, encryption at rest, object versioning on, access logged. Which secrets provider you choose per stack, and how you rotate it, is its own subject.

Stacks Are Separate Ledgers for the Same Program

A stack is one independent deployment of your program. dev, staging and prod are three stacks running identical code with different config and, more to the point, their own checkpoint. No stack can write into another stack's ledger. It can read across, in a controlled way: a StackReference lets one stack consume another's published outputs, which is a one-way window rather than a shared book. That separation is why a botched dev run cannot quietly reach into prod, and it is also why every command below acts on whichever stack happens to be selected, whether you thought about that or not.

pulumi stack init prod          # note: init also selects the new stack
pulumi stack select dev         # so switch back before doing anything else
pulumi stack ls
pulumi stack --show-urns

Created stack 'prod'
 
NAME  LAST UPDATE    RESOURCE COUNT  URL
dev*  4 minutes ago  4               file://~
prod  n/a            n/a             file://~
 
Current stack is dev:
    Managed by ops-runner
    Last updated: 4 minutes ago (2026-07-21 10:12:44.118 +0530 IST)
    Pulumi version used: v3.148.0
 
Current stack resources (4):
    TYPE                                 NAME
    pulumi:pulumi:Stack                  acme-net-dev
        URN: urn:pulumi:dev::acme-net::pulumi:pulumi:Stack::acme-net-dev
    pulumi:providers:aws                 default_6_66_2
        URN: urn:pulumi:dev::acme-net::pulumi:providers:aws::default_6_66_2
    aws:ec2/securityGroup:SecurityGroup  web-sg
        URN: urn:pulumi:dev::acme-net::aws:ec2/securityGroup:SecurityGroup::web-sg
    aws:ec2/instance:Instance            web
        URN: urn:pulumi:dev::acme-net::aws:ec2/instance:Instance::web
 
Current stack outputs (1):
    OUTPUT  VALUE
    sgId    sg-0a1b2c3d4e5f67890

The star next to dev is the selected stack, and selection is a setting your workspace remembers between commands, so read that star before you run anything destructive. The count says four resources while the update said three created, and both are right: the ledger also tracks the default AWS provider that Pulumi configured for you, which never shows up in the update summary.

Look at what the URN carries: the logical name you typed in code, not the cloud identifier. Rename a resource from web-sg to edge-sg in your program and its URN changes with it, so Pulumi sees one resource that vanished and one that appeared. Preview will show you a delete and a create, not a rename, and up will carry that out happily. When you want to rename in code without churning real infrastructure, set the aliases resource option to the old URN so Pulumi knows the two rows describe the same crate. Export the checkpoint before you try it, either way.

Preview, Then Up

The daily loop is two commands. preview runs your whole program, builds the resource graph it describes, diffs that graph against the checkpoint, and prints what would happen without touching anything. up does the same diff and then carries it out, walking the dependency graph in order. The change below is the one worth studying, because it mixes an in-place update with a replacement. Here the logical name stays put and the security group's real name property changes, which AWS cannot alter on a group that already exists.

pulumi preview --diff

Previewing update (dev)
 
     Type                      Name          Plan     Info
     pulumi:pulumi:Stack       acme-net-dev
 +-  ├─ aws:ec2:SecurityGroup  web-sg        replace  [diff: ~name]
 ~   └─ aws:ec2:Instance       web           update   [diff: ~vpcSecurityGroupIds]
 
    +-aws:ec2/securityGroup:SecurityGroup: (replace)
        [id=sg-0a1b2c3d4e5f67890]
        [urn=urn:pulumi:dev::acme-net::aws:ec2/securityGroup:SecurityGroup::web-sg]
      ~ name: "web-sg" => "edge-sg"
 
    ~ aws:ec2/instance:Instance: (update)
        [id=i-0f4c2a9b8d7e61532]
        [urn=urn:pulumi:dev::acme-net::aws:ec2/instance:Instance::web]
      ~ vpcSecurityGroupIds: [
          ~ [0]: "sg-0a1b2c3d4e5f67890" => output<string>
        ]
 
Resources:
    +-1 to replace
    ~ 1 to update
    2 changes. 2 unchanged

The +- marker means replace. The provider cannot change that property on a living resource, so Pulumi builds a new one and destroys the old. Default order is create first, delete second, which usually avoids a gap in service. You can see the knock-on effect right below it: the instance has to be pointed at the new group, and at preview time the new group does not have an id yet, which is why the diff prints output<string> instead of a value. Watch replacements closely. When a name has to be globally unique, or the resource carries the deleteBeforeReplace option, the order flips and you get a real outage window. Replace a database or a volume and the data goes with it.

Two habits are worth building here. First, up writes the checkpoint as each resource settles rather than once at the end, so a run that dies halfway still records what succeeded and the next run resumes instead of starting from nothing. Second, pulumi up --target 'urn:...' applies only the resources you name and skips the rest of your program, which makes it a repair tool rather than a deploy tool. Reach for it when one resource is wedged, not when you are shipping.

pulumi up --yes                       # no confirmation prompt; how continuous integration runs it
pulumi up --target 'urn:pulumi:dev::acme-net::aws:ec2/instance:Instance::web'
pulumi up --expect-no-changes         # fail loudly if anything would change

Updating (dev)
 
     Type                      Name          Status         Info
     pulumi:pulumi:Stack       acme-net-dev
 +-  ├─ aws:ec2:SecurityGroup  web-sg        replaced (8s)  [diff: ~name]
 ~   └─ aws:ec2:Instance       web           updated (11s)  [diff: ~vpcSecurityGroupIds]
 
Outputs:
  ~ sgId: "sg-0a1b2c3d4e5f67890" => "sg-0f9e8d7c6b5a43210"
 
Resources:
    +-1 replaced
    ~ 1 updated
    2 changes. 2 unchanged
 
Duration: 24s

Drift Is the Gap Between the Ledger and the Warehouse

Somebody opens the AWS console at 2am during an incident and adds SSH (Secure Shell, the standard remote login protocol, which listens on port 22) from 0.0.0.0/0, meaning every address on the internet, to your production security group. Nothing about your program or your checkpoint changes, because the edit happened out in the warehouse and nobody wrote in the book. Run pulumi preview the next morning and it reports no changes, confidently and wrongly. It compared two things that still match each other. That gap is drift, and refresh is the command that closes it: Pulumi asks the provider about every resource it tracks and rewrites the checkpoint to match whatever comes back.

pulumi refresh --yes --diff

Refreshing (dev)
 
     Type                      Name          Status   Info
     pulumi:pulumi:Stack       acme-net-dev
 ~   ├─ aws:ec2:SecurityGroup  web-sg        updated  [diff: ~ingress]
     └─ aws:ec2:Instance       web
 
    ~ aws:ec2/securityGroup:SecurityGroup: (refresh)
        [id=sg-0f9e8d7c6b5a43210]
      ~ ingress: [
          + [1]: {
                  + cidrBlocks: [ "0.0.0.0/0" ]
                  + fromPort  : 22
                  + protocol  : "tcp"
                  + toPort    : 22
                }
        ]
 
Resources:
    ~ 1 updated
    3 unchanged
 
Duration: 6s

Now the ledger tells the truth, and the next preview finally disagrees with it. Pulumi wants to take that rule back out, because your program never asked for port 22 to be open to the world. Run pulumi up --refresh to read and reconcile in one pass, or set options.refresh: always in Pulumi.yaml so every preview, update and destroy starts from a fresh read. Each read costs one provider round trip per resource, which drags on a large stack, so plenty of teams put it on a schedule rather than pay for it on every deploy. If you only want to be told about drift and would rather not have a job rewriting prod state at 3am, pulumi preview --refresh does the read and throws it away, because preview never writes the checkpoint.

name: acme-net
runtime: nodejs
description: Edge network for acme
backend:
  url: file://~
options:
  refresh: always

Locks, Stuck Runs and State Surgery

Because the checkpoint is a single shared file, two updates writing at once would shred it. So Pulumi hangs a key on a hook: it takes a lock on the stack for the length of a run, and the second run has to wait for the key. Pulumi Cloud refuses the second update with an HTTP 409 Conflict (Hypertext Transfer Protocol, the web's request format; 409 is its standard "someone got there first" reply). A self-managed backend drops a lock file in the backend under .pulumi/locks/. Press Ctrl-C twice, or lose the runner mid-deploy, and that lock outlives the process that took it.

pulumi up --yes

error: the stack is currently locked by 1 lock(s). Either wait for the other process(es)
to end or delete the lock file with `pulumi cancel`.
  file:///home/sachin/.pulumi/locks/organization/acme-net/dev/a3f1c9e0-2b44-4f6d-9a17-6d0f2c8b91de.json:
  created by sachin@ops-runner (pid 41883) at 2026-07-21T10:31:02Z

pulumi cancel releases it. Confirm the other process really is dead first, because cancelling a live update leaves you with a checkpoint that describes a deployment still in flight. The rest of the surgery kit is short and sharp: stack export and stack import round-trip a malformed checkpoint, state delete stops tracking a resource while leaving the cloud object alive and running, state unprotect lifts the protect flag so a resource can be destroyed, and import adopts an existing cloud resource into state. When you are handing a resource off permanently, reach for the retainOnDelete resource option instead of editing state by hand.

pulumi stack export --file /var/backups/pulumi/dev-$(date +%F).json   # silent on success
pulumi stack export | pulumi stack import                            # round-trip to repair
pulumi state delete --yes 'urn:pulumi:dev::acme-net::aws:ec2/instance:Instance::web'
pulumi state unprotect --yes 'urn:pulumi:dev::acme-net::aws:ec2/instance:Instance::web'

error: This resource can't be safely deleted because the following resources depend on it:
 * "web-eip" (urn:pulumi:dev::acme-net::aws:ec2/eip:Eip::web-eip)
 
Delete those resources first or pass --target-dependents.
 
Resource unprotected successfully

Put the Drift Check on a Timer

A drift gate is worth exactly as much as its schedule. Two commands do the work: refresh to make the ledger honest, then preview --expect-no-changes, which fails the moment your program and the refreshed checkpoint disagree (the Pulumi CLI exits 255 on that kind of error, which is all a scheduler needs to see). On a Linux runner, systemd (the service and timer manager that starts everything on Ubuntu, Debian and most modern distributions) turns those two lines into an alarm nobody can forget to run.

[Unit]
Description=Pulumi drift check for acme-net/prod
Wants=network-online.target
After=network-online.target
 
[Service]
Type=oneshot
User=pulumi
WorkingDirectory=/srv/infra/acme-net
Environment=PULUMI_SKIP_UPDATE_CHECK=true
EnvironmentFile=/etc/pulumi/drift.env
ExecStart=/usr/local/bin/pulumi refresh --stack prod --yes --diff --non-interactive
ExecStart=/usr/local/bin/pulumi preview --stack prod --expect-no-changes --non-interactive
NoNewPrivileges=true
PrivateTmp=true
ProtectSystem=strict
ReadWritePaths=/home/pulumi/.pulumi /srv/infra/acme-net

[Unit]
Description=Nightly Pulumi drift check
 
[Timer]
OnCalendar=*-*-* 03:15:00
RandomizedDelaySec=300
Persistent=true
 
[Install]
WantedBy=timers.target

# drift.env holds the cloud credentials and PULUMI_CONFIG_PASSPHRASE
sudo chown root:root /etc/pulumi/drift.env && sudo chmod 0600 /etc/pulumi/drift.env
sudo systemctl daemon-reload
sudo systemctl enable --now pulumi-drift.timer
systemctl list-timers pulumi-drift.timer
journalctl -u pulumi-drift.service -n 8 --no-pager

Created symlink /etc/systemd/system/timers.target.wants/pulumi-drift.timer → /etc/systemd/system/pulumi-drift.timer.
 
NEXT                        LEFT     LAST                        PASSED  UNIT               ACTIVATES
Wed 2026-07-22 03:15:00 IST 15h left Tue 2026-07-21 03:15:00 IST 8h ago  pulumi-drift.timer pulumi-drift.service
 
1 timers listed.
 
Jul 21 03:15:04 ops-runner pulumi[41883]: Refreshing (prod)
Jul 21 03:15:09 ops-runner pulumi[41883]:  ~  aws:ec2:SecurityGroup  web-sg  updated  [diff: ~ingress]
Jul 21 03:15:11 ops-runner pulumi[41891]: Previewing update (prod)
Jul 21 03:15:14 ops-runner pulumi[41891]:  ~  aws:ec2:SecurityGroup  web-sg  update   [diff: ~ingress]
Jul 21 03:15:14 ops-runner pulumi[41891]: error: no changes were expected but changes occurred
Jul 21 03:15:14 ops-runner systemd[1]: pulumi-drift.service: Main process exited, code=exited, status=255/n/a
Jul 21 03:15:14 ops-runner systemd[1]: pulumi-drift.service: Failed with result 'exit-code'.

Type=oneshot is what lets one unit run both ExecStart lines in order and stop at the first failure, which is exactly the shape of refresh-then-check. The credentials file can stay 0600 root:root, unreadable by the service account, because systemd reads EnvironmentFile as root before it drops to the pulumi user. ProtectSystem=strict makes the whole filesystem read-only for that process, so the two ReadWritePaths entries are the only places it can write: the plugin cache and the project directory. Add OnFailure=drift-alert@%n.service to the unit and that status=255 line stops being a log entry nobody opens and starts being a page at 3:15am.

Try this

Run pulumi login file://~ on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: refresh can launder someone else's change. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related