Pulumi in CI/CD & securing secrets

Automate up; protect state and secrets.

Running pulumi up from your laptop is a conversation. Pulumi prints a diff, you read it, you type yes. A build runner has nobody to read it. Automating Pulumi means turning that conversation into a contract: the pipeline gets its own identity, it applies only what a person already read, and it moves secrets around without printing them anywhere. Get the identity or the secrets wrong and your CI/CD system (continuous integration and continuous delivery, the automation that builds, tests, and ships your changes) becomes the softest target in the whole stack. It holds keys. It runs unattended at three in the morning. Its logs are readable by anyone with access to the repository.

Give The Pipeline Its Own Badge

A contractor working in your building does not get a copy of the master key. They get a badge that opens two doors, expires at the end of the shift, and leaves a line in the front desk log every time it touches a reader. That is the shape you want for a build runner (the throwaway machine that executes your pipeline steps). It has two jobs: read and write the Pulumi state (the file recording what Pulumi believes it has already built), and create resources in your cloud account. Those are two separate identities, and the second one is where teams get it wrong.

Take state first. With Pulumi Cloud as the backend, the CLI (command line interface, the pulumi program itself) authenticates with an access token in the PULUMI_ACCESS_TOKEN environment variable. With a self-managed backend, say an S3 bucket (Simple Storage Service, Amazon's object store) that you own, the bucket credentials are what guard the state. Either way, that credential covers state and nothing else. Deploying into AWS, Google Cloud, or Azure needs a second identity, and that is the fussy one, because it is allowed to create, change, and delete real infrastructure.

The lazy version of that second identity is a long-lived access key pasted into a repository secret. It works on the first try, it never expires, and it is exactly what an attacker shops for: one string, valid forever, living inside a system that runs code from pull requests. Use OIDC instead. OpenID Connect works like a visitor pass printed at the front desk on demand rather than a key left under the mat: the pass says who you are, which door you are heading to, and it expires by itself. GitHub hands the running job a short-lived signed token naming the repository, the branch, and the environment it is running for. AWS Security Token Service (STS, the service that issues temporary credentials) checks that signature against a rule you wrote and swaps the token for credentials that are good for an hour by default. Nothing static sits in the repository waiting to be copied out.

# First thing any new pipeline should print: proof of who it thinks it is.
aws sts get-caller-identity
pulumi whoami -v

{
    "UserId": "AROA3XFRBF535PLBIFPI4:GitHubActions",
    "Account": "123456789012",
    "Arn": "arn:aws:sts::123456789012:assumed-role/pulumi-ci-write/GitHubActions"
}
User: pulumi-ci
Organizations: acme
Backend URL: https://app.pulumi.com
Token type: organization

name: deploy
on:
  push:
    branches: [main]
 
permissions:
  id-token: write        # let the runner mint an OIDC token
  contents: read
 
concurrency:
  group: pulumi-webapp-prod   # one apply at a time, no matter how fast people merge
  cancel-in-progress: false
 
jobs:
  up:
    runs-on: ubuntu-22.04
    environment: prod      # protected environment: approval gate + scoped secrets
    steps:
      - uses: actions/checkout@v4
 
      - uses: pulumi/auth-actions@v1          # short-lived Pulumi token, nothing stored
        with:
          organization: acme
          requested-token-type: urn:pulumi:token-type:access_token:organization
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/pulumi-ci-write
          aws-region: us-west-2
 
      - uses: pulumi/actions@v6
        with:
          command: up                          # the Action runs non-interactively
          stack-name: acme/webapp/prod         # <org>/<project>/<stack>
          suppress-outputs: true               # keep stack outputs out of the log

The same trick works for the Pulumi token itself. pulumi/auth-actions trades the runner's OIDC token for a Pulumi access token that lives only as long as the job, so there is no stored token to rotate and none to steal. It works once you have registered GitHub's token issuer as a trusted identity provider in your Pulumi organization settings, which is a one-time piece of setup. If you keep a stored token instead, scope it to a single organization and put its rotation on a calendar you actually honor.

The trust decision lives in your cloud account, not in the workflow file, and it is the piece people copy off a blog post and never tighten. The role's trust policy names which OIDC claims are allowed to assume it. Pin the sub claim (short for subject, the line in the token that says exactly which repository, branch, and environment this job belongs to) to one exact value.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::123456789012:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": "repo:acme/webapp:environment:prod"
      }
    }
  }]
}

That sub value matches only a job that declares environment: prod, which is the environment your approval rules protect. Compare it with what people usually write: repo:acme/webapp:* under StringLike. With the wildcard, every branch and every pull request in the repository can assume the production write role. Anyone who can push a branch, including a compromised bot account or a maintainer whose laptop got popped, adds a workflow file that assumes the role and does whatever it likes, and your audit trail files it as a normal deploy. Give preview and apply two different roles with two different sub conditions.

Preview On The Pull Request, Apply On Merge

Two workflows, two identities. On a pull request (a proposed change waiting for review), run pulumi preview with a role that can read your cloud but not change it, and let the Action post the resource diff as a comment. Reviewers then approve the change itself, not only the code that produced it. On merge to main, run pulumi up with a role that can genuinely write. The split buys you two things. The destructive step only ever runs against a diff a person has already seen, and the preview step cannot mutate anything even if the code it runs is hostile, because the credentials it holds do not allow it.

Keep in mind what preview really does: it executes your Pulumi program. A pull request that edits index.ts gets its code run on your runner with whatever that runner is holding. That is why the preview role is read-only, and why pull requests from forks must never receive secrets or an OIDC token. GitHub already withholds both on the ordinary pull_request trigger. The pull_request_target trigger deliberately undoes that: it runs with the base branch's permissions and full access to secrets, which makes it the wrong choice for any job that checks out contributor code.

name: preview
on:
  pull_request:
    branches: [main]
 
permissions:
  id-token: write
  contents: read
  pull-requests: write   # needed so the diff can be posted as a PR comment
 
jobs:
  preview:
    runs-on: ubuntu-22.04
    # fork PRs get no OIDC token and no secrets, so this job is skipped for them
    if: github.event.pull_request.head.repo.full_name == github.repository
    steps:
      - uses: actions/checkout@v4
 
      - uses: pulumi/auth-actions@v1
        with:
          organization: acme
          requested-token-type: urn:pulumi:token-type:access_token:organization
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::123456789012:role/pulumi-ci-readonly
          aws-region: us-west-2
 
      - uses: pulumi/actions@v6
        with:
          command: preview          # shows the diff, applies nothing
          stack-name: acme/webapp/prod
          diff: true
          comment-on-pr: true
          github-token: ${{ secrets.GITHUB_TOKEN }}

One honest wrinkle: the read-only role is not perfectly read-only. A preview has to decrypt your config secrets to work out the desired state, so it still needs kms:Decrypt on the key that wraps the stack's data key. Grant that, grant Describe* and Get* on the services you actually use, and nothing else. Then attach the write role to a protected GitHub Environment so production applies wait for a named approver instead of firing the second a merge lands.

Pin The Plan A Human Already Read

A preview is a photograph, and the apply happens later. Between the two the scene can move: a second pull request merges, another pipeline updates the same stack, a provider version bump starts computing an input differently. Pulumi can close that gap with update plans. Save the plan the preview produced, then tell the apply it may not do anything the plan did not describe.

export PULUMI_EXPERIMENTAL=true
 
# On the pull request: record exactly what the reviewed diff would do.
pulumi preview --stack acme/webapp/prod --save-plan=plan.json --diff

Previewing update (acme/webapp/prod)
 
     Type                              Name         Plan     Info
     pulumi:pulumi:Stack               webapp-prod
 ~   └─ aws:rds/instance:Instance      db           update   [diff: ~instanceClass]
 
Resources:
    ~ 1 to update
    11 unchanged
 
Duration: 4s

# On merge, with plan.json downloaded from the preview job's artifacts.
# Refuse to do anything the saved plan did not describe.
pulumi up --stack acme/webapp/prod --plan=plan.json --yes

Updating (acme/webapp/prod)
 
     Type                                     Name         Status
     pulumi:pulumi:Stack                      webapp-prod  **failed**
 ~   └─ aws:ec2/securityGroup:SecurityGroup   web          **failed**
 
Diagnostics:
  aws:ec2/securityGroup:SecurityGroup (web):
    error: update is not allowed by the plan: this resource is constrained to same
 
error: update failed

That failure is the feature. Between the review and the apply, something else touched the web security group, so Pulumi stopped instead of folding an unreviewed change into an approved deploy. Two honest limits. The plan is checked against your program and your state file, not against the live cloud, so a change somebody clicked into the console does not trip it (the nightly refresh check further down is what catches that). And update plans sit behind an experimental flag, so a provider that fills in a value at apply time can fail the check for boring reasons. Run pulumi preview --help on the exact CLI version you pin before you make this a required job.

One Writer At A Time

Two updates hitting one stack at once is two cooks writing in the same recipe book with different pens. Pulumi Cloud holds a lock for you and refuses the second one with [409] Conflict: Another update is currently in progress. A self-managed S3 or GCS (Google Cloud Storage) backend does not lock at all by default, so two green pipelines can interleave their writes and leave behind a state file describing a world that never existed. Set PULUMI_SELF_MANAGED_STATE_LOCKING=1 in every job that touches the stack, and add a concurrency group in the workflow so a busy merge queue cannot pile three applies on top of each other.

export PULUMI_SELF_MANAGED_STATE_LOCKING=1
pulumi up --stack prod --yes

error: the stack is currently locked by 1 lock(s). Either wait for the other processes to end or delete the lock file with `pulumi cancel`.
  s3://acme-pulumi-state/.pulumi/locks/organization/webapp/prod/8f1e3c02-4d9a-4a71-b0f5-2c9f0a7b1de4.json: created by runner@fv-az1042-3 at 2026-07-21T09:14:02Z

Read that second line before you clear it. It names the host and the timestamp, which tells you whether a real run is in flight or whether a runner was killed mid-update three hours ago. pulumi cancel --yes removes the lock, and it is safe only in the second case. A cancelled update can leave resources that exist in the cloud but were never recorded in state, so follow every cancel with pulumi refresh before the next apply.

Who Holds The Key To The State

pulumi config set --secret encrypts a value before it ever touches disk. The ciphertext lands in Pulumi.<stack>.yaml beside your ordinary settings, and wherever that value flows into a resource, the state file stores it encrypted too. Good. The question that decides whether any of that helps you is who holds the key. With the default provider the key is a passphrase in PULUMI_CONFIG_PASSPHRASE, and every runner that touches the stack needs it: one shared string, sitting in every job, guarding every secret. A KMS-backed provider (Key Management Service, your cloud's key vault, which will use a key on your behalf but never hands the key to you) moves that key out of the pipeline entirely. State holds ciphertext, and the runner asks the cloud to decrypt through an API call you can scope to one role, rotate on a schedule, and read back in the audit log. When a principal you did not expect starts calling kms:Decrypt on the stack key, that is a detection you can alert on. A shared passphrase gives you no such signal at all.

# Point at your own state bucket, then create the stack with a key you control.
pulumi login s3://acme-pulumi-state
pulumi stack init prod \
  --secrets-provider="awskms://alias/pulumi-secrets?region=us-west-2"
 
pulumi config set aws:region us-west-2
pulumi config set --secret dbPassword 'S3cr3t-from-your-vault'
pulumi config

Logged in to acme-pulumi-state as runner (s3://acme-pulumi-state)
Created stack 'prod'
KEY          VALUE
aws:region   us-west-2
dbPassword   [secret]

Your program passes that config value into the database, so you can read it back out of the state and see what it looks like there. pulumi stack export prints the raw state without decrypting anything, and jq (a command line tool for picking fields out of JSON) pulls out the one input you care about.

# Prove the state holds ciphertext, not the password.
pulumi stack export --stack prod \
  | jq '.deployment.resources[]
        | select(.type == "aws:rds/instance:Instance")
        | .inputs.password'

{
  "4dabf18193072939515e22adb298388d": "1b47061264138c4ac30d75fd1eb44270",
  "ciphertext": "v1:kQ8Zr2mXwF3pT9aC:0f7Yb1nS4uJq6dLxA2hVgR8kM5tPzE3wC1oI9sD0"
}

That pair of hexadecimal strings is Pulumi's marker for an encrypted value, and seeing it with nothing readable beside it is your check that the secrets provider is doing its job. Run the same export against a stack you inherited. Plaintext where you expected ciphertext means somebody set the value with plain pulumi config set and nobody noticed.

# Migrating an existing passphrase stack onto a KMS key.
# The old passphrase must still be readable while this runs.
export PULUMI_CONFIG_PASSPHRASE_FILE=/run/secrets/pulumi-passphrase
pulumi stack change-secrets-provider \
  "awskms://alias/pulumi-secrets?region=us-west-2" --stack prod

Migrating old configuration and state to new secrets provider

Your state backend holds the whole resource graph: every ciphertext, every resource name, every address, every relationship in your estate. It is a map of the building with the alarm panel drawn on it. Lock the bucket down to the CI role plus one break-glass role, switch on object versioning so a corrupt write can be rolled back, and keep the KMS grant narrow enough that reading the bucket without the key hands an attacker nothing but noise.

Treat Every Build Log As Public

Build logs get pasted into tickets, forwarded to log aggregators, and read by everyone with repository access. Pulumi masks secret values in its own output and prints [secret] instead, so the default path is safe. The unsafe paths are the ones you type yourself.

pulumi stack output
 
# Never run either of these in a pipeline. Both print plaintext to stdout:
#   pulumi stack output dbPassword --show-secrets
#   pulumi config get dbPassword

Current stack outputs (2):
    OUTPUT       VALUE
    bucketName   assets-8a3f1c2
    dbPassword   [secret]

Pulumi carries secretness through the program, so an output computed from a secret stays secret. A console.log inside an apply does not care. It writes the plaintext to stdout, and GitHub only masks values it was told about in advance. If you genuinely need to surface something derived, register it first with echo "::add-mask::$value" before anything else prints it. Pass --suppress-outputs on the apply so stack outputs never render at all, and keep --show-secrets out of every script a runner can reach.

Verify It Before You Trust It

Four checks turn this from a diagram into something you believe. Start with the preview role: assume it locally and try to write something trivial. The --dry-run flag on an AWS EC2 (Elastic Compute Cloud) call is built for exactly this, because it runs the permission check and then stops before touching anything.

aws ec2 authorize-security-group-ingress --dry-run \
  --group-id sg-0a1b2c3d4e5f67890 \
  --protocol tcp --port 22 --cidr 10.0.0.0/8

An error occurred (UnauthorizedOperation) when calling the AuthorizeSecurityGroupIngress operation: You are not authorized to perform this operation. User: arn:aws:sts::123456789012:assumed-role/pulumi-ci-readonly/preview is not authorized to perform: ec2:AuthorizeSecurityGroupIngress on resource: arn:aws:ec2:us-west-2:123456789012:security-group/sg-0a1b2c3d4e5f67890 because no identity-based policy allows the ec2:AuthorizeSecurityGroupIngress action.

UnauthorizedOperation is the answer you want. DryRunOperation means the call would have succeeded and your read-only role is not read-only. Next, go and read what the cloud recorded when the runner logged in. CloudTrail (AWS's own audit log of API calls) stores the sub claim from the OIDC token on every AssumeRoleWithWebIdentity event, which is where a too-wide trust policy shows itself.

aws cloudtrail lookup-events --region us-west-2 --max-results 1 \
  --lookup-attributes AttributeKey=EventName,AttributeValue=AssumeRoleWithWebIdentity \
  --query 'Events[0].CloudTrailEvent' --output text | jq '.userIdentity'

{
  "type": "WebIdentityUser",
  "principalId": "token.actions.githubusercontent.com:repo:acme/webapp:environment:prod",
  "userName": "repo:acme/webapp:environment:prod",
  "identityProvider": "token.actions.githubusercontent.com"
}

Read userName on every one of those events. If it says environment:prod on the applies and pull_request on the previews, your trust policies are doing their job. If you see a branch name you do not recognise assuming the write role, you have found the wildcard. The third check is the state export from earlier, run against every stack you own. The fourth is a scheduled drift check, and drift here means the gap between what your code says and what actually exists, which is mostly created by people clicking in the console.

# Nightly drift job. --refresh is what makes this real: without it you are
# comparing your code against the state file, not against the cloud.
pulumi preview --stack acme/webapp/prod \
  --refresh --expect-no-changes --non-interactive
echo "exit code: $?"

Previewing update (acme/webapp/prod)
 
     Type                                     Name         Plan     Info
     pulumi:pulumi:Stack                      webapp-prod
 ~   └─ aws:ec2/securityGroup:SecurityGroup   web          update   [diff: ~ingress]
 
Resources:
    ~ 1 to update
    11 unchanged
 
error: no changes were expected but changes occurred
exit code: 1

Run that at 06:00 UTC, have the job open an issue on any non-zero exit with the resource URN (uniform resource name, Pulumi's unique address for one resource) in the title, and treat the first failure as a question about who still has console access to production rather than as a flaky job.

Try this

Run aws sts get-caller-identity on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a lost passphrase is a lost stack. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related