SOPS in CI/CD

Decrypt safely in pipelines.

A hotel does not hand the cleaning crew a master key that opens every room forever. Each cleaner gets a card cut for one floor, good for one shift, and every door records which card touched it. That is the arrangement your build pipeline should have with your secrets. The job needs to open one SOPS (Secrets OPerationS) file, once, to ship a release. It never needs a key it gets to keep.

In GitOps (the pattern where a controller running inside your cluster pulls the desired state from Git and applies it), the cluster does the decrypting and CI (continuous integration, the automated system that builds and ships your code) never sees a plaintext value. Plenty of work sits outside that pattern. A Terraform apply needs a cloud provider credential. A deploy script needs an API (application programming interface) token. An Ansible run needs host passwords. There, the pipeline itself is the consumer, so the pipeline has to decrypt. The only interesting question left is how briefly, and how narrowly.

Three properties make that decryption safe, and you need all three. No standing decryption key sitting on the runner or in the CI secret store. Plaintext that exists inside one process, for the length of one command. Nothing decrypted written to a log, an artifact, or the disk. Drop any one and the other two stop paying for themselves. Fifteen-minute credentials do not save you if the deploy script prints the database password into a build log that a stranger can read.

Borrow the Key, Never Keep It

The first thing most teams try is pasting an age private key into the CI secret store as SOPS_AGE_KEY. (age is the file encryption tool SOPS leans on when there is no cloud key service around; its public halves look like age1... and its private halves like AGE-SECRET-KEY-1....) It works on the first afternoon. Now look at what you built. That is a permanent decryption key parked inside a system that runs code written by everyone on the team, on runners that pull down dozens of third-party actions. Anyone who can merge a workflow change can print it, base64 it, and ship it somewhere. And age decryption is local arithmetic on the runner, so nothing anywhere records that it happened.

There is another way to open a door. A courier with no key shows photo ID at the front desk, and the desk issues a badge that works on one floor until five o'clock. OIDC (OpenID Connect, a standard way for one system to prove who it is to another without a shared password) is that arrangement, for machines. Your CI platform signs a short-lived token describing the running job: this repository, this branch, this environment. You configure AWS to trust that issuer, and to swap a token matching one exact description for temporary credentials from STS (Security Token Service, the AWS desk that issues short-term keys). Those credentials expire in fifteen minutes and carry exactly one permission, kms:Decrypt on one key. KMS (Key Management Service) is the locked cabinet that never lets a key leave the building: you hand it a wrapped data key, it hands back the unwrapped one. Nothing sensitive rests on the runner between jobs, and every unwrap lands in CloudTrail (AWS's ledger of every API call made in the account) with the job's session name attached.

name: deploy
on:
  push:
    branches: [main]
 
permissions: {}                    # deny everything by default; opt in per job
 
jobs:
  deploy:
    runs-on: ubuntu-24.04
    environment: production        # lands in the OIDC subject; gate it with reviewers
    permissions:
      contents: read               # clone the repo, nothing else
      id-token: write              # mint the OIDC token (no token request without it)
    steps:
      - uses: actions/checkout@v4
      - uses: sigstore/cosign-installer@v4   # pin third-party actions to a full SHA
 
      # releases live under getsops/ since the project left Mozilla for the CNCF.
      # v3.7.3 was the last Mozilla release, so old mozilla/sops URLs are frozen.
      - name: install sops, verified before we trust it with decrypt rights
        run: |
          v=3.13.2
          base=https://github.com/getsops/sops/releases/download/v${v}
          curl -fsSLO $base/sops-v${v}.linux.amd64
          curl -fsSLO $base/sops-v${v}.checksums.txt
          curl -fsSLO $base/sops-v${v}.checksums.sigstore.json
          cosign verify-blob sops-v${v}.checksums.txt \
            --bundle sops-v${v}.checksums.sigstore.json \
            --certificate-identity-regexp '^https://github\.com/getsops/sops/\.github/workflows/' \
            --certificate-oidc-issuer https://token.actions.githubusercontent.com
          sha256sum --ignore-missing -c sops-v${v}.checksums.txt
          sudo install -m 0755 sops-v${v}.linux.amd64 /usr/local/bin/sops
 
      - uses: aws-actions/configure-aws-credentials@v4
        with:
          role-to-assume: arn:aws:iam::111122223333:role/ci-sops-decrypt
          aws-region: us-east-1
          role-session-name: gha-deploy-${{ github.run_id }}
          role-duration-seconds: 900       # 900s is the shortest AWS will issue
 
      - name: deploy
        run: sops exec-env secrets/prod.enc.env './deploy.sh'

Read the permissions block first, because it is the part people copy without understanding. id-token: write is what lets the job mint an OIDC token at all. Leave it out and the credentials step dies on the runner, before AWS is ever contacted, because the token endpoint is not exposed to the job. contents: read buys a clone and nothing else. The environment: production line carries more weight than it looks: it becomes part of the token's subject claim (the string that says who this token belongs to), so your trust policy can accept only jobs targeting that environment, and if you put required reviewers on the environment, GitHub holds the job at that gate before any token exists.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Effect": "Allow",
    "Principal": {
      "Federated": "arn:aws:iam::111122223333:oidc-provider/token.actions.githubusercontent.com"
    },
    "Action": "sts:AssumeRoleWithWebIdentity",
    "Condition": {
      "StringEquals": {
        "token.actions.githubusercontent.com:aud": "sts.amazonaws.com",
        "token.actions.githubusercontent.com:sub": "repo:acme/payments-api:environment:production"
      }
    }
  }]
}

That trust policy is the door itself. StringEquals on the full subject, rather than StringLike with a wildcard, is the difference between "this one repository, deploying to production" and "anything anyone runs anywhere in the organization". A wildcard here is the most common way an OIDC setup gets quietly widened until it is no better than the stored key it replaced. Behind the door sits the permission policy, and it is deliberately tiny.

{
  "Version": "2012-10-17",
  "Statement": [{
    "Sid": "DecryptProductionSopsFilesOnly",
    "Effect": "Allow",
    "Action": "kms:Decrypt",
    "Resource": "arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7890-ab12-cd34ef567890",
    "Condition": {
      "StringEquals": { "kms:EncryptionContext:env": "production" }
    }
  }]
}

# path_regex is matched against the file path relative to the directory holding
# .sops.yaml, so a nearer .sops.yaml further down the tree, not your working
# directory, is what changes the answer.
creation_rules:
  # production files: one KMS key, stamped with an encryption context
  - path_regex: ^secrets/prod\.
    key_groups:
      - kms:
          - arn: arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7890-ab12-cd34ef567890
            context:
              env: production
 
  # staging files: SAME key, different context, so a different role opens them
  - path_regex: ^secrets/staging\.
    key_groups:
      - kms:
          - arn: arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7890-ab12-cd34ef567890
            context:
              env: staging

An encryption context is a coat check ticket. KMS writes the label onto the envelope when it wraps your data key, and demands the same label back before it will unwrap it. SOPS copies whatever you put under context: into the file's metadata, sends it with every decrypt call, and KMS refuses if the two disagree. Pair that with the IAM (Identity and Access Management, the AWS rulebook for who may call what) condition above and one key can serve every environment while each pipeline role opens only its own files. Notice what is absent from that policy, too. There is no kms:Encrypt. SOPS needs Encrypt to wrap a new data key, so a decrypt-only role can read production secrets but cannot mint a fresh file encrypted to a key an attacker controls, and cannot rewrite the files you already have.

Decrypt Into the Process, Not Onto the Disk

The move to avoid is sops decrypt secrets.enc.env > .env. That plaintext now lives on the runner's filesystem for the rest of the job. It gets swept up by an upload-artifact step pointed at the wrong directory. It survives in a cached workspace on a self-hosted runner. It sits one careless git add -A away from the repository it was encrypted to stay out of. sops exec-env skips the file entirely: it decrypts in memory, sets the values as environment variables on a child process, runs your command, and the values die with that process.

# what is actually committed: names in the clear, values encrypted
head -2 secrets/prod.enc.env
 
# prove the value reached the child process, without printing the value
sops exec-env secrets/prod.enc.env 'printenv STRIPE_API_KEY | wc -c'

DATABASE_URL=ENC[AES256_GCM,data:Xr9tQmKcC1nR7fLd0pVb8yHs2wAeGjU4mZq6TvB3xNdP5cRk1YoW7SgF2hLiE8uJ0aQzM4bXnT6yVpCr,iv:1ehsf9AKvfeSSGnp9J45c1H8Ba/uW9GTuZBGCVHBFn8=,tag:aTJ8d3wJ7IqBa1AJdercMg==,type:str]
STRIPE_API_KEY=ENC[AES256_GCM,data:Lm2pR8vC0dQz4Ky7WsX3jNbT9aHu1FgE5oPzM6iDvR8=,iv:9RmT2xQe5bVn8ZpL1cWd3AoU6yGh0KjS4fXr7NtMpQw=,tag:Zk4VhgQ2sP9dLmR8xTuA1w==,type:str]
33

Steal that wc -c trick. It proves the variable arrived in the child process and prints one number instead of a live credential, which is what you want while debugging a pipeline that handles secrets. Look at the file itself too, because this is the part people have backwards. SOPS encrypts values. It never encrypts keys. DATABASE_URL and STRIPE_API_KEY sit there in the clear, and so does the shape of the file, which is precisely what makes a pull request reviewable: you can see that someone added a Stripe key without seeing the key. The cost is that anyone with read access to the repository gets an inventory of which secrets you hold. Comments get encrypted along with the values, which surprises most people the first time they open a file.

exec-env writes through the SOPS dotenv writer, one KEY=value line per entry, so it wants flat data. Point it at a YAML file with nesting under a key and it stops rather than flattening or guessing. For structured files you have two better moves. sops exec-file decrypts to a temporary path and substitutes that path for {} in your command, streaming the bytes through a FIFO (a named pipe, which behaves like a chute between two rooms: bytes pass through, nothing stays in the chute) unless you pass --no-fifo for tools that stat the file or read it twice. --filename gives that temporary file whatever name the tool insists on. Or pull one value out with --extract, so only that one value reaches stdout, even though SOPS still decrypts the whole tree in memory.

# Terraform wants a real file it can stat and re-read, with a name it recognizes
sops exec-file --no-fifo --filename prod.tfvars.json secrets/prod.enc.json \
  'terraform apply -var-file={} -auto-approve'

aws_db_parameter_group.payments: Modifying... [id=payments-prod]
aws_db_parameter_group.payments: Modifications complete after 4s [id=payments-prod]
 
Apply complete! Resources: 0 added, 1 changed, 0 destroyed.

# print one value instead of the whole file; the whole tree is still decrypted
sops decrypt --extract '["database"]["password"]' secrets/prod.enc.yaml | wc -c
 
# --pristine: the child gets the decrypted values and nothing else the runner had
sops exec-env --pristine secrets/prod.enc.env \
  'printenv DATABASE_URL | wc -c; /usr/local/bin/aws sts get-caller-identity'

25
61
Unable to locate credentials. You can configure credentials by running "aws configure".

That second run is --pristine showing both of its faces at once. The decrypted DATABASE_URL arrived, all 60 characters of it. The AWS_* credentials that the OIDC step exported into the job did not, which is why the AWS call has nothing to authenticate with. Reach for --pristine when the command is a self-contained binary. Do not reach for it when the command is a shell script that expects a normal environment, unless you enjoy rebuilding that environment one variable at a time.

The Log Is Where Secrets Actually Leak

GitHub Actions redacts values that came from ${{ secrets.* }} because it was handed those exact strings in advance. A value you decrypt at runtime is invisible to it. Nothing is masked for you, and that catches people out. So set -x in a deploy script, a printenv somebody added while debugging, curl -v printing an Authorization header, a stack trace that dumps config: every one of them writes a live credential into a build log that is kept for months and, on a public repository, readable by anyone with a browser.

#!/usr/bin/env bash
set -euo pipefail          # NOT set -x: xtrace prints commands AFTER expansion
 
# fail fast on a missing value without ever echoing it
: "${DATABASE_URL:?DATABASE_URL not set - did sops exec-env run?}"
 
# register a mask so the runner redacts this string from every later log line
echo "::add-mask::$STRIPE_API_KEY"
 
curl -sS --fail-with-body \
  -H "Authorization: Bearer $STRIPE_API_KEY" \
  https://api.stripe.com/v1/charges >/dev/null
 
psql "$DATABASE_URL" -c 'select 1' >/dev/null
echo "deploy ok"

echo "::add-mask::$VALUE" tells the runner to redact that string from every line after it, and the runner swallows the add-mask line itself instead of printing it. Treat it as a backstop, not a plan. It matches the exact string only, so a token that gets URL-encoded, base64'd, or wrapped across two lines walks straight past it, and anything printed before you registered the mask is already in the log forever. The plan is to never print secrets. The mask is for the day someone else edits your script.

# filestatus reads metadata only; it never needs to decrypt anything
sops filestatus secrets/prod.enc.env
sops filestatus secrets/new-service.env
 
# CI guard: nothing under secrets/ is allowed to be plaintext
rc=0
for f in secrets/*; do
  if [ "$(sops filestatus "$f" | jq -r .encrypted)" != "true" ]; then
    echo "NOT ENCRYPTED: $f"; rc=1
  fi
done
exit $rc

{"encrypted":true}
{"encrypted":false}
NOT ENCRYPTED: secrets/new-service.env

sops filestatus, which arrived in 3.9.0, reports whether a file carries valid SOPS metadata: a version, the key groups, and a MAC (message authentication code, a tamper-evident seal computed over the values). It answers the question and returns quietly either way, so read the JSON yourself if you want the build to stop. Running it across the whole directory turns "we assume everyone remembered to encrypt" into a step that fails out loud. That last line is a plaintext credential somebody committed on Tuesday. It is already in your Git history and in every clone made since, so the fix is to rotate the value at its source. Encrypting the file now and moving on only hides the evidence.

Prove the Blast Radius Is Small

You have not scoped anything until you have watched it refuse. Add a step that points the production role at a file it must not be able to open, and read what comes back.

# who is this job actually running as?
aws sts get-caller-identity --query Arn --output text
 
# the production role must NOT be able to open the staging file
sops decrypt secrets/staging.enc.env > /dev/null

arn:aws:sts::111122223333:assumed-role/ci-sops-decrypt/gha-deploy-9182736455
 
Failed to get the data key required to decrypt the SOPS file.
 
Group 0: FAILED
  arn:aws:kms:us-east-1:111122223333:key/1a2b3c4d-5e6f-7890-ab12-cd34ef567890: FAILED
    - | Error decrypting key: operation error KMS: Decrypt, https response error
      | StatusCode: 400, RequestID: 8b0f7c31-2d44-4a19-9a0e-6f21c5be7d10, api error
      | AccessDeniedException: User: arn:aws:sts::111122223333:assumed-role/
      | ci-sops-decrypt/gha-deploy-9182736455 is not authorized to perform:
      | kms:Decrypt on this resource because no identity-based policy allows the
      | kms:Decrypt action
 
Recovery failed because no master key was able to decrypt the file. In
order for SOPS to recover the file, at least one key has to be successful,
but none were.

Read that failure closely, because it answers two questions at once. The principal named in the message is an assumed-role session, and that session exists only because sts:AssumeRoleWithWebIdentity already succeeded, so the OIDC token and the trust policy agree with each other. The denial came one step later, from the permission policy. Either the key ARN (Amazon Resource Name, the unique address of an AWS resource) is wrong or the encryption context condition did not match, so the statement never applied and the request fell through to an implicit deny. Had the credentials step itself failed with "Not authorized to perform sts:AssumeRoleWithWebIdentity", your subject claim and your trust policy would be the ones disagreeing, which is a completely different fix. People lose afternoons to conflating those two.

# run this from your own admin session; the CI role has no cloudtrail rights
aws cloudtrail lookup-events \
  --lookup-attributes AttributeKey=EventName,AttributeValue=Decrypt \
  --max-results 4 --query 'Events[].[EventTime,Username]' --output text

2026-07-22T09:41:07+00:00	gha-deploy-9182736455
2026-07-22T08:12:55+00:00	gha-deploy-9182300118
2026-07-21T23:07:02+00:00	jsmith-cli
2026-07-21T14:33:19+00:00	gha-deploy-9179442210

Three of those are pipeline runs, each traceable back to a workflow run ID. The one at 23:07 is a person, from a laptop, at night. Maybe that was a legitimate incident fix. The point is that it is now a conversation you can have, with a timestamp attached, instead of an event nobody could have noticed. That visibility is exactly what you give up when you store an age key in CI, because age decryption never phones home.

Environment Variables Are Not a Vault

exec-env beats a plaintext file on disk. Be precise about the size of that win. Environment variables are readable by any process running as the same user, through /proc/<pid>/environ. Every child process inherits them. A crash dump, an error-reporting agent, or docker inspect on a container you started with --env will show them without hesitating. On an ephemeral, single-tenant GitHub-hosted runner that is destroyed when the job ends, that exposure is genuinely small. On a shared self-hosted runner where three teams' jobs run as the same user on the same box, it is not small at all, and exec-file through a FIFO is the better default there.

The cheapest way to shrink this further is to decrypt somewhere other than CI. Commit the ciphertext and let the cluster's controller decrypt at apply time, or let ESO (External Secrets Operator, a Kubernetes add-on that pulls values from a secret store at runtime) do the fetching, and the pipeline never holds decrypt rights in the first place. Save CI decryption for the jobs where the pipeline genuinely is the consumer. A Terraform apply that needs a database password is the honest case: give that one job one key, one encryption context, and fifteen minutes.

Try this

Run head -2 secrets/prod.enc.env on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a stored age key in CI is a key you have already handed out. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related