Hardening the pipeline

Pin actions by SHA, OIDC, ephemeral runners.

Expert35 min · lesson 15 of 15

On 14 March 2025 an attacker who had stolen a GitHub personal access token pushed one malicious commit to the popular tj-actions/changed-files action and then re-pointed almost every version tag, from v1 through the then-current v45, at it. Any workflow that referenced the action by tag suddenly executed code that reached out to a GitHub gist for a short Python routine, dumped the runner's Runner.Worker process memory, and printed the harvested CI secrets, double-base64-encoded, straight into the build log. The action was referenced by more than 23,000 repositories, and every one running a public workflow risked exposing those secrets to anyone who could read the log. The teams that walked away unscathed had done two unglamorous things: they pinned the action to an immutable commit SHA, and they ran a runtime egress monitor that noticed the unexpected outbound connection. This lesson hardens the pipeline itself, the highest-value target in the entire supply chain, because whoever controls CI inherits its tokens, its signing identity, and its reach into every downstream consumer.

Pin every action to a commit SHA

A Git tag is a mutable pointer. A lightweight tag is a ref that names a commit directly; an annotated tag names a small tag object that in turn names the commit. Either way, the mapping from a tag to a specific tree of code is resolved at checkout time and can be rewritten at any moment by anyone with push access, which is exactly what the tj-actions attacker did. A commit SHA is different: it is the hash of the commit object, which itself commits to the root tree hash, the parent commits, the author, and the message. Change one byte of the action's source and the tree hash changes, so the commit hash changes; the reference is content-addressed and collision-resistant, and Git is migrating from SHA-1 to SHA-256 to keep it that way. Referencing the action by its 40-character SHA therefore nails the exact bytes that will run: the maintainer can publish v5, get compromised, or delete the repo and your pipeline keeps running the reviewed code. Always leave the human-readable tag in a trailing comment so upgrades stay legible, and review each bump as if it were application code, because it is. Automate this rather than doing it by hand: tools such as pinact or ratchet rewrite every tag reference to its current SHA and keep them current, and Dependabot understands SHA-pinned actions, opening pull requests that bump the SHA and its trailing comment together so you still review the diff. GitHub's newer immutable-actions and immutable-releases features raise the floor, but the commit SHA remains the guarantee that predates and outlives any single registry feature.

resolve a tag to its immutable SHA, then pin every workflow
# Dereference a tag to the exact commit it points to right now
gh api repos/step-security/harden-runner/commits/v2.13.1 --jq .sha
# Then pin (and keep pinned) every action reference across all workflows
pinact run
output
f4a75cfd619ee5ce8d5b864b0d183aff3c69b55a
[pinact] .github/workflows/release.yml
step-security/[email protected] -> @f4a75cfd619ee5ce8d5b864b0d183aff3c69b55a # v2.13.1
actions/[email protected] -> @11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
actions/[email protected] -> @3041bf56c941b39c61721a86cd11f3bb1338122a # v5.2.0
sigstore/[email protected] -> @dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3.7.0

Least-privilege GITHUB_TOKEN and OIDC

Every job receives an automatically-minted GITHUB_TOKEN. On many repositories its default scope is still read and write across contents, packages, and more, so a single compromised step can push commits, publish packages, or open a backdoor pull request. Declare permissions explicitly: set the top-level default to contents read, or an empty map for nothing, and elevate only the specific jobs that need more, to the narrowest scope. The signing job needs id-token write, and that permission lets the job request a short-lived OIDC JWT from GitHub's token endpoint, audience-scoped and carrying verifiable claims about the repository, ref, workflow, and runner environment. cosign exchanges that JWT with Fulcio, which mints a keyless signing certificate whose subject is bound to exactly those workflow-identity claims, and your cloud provider trusts the same JWT for federated login. There are no long-lived signing keys and no static cloud secrets to steal in the first place. Scope matters at the trigger level too: avoid pull_request_target combined with a checkout of untrusted PR code, the classic path by which a fork's build script runs with your write token and secrets. Least privilege plus OIDC shrinks both the blast radius of a compromised step and the standing credential inventory an attacker can harvest.

.github/workflows/release.yml
permissions:
contents: read # least-privilege default for GITHUB_TOKEN
jobs:
build-and-sign:
runs-on: ubuntu-24.04 # GitHub-hosted: a fresh, ephemeral VM per job
permissions:
contents: read
id-token: write # OIDC: keyless cosign + federated cloud auth
steps:
# harden-runner MUST be the first step so it monitors everything after it
- uses: step-security/harden-runner@f4a75cfd619ee5ce8d5b864b0d183aff3c69b55a # v2.13.1
with:
egress-policy: block # deny-by-default: only the endpoints below
disable-sudo: true
allowed-endpoints: >
github.com:443
api.github.com:443
objects.githubusercontent.com:443
proxy.golang.org:443
sum.golang.org:443
fulcio.sigstore.dev:443
rekor.sigstore.dev:443
tuf-repo-cdn.sigstore.dev:443
- uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
- uses: actions/setup-go@3041bf56c941b39c61721a86cd11f3bb1338122a # v5.2.0
- uses: sigstore/cosign-installer@dc72c7d5c4d10cd6bcb8cf6e3fd625a9e5e537da # v3.7.0
- name: Build
run: go build -trimpath -o app ./...
- name: Sign the artifact keylessly (uses the OIDC token, no stored keys)
run: cosign sign-blob --yes --bundle app.cosign.bundle app

harden-runner: runtime egress control

Pinning fixes what source runs; it cannot stop that source from doing something malicious at runtime, so add a network tripwire. StepSecurity's harden-runner installs a security agent as its first action: it points the runner's DNS resolver at a local proxy and loads eBPF programs into the kernel that observe every process, file, and outbound-network syscall for the life of the job. Because it hooks the kernel's real connect and DNS syscalls rather than trusting the application, it sees egress from any language, any subprocess, and any statically linked binary a compromised step spawns. In audit mode it records and reports each connection, and you start here for a week to learn your build's genuine endpoints; in block mode it permits DNS and TCP only to hosts named in allowed-endpoints and drops everything else, attributing each attempt to the exact process and step that made it. The insights dashboard diffs each run against a learned baseline and flags new destinations, which is how StepSecurity spotted the tj-actions anomaly in the wild within hours. When the compromised action reached out to fetch its second-stage memory-dumping script from an unexpected host, block mode simply refused to resolve and connect: the download failed, the memory dump never ran, and the run was annotated with the offending destination.

How harden-runner decides on an outbound connection
Outbound connection attempt on the runner
eBPF connect()/DNS hook intercepts it before it leaves
host in allowed-endpoints
Allowed
connection proceeds normally
not listed + egress-policy: block
Dropped
resolve/connect refused, GitHub annotation, logged to insights
not listed + egress-policy: audit
Allowed but flagged
reported so you can build the allowlist before enforcing
Roll out audit first to learn real endpoints, then switch to block so an unexpected egress fails closed instead of exfiltrating.
harden-runner blocking an unexpected egress (job log)
Run step-security/harden-runner@f4a75cfd619ee5ce8d5b864b0d183aff3c69b55a
[harden-runner] eBPF monitoring active | egress-policy: block
# ...the compromised changed-files step tries to fetch its second-stage payload...
Error: The runner attempted an outbound call that was blocked
step : Get changed files
process : node /home/runner/work/_actions/tj-actions/changed-files/dist/index.js
dns : gist.githubusercontent.com -> refused (not in allowed-endpoints)
dest : 185.199.108.133:443
policy : block
Blocked 1 endpoint. Full network log for this run:
https://app.stepsecurity.io/github/acme/webapp/actions/runs/12783440915

Ephemeral runners and closing the loop

Two final layers. Run on ephemeral runners: GitHub-hosted runners are already a fresh, single-use VM destroyed after each job, and self-hosted fleets should use actions-runner-controller with ephemeral set true so that no malware, cached credential, or poisoned build tool can survive into the next job, the isolation that SLSA Build L3 requires of a hardened build platform. Ephemerality also removes a subtler risk: a long-lived runner accumulates caches, checked-out source, and tool downloads that a later job can read or tamper with, quietly breaking the isolation SLSA assumes between independent builds. Then close the loop. Everything earlier in this course, the provenance, in-toto attestations, cosign signatures, and SBOMs, is only assurance if something actually verifies it. Protect the release branch with required review and status checks so no single account can push straight to a release, and put a verification gate in the path: slsa-verifier or cosign verify at release time, or a policy controller at admission, so unsigned or unattested artifacts cannot ship. A hardened, least-privilege, egress-controlled pipeline that both produces and enforces its own evidence is the end-to-end guarantee this whole course has been building toward.

A pinned SHA fixes the source, not what it fetches at runtime
Pinning an action by SHA guarantees the action's own bytes, but not what those bytes fetch while running. Composite actions reference other actions, often by mutable tag; Docker-container actions can start FROM a mutable base image such as alpine:latest; and steps that npm install or pip install pull unpinned dependencies live. So SHA-pinning bounds the reviewed source, not the runtime dependency graph, which is precisely why deny-by-default egress control belongs alongside it, not instead of it. Pin what you can, and constrain the network for everything you cannot.
Quick check
01You pin every third-party action to a full commit SHA. One of them is a Docker-container action whose Dockerfile begins FROM python:3-slim and runs pip install requests. What residual supply-chain risk remains?
Incorrect — The SHA pins only the action's own source tree, not the base image or packages the container pulls at runtime.
Correct — SHA-pinning bounds reviewed source, not the runtime dependency graph, so a network tripwire like harden-runner block covers what pinning cannot.
Incorrect — SHA pinning works for any action type; the reference itself is immutable regardless of how the action is implemented.
Incorrect — Broad read permissions do nothing to constrain a container's base image or its package downloads, and read-all is the opposite of least privilege.
02In the release workflow, only the signing job sets permissions: id-token: write. What does that permission enable, and what does it remove the need for?
Incorrect — that would be contents: write; id-token: write is unrelated to pushing commits.
Incorrect — the point is the opposite: keyless signing means there is no long-lived key to store or steal.
Incorrect — it adds one narrow OIDC capability and does not broaden the GITHUB_TOKEN's other permissions.
Correct — the OIDC token carries verifiable workflow-identity claims and is exchanged for ephemeral trust, eliminating standing signing keys and stored cloud credentials.
03You add StepSecurity's harden-runner but leave egress-policy: audit rather than block. During a run, a compromised step tries to reach an unlisted host to download a second-stage payload. What does harden-runner do?
Correct — audit mode observes and reports egress without enforcing, so it surfaces the anomaly but does not stop the exfiltration.
Incorrect — that is block-mode behavior; audit mode does not drop connections.
Incorrect — audit mode neither blocks the connection nor fails the job; it only records and flags.
Incorrect — audit mode does not block either DNS or TCP; blocking both is what block mode does.

Try this

Work through “Ephemeral runners and closing the loop” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: a pinned SHA fixes the source, not what it fetches at runtime. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related