Trusted, isolated builders

A build you can actually trust the output of.

A build runs your code, but it also runs everyone else's. It checks out your source, unpacks hundreds of dependencies you never read, and usually holds a key to push the result somewhere that matters: a container registry (the warehouse where finished build images are stored), a cloud account, production itself. A build is a kitchen. Your source and dependencies are the ingredients, the artifact (the finished file the build produces, a binary or a container image) is the finished dish, and the credentials are the keys to the pantry and the delivery van. Who gets into that kitchen, and what they can touch once inside, decides whether you can trust the dish that comes out. So "is my build trusted?" is really three questions about the kitchen.

What a trusted builder actually means

A trusted builder has three properties. The first is isolated: one order at a time, on its own counter, so no two builds share a surface. In Linux terms that means separate namespaces. A namespace is a feature built into the kernel (the core of the operating system that manages the hardware and every running program) that gives a process its own private view of things like process IDs (the numbers the system assigns to running programs), the filesystem, and the network. It also means no shared writable host paths and no privileged mode. One build cannot see or change another.

The second is ephemeral: the kitchen is stripped and rebuilt clean after every order. A fresh environment per build, thrown away when the build ends. A file that a poisoned dependency drops during one job cannot wait around for the next job to trip over it.

The third is least-privileged: the chef gets only the keys for tonight's service, and those keys stop working at closing time. In practice that means short-lived tokens minted with OIDC (OpenID Connect, a standard way for one machine to prove who it is to another and get a temporary token back) instead of long-lived secrets sitting in the environment.

Look at the runner you actually have

Before you trust any output, look at where it was built. A runner is the machine or container that actually executes your build jobs. From inside one, three commands tell you most of what you need.

# On a suspect runner, ask three questions from inside a job.
id                               # what user am I running as?
ls -l /var/run/docker.sock       # can I reach the host's Docker daemon?
printenv AWS_SECRET_ACCESS_KEY   # do I hold a standing, long-lived credential?

uid=0(root) gid=0(root) groups=0(root)
srw-rw---- 1 root docker 0 Jul 17 09:14 /var/run/docker.sock
wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

Every answer here is a red flag. The job runs as root (user id 0), the account that can do anything on the machine. The host's Docker daemon (the background service that builds and runs every container on that box) is reachable through its socket, mounted straight into the job, which is a door onto the machine every other job runs on. And a standing AWS (Amazon Web Services) secret key sits in the environment, valid until someone rotates it by hand. Now walk the attack. One malicious dependency in this single job runs as root, opens that Docker socket, launches a container that mounts the whole host filesystem, and reads the secrets of every other build on the box. The defender who runs these three commands sees all of that before an attacker does.

The fix is to stop handing jobs that power. On a self-hosted GitLab runner, the executor config is where you take it back.

concurrent = 4
check_interval = 3
 
[[runners]]
  name = "build-fleet-01"
  url = "https://gitlab.example.com/"
  executor = "docker"
  [runners.docker]
    image = "alpine:3.19"
    privileged = false            # true here = a root container that can break out to the host
    # volumes = ["/var/run/docker.sock:/var/run/docker.sock"]  # NEVER: hands the job the host's Docker daemon
    volumes = ["/cache"]
    pull_policy = ["always"]      # fetch a clean image each job; no tampered cached layer survives

The Docker executor already gives you a fresh container per job, so the job is ephemeral by construction. Your work is to not undo that: no privileged mode, no host Docker socket, and a fresh image pulled every time so a tampered, cached layer cannot linger. A self-hosted runner is itself a systemd service (systemd is the manager that starts and supervises background programs on most Linux systems). Run it under its own unprivileged account. Let control groups (the kernel feature that caps how much CPU and memory a job can use) keep one job from starving the rest. Then check that the confinement actually took.

# On a properly confined job, check the same ground.
id -un
readlink /proc/self/ns/pid /proc/self/ns/net
ps -e --no-headers | wc -l

build
pid:[4026532467]
net:[4026532468]
6

This job runs as build, not root. It has its own process-ID and network namespaces, and it can see only a handful of processes (its own job tree), not the hundreds running on the host. That is isolation you can measure instead of assume.

The build cannot sign its own permission slip

Here is the part that trips up teams who have done everything above and still get burned. Provenance (a signed record of how an artifact was built: which source, which commit, which builder) is only as honest as whoever writes it. It is the health inspector's certificate on the kitchen wall. If the chef prints and signs that certificate, a filthy kitchen can write "spotless" and hang it up. The certificate has to come from an inspector the chef cannot bribe.

So if the same build steps that might be compromised also generate and sign the provenance, a malicious build lies about what it did, and the signature makes the lie look official. A hardened builder generates the provenance itself, in its control plane (the part of the platform that runs the build but sits outside the build steps), out of reach of the build steps. The build can ask for a signature over the true facts, but it never holds the signing key and cannot change the facts. SLSA (Supply-chain Levels for Software Artifacts, said out loud as "salsa") draws exactly this line between Build Level 2, where a hosted platform produces signed provenance, and Build Level 3, where the build steps provably cannot influence or forge it.

You can see the least-privilege half of this in the token a good builder hands your job. Decode its payload, and never print the token itself.

# base64url -> base64, then read the claims that matter.
printenv OIDC_TOKEN | cut -d. -f2 | tr '_-' '/+' | base64 -d 2>/dev/null \
  | jq '{sub, aud, iss, iat, exp}'

{
  "sub": "repo:acme/widget:ref:refs/heads/main",
  "aud": "sts.amazonaws.com",
  "iss": "https://token.actions.githubusercontent.com",
  "iat": 1721203800,
  "exp": 1721204400
}

That token lives for ten minutes (exp minus iat is 600 seconds). It is scoped to one repository on one branch (the sub, or subject, claim), and only the AWS token service will accept it (the aud, or audience, claim). It was minted by the platform (the iss, or issuer, claim), not by your build steps, and the job trades it for temporary cloud credentials. Leak it and it is stale before an attacker can drive across town, and useless anywhere but that one audience.

The signing half shows up when you verify the finished artifact. slsa-verifier checks the provenance against what you expected.

slsa-verifier verify-artifact widget-linux-amd64 \
  --provenance-path widget.intoto.jsonl \
  --source-uri github.com/acme/widget \
  --source-tag v1.4.0

Verified signature against tlog entry index 78412356 at URL: https://rekor.sigstore.dev/api/v1/log/entries/24296fb2...
Verified build using builder "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v2.0.0" at commit 3f2b7c9d1e5a...
Verifying artifact widget-linux-amd64: PASSED
 
PASSED: SLSA verification passed

The line that matters is the builder id. That is the kitchen that signed the certificate. slsa-verifier confirms the provenance was signed by that Level 3 generator's identity, and that the signature is recorded in Rekor, Sigstore's public, append-only ledger of signatures that anyone can read but no one can quietly edit. It also confirms the source and tag match what you asked to verify. Because the generator signs inside its own trust domain, your build steps never touched the key, so they could not have forged this record. That is why a trusted, hosted builder counts as a security control, not a nicety.

Where you can, prefer your platform's hosted build service over a runner you maintain, because the platform keeps the isolation and the non-forgeable provenance running for you. On GitHub that means the SLSA generators (reusable workflows like slsa-github-generator that build the provenance in a separate, trusted job your steps cannot reach). On a hosted Linux builder you get a fresh virtual machine per run without babysitting the fleet. Self-hosting is a decision to take on that work yourself, not a default to drift into.

Run the three questions on your own runner today. If any answer comes back "root", "the socket is mounted", or "a standing key", fix that before you touch signatures or SLSA levels. Provenance generated inside an L0 kitchen is a signed lie waiting to be verified.

Try this

Run id # what user am I running as? on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a shared privileged runner is an L0 builder. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related