CoursesSoftware supply chain securityThe software supply chain & why it is targeted

The software supply chain & why it is targeted

Every link from commit to cluster is attack surface.

Advanced12 min · lesson 1 of 18

You sit down at a restaurant, the plate arrives, and you eat it. You never see the farm the vegetables came from, the truck that carried them, the walk-in fridge they sat in, or the hands of the line cook. You trust the plate because you trust the whole kitchen behind it, mostly without thinking about it. If someone tampers with any step back there, the food still arrives looking perfect, on a clean plate, with your waiter smiling. Software works the same way. Production is the plate. Your software supply chain is everything behind the kitchen door.

Your software supply chain is every input and every step between a developer's idea and the artifact running in your cluster (the group of machines running your production containers). That is the source code, the dependencies you pull in, the build system, the container base image you start from, the continuous integration pipeline (the automation that builds, tests, and packages your code, usually written CI), the registry that stores the finished image, and the deploy that pushes it live. Traditional application security guards the running program: the plate at the table. Supply-chain security guards how that program was made. The difference matters because an attacker who compromises any link ships their code inside your trusted artifact. It sails past your perimeter, wearing your name, signed by your pipeline, because as far as your systems can tell, it is yours.

One artifact's path from commit to cluster, and the door at each hop
1Commit
Author field is unauthenticated text unless signed
2Dependencies
Transitive packages run install-time code
3Build / CI
Runner holds registry and cloud credentials
4Base image
Inherit an operating system you did not audit
5Registry
Stores the image everyone downstream will trust
6Deploy gate
Last place to verify before the cluster
7Cluster
Runs whatever cleared every prior door

Every Link Is a Door

A house is only as safe as its most-forgotten window. It does not matter how solid the front door is if the bathroom window upstairs never locks. The supply chain is a row of doors between a commit and your cluster, and an attacker needs only one of them open. So the first useful habit is to stop treating your application as a single thing to defend and start counting the doors. Begin with the code you did not write.

terminal
# direct dependencies we chose on purpose
jq '.dependencies | length' package.json
# every package actually installed, direct plus transitive
npm ls --all --parseable 2>/dev/null | wc -l
output
14
1412

Fourteen packages you chose. One thousand four hundred and twelve packages you actually run. The gap is transitive dependencies: the libraries your libraries pull in, and the libraries those pull in, going down for layers you will never read. On a typical service, most of the code that ends up running, often 80 to 90 percent of it, is third-party code you did not write and rarely open. Most package managers run install-time scripts, so a single poisoned package deep in that tree can execute code on your build machine the moment you type install, before one test runs. This is how real attacks land. They rarely break your front door. They get one popular package five levels down to ship a bad version, and everyone who installs it runs the payload. As a defender, that number is your homework. You cannot read 1,412 packages. You can pin them, generate an itemized list of them, and refuse to build when the list changes without a human noticing.

You Inherit Every Hand That Touched the Base Image

Almost nobody builds a container image from nothing. You start FROM a base image, the way a baker starts from a pre-made crust instead of milling flour. That saves an enormous amount of time, and it means you inherit the base's kitchen hygiene, good or bad. Every package, every default, every patch level baked into that base is now baked into you. Look at what you are actually standing on.

terminal
docker history python:3.12-slim
output
IMAGE CREATED CREATED BY SIZE COMMENT
b1e9cef3f2a4 9 days ago CMD ["python3"] 0B buildkit.dockerfile.v0
<missing> 9 days ago RUN /bin/sh -c set -eux; pip install --upgr… 12.1MB buildkit.dockerfile.v0
<missing> 9 days ago ENV PYTHON_SHA256=a1c1f9… 0B buildkit.dockerfile.v0
<missing> 9 days ago ENV PYTHON_VERSION=3.12.4 0B buildkit.dockerfile.v0
<missing> 9 days ago RUN /bin/sh -c set -eux; apt-get update; apt… 42.6MB buildkit.dockerfile.v0
<missing> 3 weeks ago /bin/sh -c #(nop) CMD ["bash"] 0B
<missing> 3 weeks ago /bin/sh -c #(nop) ADD file:5d1b…c in / 74.8MB

Read that from the bottom up. The 75-megabyte layer at the bottom is the Debian base filesystem: an entire operating system you did not assemble and will not audit line by line. On top of it sit the Python install steps. You wrote none of this, yet all of it runs as part of your service, with your service's access. If the people who publish that base image have a bad day, or a stolen account, their bad day becomes your production. That is why teams pin base images to a content digest (the cryptographic fingerprint of the exact bytes, written like sha256:...) instead of a moving tag like slim, and re-verify that digest on every build. A tag can be repointed at new bytes under you overnight. A digest cannot.

The Build Box Holds the Master Key

In a kitchen, the walk-in fridge, the register, and the back door all open with one master key, and that key tends to hang on a hook by the pass where everyone reaches past it. Your build system is that key. To do its work, the machine that runs your pipeline has to pull private code, push finished images to your registry, and often deploy straight to your cloud. So it holds the credentials for all three. Look at a self-hosted runner (a build machine you own that executes your pipeline jobs) set up as a systemd service (systemd is the program that starts and supervises background services on modern Linux).

terminal
systemctl cat actions-runner.service
output
# /etc/systemd/system/actions-runner.service
[Unit]
Description=GitHub Actions Runner (self-hosted)
After=network-online.target
Wants=network-online.target
[Service]
Type=simple
User=runner
WorkingDirectory=/opt/actions-runner
EnvironmentFile=/opt/actions-runner/.env
ExecStart=/opt/actions-runner/run.sh
Restart=always
RestartSec=5
[Install]
WantedBy=multi-user.target

Two lines decide the blast radius. EnvironmentFile=/opt/actions-runner/.env loads a set of secrets into every job this machine runs. ExecStart=/opt/actions-runner/run.sh executes whatever the pipeline hands it, including build steps proposed in a pull request (a request to merge someone's code, written PR) from outside your team. Now read the secrets that every one of those jobs can see.

terminal
sudo cat /opt/actions-runner/.env
output
REGISTRY_TOKEN=ghp_1a2B3c4D5e6F7g8H9i0JkLmNoPqRsTuVwXyZ
AWS_ACCESS_KEY_ID=AKIAIOSFODNN7EXAMPLE
AWS_SECRET_ACCESS_KEY=wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY

A registry push token and cloud (Amazon Web Services, written AWS) keys, sitting in a file that any build step on this box can read with a single cat. If an attacker gets one line of their code to run here, through a malicious pull request or a poisoned dependency's install script from two sections ago, they do not need to break your registry or your cloud. They already hold the keys. They read this file, push a backdoored image under your name, and walk out the back door you left propped open. From the outside it looks like a normal build.

Self-hosted runners plus public pull requests equal remote code execution
A self-hosted runner that builds pull requests from a public repository will happily run code a stranger wrote, on a machine holding your production credentials. Anyone can open a pull request. If your pipeline runs their build steps before a human approves, that is arbitrary code execution with your secrets, by design, not by bug. Require manual approval before any workflow runs on an untrusted pull request, hand runners short-lived scoped tokens instead of long-lived cloud keys, and keep the real credentials behind a throwaway builder, never sitting on it.

From 'Is It Vulnerable' to 'Can I Prove It'

A health inspector walking a kitchen checks for visible problems: mold on the wall, a fridge running warm, expired stock on the shelf. That is a vulnerability scan. It asks whether a known-bad thing is present, and you should keep doing it. But it cannot tell you whether the sealed box that arrived this morning actually came from the supplier on the label, or whether someone opened it in transit and swapped the contents. For that you want a tamper-evident seal and a signature you can check against a name you trust. Supply-chain security is that second question. Not 'does this artifact contain a known flaw' but 'can I prove this artifact is what it claims to be, built from the source I reviewed, by the pipeline I control, from the dependencies I approved, untouched the whole way.'

Answering that turns trust from a feeling into something you can check with a command. Three pieces do the work. A software bill of materials (an itemized ingredient list of everything inside an artifact, written SBOM) tells you what is in the box. Provenance (a signed record of who built the artifact, from which commit, on which machine) tells you where it came from. A signature ties that record to a key you trust, so a stranger cannot forge it. Then you stand a gate at deploy that refuses anything failing the check. Here is that gate saying no.

terminal
cosign verify \
--certificate-identity-regexp '.*' \
--certificate-oidc-issuer-regexp '.*' \
registry.example.com/team/app:latest
output
Error: no signatures found
main.go:74: error during command execution: no signatures found

The image exists, and it very likely passes a vulnerability scan. It still fails here, because it carries no proof of where it came from. That is the whole shift in one command. A scanner asks whether the food looks moldy. Verification asks the box to prove it is sealed, and throws it out when it cannot. The rest of this course builds that chain one link at a time: signing commits so the author field cannot be faked, generating provenance inside the pipeline, producing and storing an SBOM for every build, signing images, and wiring the gate that checks all of it before anything reaches your cluster.

Quick check
01You run cosign verify against registry.example.com/team/app:latest and get 'Error: no signatures found', yet the same image scanned clean an hour ago with zero findings. What do those two results together tell you?
Incorrect — cosign checks a signature, not a package database. Refreshing scanner feeds changes which flaws you see and leaves the missing signature exactly where it was.
Incorrect — Passing '.*' to both the identity and issuer flags accepts any signer at all, which is the loosest check you can ask for. Even that found nothing to examine.
Correct — A clean scan tells you nobody has filed a known flaw against what is inside the box. It says nothing about who packed the box or whether the contents were swapped on the way, so you want both answers before you ship.
Incorrect — Treating a clean scan as permission to deploy is the exact habit this course exists to break. Someone who ships their code inside your artifact does not need a known flaw to hurt you.
02jq '.dependencies | length' package.json prints 14, and npm ls --all --parseable | wc -l prints 1412. A teammate shrugs that the gap only affects image size. Where does that reasoning break?
Incorrect — Depth in the tree changes nothing about whether code executes. Something fourteen levels down installs and runs with the same access as a package you typed yourself.
Incorrect — The risky moment is install, not startup. Your package manager hands control to setup hooks while the tree unpacks, on your build machine, long before anything serves a request.
Incorrect — Deduplication trims copies, but that listing still reflects real code landing on disk and being set up. Even if some entries repeat, you are not going to read what is left.
Correct — That is why the number is your homework rather than trivia. You cannot review 1412 packages, so you pin versions, keep an itemized list of them, and refuse to build when the list moves without a human noticing.
03systemctl cat actions-runner.service shows EnvironmentFile=/opt/actions-runner/.env, and that file holds a registry push token plus AWS keys. The same runner builds pull requests from your public repository. What is the exposure and the fix?
Incorrect — EnvironmentFile loads those values into the environment of every job this unit starts, so a build step inherits them without opening any file at all. The runner account is the account holding the keys.
Correct — Anyone can open a pull request against a public repository, and this box runs the steps that request proposes. Approval decides who gets to execute at all, and tokens with a short life shrink what an attacker keeps if one does.
Incorrect — A build step sees every value the unit loaded, so the token and the cloud keys leave together in one go. Weekly rotation only limits how long a stolen token stays useful afterwards.
Incorrect — Restart=always brings the service back after a crash and nothing more. It scrubs nothing, and it reloads the same file with the same keys the moment it comes up.

One idea sits under all of it. Trust is inherited. The moment you import a library or start FROM a base image, you take on the trust posture of everyone upstream who touched it: the maintainers, their accounts, the machines their code was built on. A single stolen maintainer login or leaked CI token, three hops away from you, becomes your incident, in your production, wearing your artifact's name. You do not fix that by trusting harder. You fix it by proving instead of assuming, at every link. Before you reach for a new tool, do the unglamorous first step: open your pipeline config and write down every hop from commit to cluster. That list is your attack surface. Every line on it needs an answer to one question. How would I know if this had been tampered with?

Try this

Run jq '.dependencies | length' package.json on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: self-hosted runners plus public pull requests equal remote code execution. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related