>_SecOpsLog_
CoursesSoftware supply chain securityAnatomy of real supply-chain attacks

Anatomy of real supply-chain attacks

SolarWinds, dependency confusion, Codecov, xz.

Advanced14 min · lesson 2 of 18

Every one of these four break-ins used a door the security team believed was locked. None of them picked the lock on the front door, your source code. They came in through the loading dock, the mail room, or a trusted supplier's van. Study the shape of each one and you learn the same lesson four times: the code sitting in your repository can be spotless while the artifact you actually ship is poisoned.

Code review looks at the source. That is one stage in a long chain that runs source, then build, then package, then registry, then the machine that installs it. An attacker who cannot sneak a malicious change past your reviewers still has the other stages to work with. SolarWinds hit the build. Codecov hit a build tool everyone trusted. Dependency confusion hit the resolver that chooses which package to fetch. The xz backdoor hit a maintainer, and through him, a release download. Not one of them touched the reviewed source.

Where each attack lived, and where review looks
The source (what review sees)
Pull request
read by humans
Git history
looks clean, and was clean
The build (SolarWinds, Codecov)
Compile step
backdoor injected at build time
Uploader script
swapped to steal secrets
The dependencies (confusion, xz)
Resolver
picks a public impostor by version
Release tarball
backdoor absent from git
Code review inspects the first zone. Every attack on this page lived in one of the other two.

Compromise the build: SolarWinds

Think of a print shop for a trusted newspaper. Every reporter's copy is checked and clean. Someone bribes the pressman, and while the plates are being cut he slides an extra paragraph onto page three. The signed, sealed, home-delivered paper now carries words no editor approved. That is SolarWinds. In 2020, attackers got onto SolarWinds' build servers and, during compilation, injected a backdoor into a component of the Orion product, a DLL (dynamic-link library, a piece of compiled code a program loads while it runs). The source in version control was clean. The build turned it into something else. The normal pipeline then signed that artifact with SolarWinds' own key and pushed it as a routine update to roughly eighteen thousand customers.

Here is the part that trips people up. That update was validly signed. The signature was genuine. Signing proves who built an artifact, not that the artifact matches the reviewed source. To close that gap you need provenance, a signed record that ties a finished artifact back to the exact commit and build steps that produced it. SLSA (Supply-chain Levels for Software Artifacts, pronounced 'salsa') is the framework that standardizes this, and a tool like slsa-verifier checks it.

terminal
# A valid signature says WHO built it. Provenance says WHAT was built, and from which source.
slsa-verifier verify-artifact orion-core.dll \
  --provenance-path orion-core.intoto.jsonl \
  --source-uri github.com/acme/orion \
  --source-tag v2020.2.1
output
Verified signature against tlog entry index 74619283
Verified build using builder "https://github.com/slsa-framework/slsa-github-generator/.github/workflows/generator_generic_slsa3.yml@refs/tags/v1.9.0" at commit 5d41402abc4b2a76b9719d911017c592e2e30f5f
Verifying artifact orion-core.dll: PASSED
 
PASSED: Verified SLSA provenance

With provenance in place, an artifact whose build did not start from the reviewed commit fails this check, however clean its signature looks. The signature and the provenance answer two different questions, and SolarWinds is what happens when you only ask the first one.

Poison the tool everyone trusts: Codecov

Codecov in 2021 is the poisoned water cooler. Thousands of CI (continuous integration, the automated system that builds and tests your code on every change) pipelines did not depend on Codecov's code so much as drink from it. On every run they fetched a script off Codecov's servers and ran it on the spot. The idiom was everywhere:

terminal
# The one-liner that lived in thousands of CI pipelines:
curl -s https://codecov.io/bash | bash

Read that the way an attacker does. It pulls a script over the network and runs it immediately, with no check in between, inside a job that holds your build secrets. Attackers got into Codecov's storage, edited that script, and added a line that copied every environment variable (your cloud keys, your registry tokens, your signing secrets) to a server they controlled. Every pipeline that ran the one-liner shipped its secrets on the next build. The fix is not exotic. Download the script, check its fingerprint against a value published through a separate channel, and only then run it.

terminal
# Download to a file. Verify against a signed checksum from an independent channel. THEN run.
curl -fsSL https://codecov.io/bash -o codecov.sh
sha256sum -c codecov.SHA256SUM
output
codecov.sh: FAILED
sha256sum: WARNING: 1 computed checksum did NOT match

When the fingerprints disagree, sha256sum exits non-zero and the pipeline stops before it runs a single tampered line. SHA-256 is a hashing function, it turns any file into a short fixed fingerprint, and changing one byte of the script changes the fingerprint beyond recognition. This catches the tamper only because the checksum came through a channel the attacker had not also edited, in Codecov's case a list signed with GPG (GNU Privacy Guard, a tool for signing and verifying files). Stronger still, pin the known-good hash in your own repository, where the attacker has no reach at all.

Piping the network into a shell
curl ... | bash runs whatever the server sends, the instant it arrives, with your pipeline's full permissions. There is no window to inspect it and no pinned version to compare against, so if the source is ever compromised, so is every build that trusts it. Download to a file, verify a pinned hash from an independent channel, then execute. Every time.

Beat the resolver: dependency confusion

Now the mail room. Your company buys from an internal supplier called acme-auth that exists only in your private catalog. One morning a stranger registers a business with the same name in the public directory and lists a much newer product number. Your purchasing clerk, under standing orders to always take the newest version, orders from the stranger. That is dependency confusion, published by researcher Alex Birsan in 2021. Many resolvers, the part of a package manager that decides which copy to fetch, will happily pick a public package over your private one when the public version number is higher.

Python's installer, pip, has a sharp edge here. Point it at your internal index and add the public one behind it, and pip does not treat the internal index as preferred. It searches both and takes the highest version it finds anywhere.

terminal
# --extra-index-url does NOT mean "fall back to". It means "also search here, take the highest version".
pip install acme-auth \
  --index-url https://pypi.internal.acme.corp/simple/ \
  --extra-index-url https://pypi.org/simple/
output
Looking in indexes: https://pypi.internal.acme.corp/simple/, https://pypi.org/simple/
Collecting acme-auth
  Downloading https://files.pythonhosted.org/packages/9c/1e/acme_auth-99.0.0-py3-none-any.whl (12 kB)
Installing collected packages: acme-auth
Successfully installed acme-auth-99.0.0

The tell is right there in the output. A package that should exist only inside your walls was downloaded from files.pythonhosted.org, public PyPI (the Python Package Index), at an absurd 99.0.0 that nobody on your team ever cut. The fix is to give the resolver exactly one place to look, and to make internal names impossible to squat.

/etc/pip.conf
[global]
index-url = https://pypi.internal.acme.corp/simple/
# No extra-index-url. Internal names never fall back to the public index.

In JavaScript, npm (Node's package manager) handles the naming half with scopes. A scoped name like @acme/auth belongs to your organization, and you bind that scope to your registry (the server that stores and serves your packages), so a public acme-auth can never stand in for it.

.npmrc
@acme:registry=https://npm.internal.acme.corp/
//npm.internal.acme.corp/:_authToken=${NPM_TOKEN}

Play the long game: xz

The last one is the long con, the quiet stranger who volunteers at the town archive for two years, becomes the person everyone trusts with the keys, and only then rewrites a single record. In 2022 an account using the name Jia Tan started contributing to xz-utils, the compression project behind the xz command and its library liblzma (the code that does the actual squeezing and unsqueezing of data). The work was real and useful. Over two years the account earned co-maintainer status while other accounts leaned on the burned-out original maintainer to hand over control. In early 2024, with that trust in hand, the attacker shipped a backdoor in xz versions 5.6.0 and 5.6.1, tracked as CVE-2024-3094 (Common Vulnerabilities and Exposures, the public catalog of known security bugs).

The clever part rhymes with SolarWinds. The backdoor was not in the git source that anyone could read. It rode in on the release tarball (the packaged download people actually build from), planted by a rigged build step and disguised inside files that looked like test data for the compressor. Built into liblzma, the payload hooked into sshd (the OpenSSH server, the program that answers remote logins) on Debian and Ubuntu machines, where sshd loads liblzma indirectly through a system library called libsystemd. Given the right secret key, it would have let the attacker run commands on exposed servers across the internet.

It was caught by luck and a sharp nose. Andres Freund, a Postgres developer, noticed his ssh logins were taking about half a second longer than they should and that a benchmark was throwing strange errors. He pulled the thread and found the backdoor before the poisoned versions reached most stable distributions. As a defender you can check two things in seconds: which version you have, and whether your sshd even loads the library.

terminal
xz --version
ldd /usr/sbin/sshd | grep liblzma
output
xz (XZ Utils) 5.4.5
liblzma 5.4.5
	liblzma.so.5 => /lib/x86_64-linux-gnu/liblzma.so.5 (0x00007f8b2c4a1000)

Version 5.4.5 predates the poisoned releases and is safe. If that first line read 5.6.0 or 5.6.1, you would be exposed. The second line shows sshd is linked to liblzma, which is normal on these systems and not a problem by itself. That is the whole point. Nothing here looks wrong at a glance. The danger was one specific version of a library that a critical service pulls in without anyone choosing it on purpose.

Quick check
01SolarWinds shipped an Orion DLL that was validly signed with the company's own key, but the backdoor was woven in while the code was being compiled. Which control would have flagged that artifact before you installed it?
Incorrect — Nobody forged anything here. The signature on that update was genuine, produced by the vendor's real key in the vendor's real pipeline, so hardening the key protects a door that was never touched.
Incorrect — Reviewers read source. The malicious code appeared later, while the compiler ran, at a stage no reviewer ever sees, so you could double the reviewers and still ship the same DLL.
Correct — That check ties the finished file back to the commit and the builder that produced it, so a build that started from tampered inputs fails even though its signature looks perfect.
Incorrect — Pinning locks you to one file and stops surprise upgrades, but the file you pinned is the poisoned one, so you would install the backdoor deliberately and call it reproducible.
02Your pipeline still runs 'curl -s https://codecov.io/bash | bash' on every build. Four teammates each propose a change. Which one actually removes the risk that drained secrets out of Codecov's users?
Incorrect — Caching answers a speed question. If the copy you cached happens to be the edited one, you have made the compromise permanent, because nothing in that flow ever compares the script against a known-good value.
Correct — This is the shape of the fix. The file lands on disk, its fingerprint gets compared with one published elsewhere, and a mismatch exits non-zero so the job stops before the first tampered line runs.
Incorrect — Neither the interpreter nor the quiet flag changes where the code came from. sh reads the same bytes bash would, and the job still hands its whole environment to whatever the server sent.
Incorrect — Retries and timeouts fix flaky networks. A tampered script downloads perfectly and runs perfectly, leaving the build green while your registry tokens and cloud keys travel somewhere you did not choose.
03On a Debian host, 'xz --version' reports 5.4.5 and 'ldd /usr/sbin/sshd | grep liblzma' prints a line for liblzma.so.5. What do those two facts tell you about CVE-2024-3094?
Correct — Between them the two commands settle it. You never received a poisoned build, and on Debian and Ubuntu sshd reaches liblzma indirectly through libsystemd on every machine, compromised or not.
Incorrect — If the link alone were the finding, every Debian and Ubuntu server in the world would be compromised. What mattered was which build of the library loaded, not that a library loaded at all.
Incorrect — The hiding place was real, but you do not have to repeat the discovery to answer your own question. The version number tells you whether one of the tainted builds ever reached this host.
Incorrect — The account's earlier contributions were genuine work that people still run safely. Dropping below a known-good release costs you real fixes and buys nothing, since the payload landed in two builds only.

What to check on your own pipeline

Turn the four break-ins into four questions about your own setup. Can you rebuild a shipped artifact from its source commit and get the same bytes, or do you only hold a signature that proves who ran the build? Does any job pipe a remote script into a shell without a pinned, verified hash? Does any resolver in your builds have a public fallback for names that should exist only inside your walls? Do you know which of your critical services load libraries you have never audited, the way sshd loads liblzma? Each question points at one of these attacks, and each has an answer you can wire into the pipeline instead of writing into a policy document.

Try this

Run curl -s https://codecov.io/bash | bash on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: piping the network into a shell. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

The software supply chain & why…DevSecOps: shift left, assume…

Related

What CI is, and where Jenkins fitsJenkins foundations, done rightThe .gitlab-ci.yml modelSecure CI/CD with GitLabSupply-chain threat modelSoftware supply chain in depthCI/CD interview questionsInterview prep