>_SecOpsLog_
CoursesSoftware supply chain securitySupply-chain incident response

Supply-chain incident response

When a dependency or build is compromised.

Advanced12 min · lesson 17 of 18

A food company finds a contaminated batch of one ingredient sitting in its warehouse. It does not wipe down a single jar and call it done. It pulls every product made from that batch, off every shelf in every store, and traces the batch number back to the supplier who shipped it. That is a recall. A supply-chain incident response is a software recall. A dependency turns out to be backdoored, a build server gets breached, or a signing identity gets misused, and the first question is not which machine was hit. It is which artifacts were built with the poisoned part, and every place those artifacts were deployed. The damage travels outward along the supply chain, quietly, carried by trust.

That flips your working assumption. In an ordinary incident you isolate one compromised host and clean it. Here you treat everything the compromised link touched or produced as suspect until you can prove otherwise. The blast radius is a tree, not a dot. One poisoned library sits inside dozens of built images. Each image runs in many places. Each of those places holds credentials the attacker may now be able to reach. Underestimating that reach, telling yourself it was one library in one service, is how a compromise you think you contained keeps breathing.

Your build records are the recall paperwork

This is where the earlier parts of this course pay off, on the worst day. Three kinds of records turn a blind hunt into a handful of queries. A software bill of materials (SBOM) is a full ingredient list of every component that went into a build, so it tells you which artifacts contain the compromised part. Build provenance is like the stamp on a food jar that names the factory and the shift that made it: a signed record of how and where an artifact was built (this follows the SLSA standard, short for Supply-chain Levels for Software Artifacts), so it tells you which pipeline produced an artifact and from which source commit. And a transparency log is an append-only public ledger of every signature (called Rekor in the Sigstore project), so it tells you when something was signed and by which identity, letting you spot a key or identity being used when it should not have been. You already generate all three on every build. The evidence you produce routinely is the same evidence you need in an emergency.

Take a real one. The xz-utils backdoor, tracked as CVE-2024-3094 (CVE stands for Common Vulnerabilities and Exposures, the public identifier for a known flaw), planted malicious code in versions 5.6.0 and 5.6.1 of the liblzma compression library and aimed it at the SSH server (SSH, or Secure Shell, the service admins use to log into machines over the network). Your first question on hearing that is not whether you use xz. Everything uses xz. It is which of the images you ship contain the poisoned versions. Your SBOMs answer that directly.

terminal
# CVE-2024-3094: which of our images shipped xz/liblzma 5.6.0 or 5.6.1?
for sbom in sboms/*.cdx.json; do
  jq -r --arg img "$(basename "$sbom" .cdx.json)" '
    .components[]
    | select(.name == "liblzma5" or .name == "xz-utils")
    | select(.version == "5.6.0" or .version == "5.6.1")
    | "\($img)\t\(.name)\t\(.version)"' "$sbom"
done
output
payments-api    liblzma5   5.6.1
payments-api    xz-utils   5.6.1
notify-worker   liblzma5   5.6.1

Two images came back: a payments service and a notification worker. That is your recall list. Every other image is cleared by the same query, which matters as much as the hits, because it stops you from tearing apart services that were never affected.

Next, read the build's paperwork. cosign (a tool that signs and verifies container images and the attestations attached to them) checks the provenance against the identity that was allowed to build it.

terminal
cosign verify-attestation \
  --type slsaprovenance \
  --certificate-identity-regexp '^https://github.com/acme/.+/\.github/workflows/release\.yml@.+' \
  --certificate-oidc-issuer https://token.actions.githubusercontent.com \
  registry.acme.internal/payments-api@sha256:9b2c1e4f0ab3...f0
output
Verification for registry.acme.internal/payments-api@sha256:9b2c1e4f0ab3...f0 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The code-signing certificate was verified using trusted certificate authority certificates
  - Existence of the claims in the transparency log was verified offline
Certificate subject: https://github.com/acme/payments-api/.github/workflows/release.yml@refs/heads/main
Certificate issuer URL: https://token.actions.githubusercontent.com
{"payloadType":"application/vnd.in-toto+json","payload":"eyJfdHlwZSI6...","signatures":[{"sig":"MEUCIQ..."}]}

Read what that says. The certificate subject is the exact GitHub Actions workflow and git branch that built the image, and the issuer is GitHub's OpenID Connect service (OIDC, a way to prove identity with short-lived tokens instead of long-lived keys). For this incident, that verification passes, and the image is still poisoned. That is not a contradiction. Provenance proves the artifact came from your real pipeline; it says nothing about a backdoor that your pipeline pulled in from upstream in good faith. Hold onto that distinction. It decides what you do next.

The other failure mode is the reverse: someone misused a signing identity, or breached the builder, and signed something you never authorized. That is where the transparency log earns its keep. It is append-only and public, like a notary's logbook where pages cannot be torn out, so an attacker who signs with your identity leaves a permanent, timestamped record they cannot erase. Search it by artifact hash, then read the entry.

terminal
# rekor-cli search --sha <digest> returned one entry; read it:
rekor-cli get --uuid 108e9186e8c5677afd21e3b0c1d4a7...9c
output
LogID: c0d23d6ad406973f9559f3ba2d1ca01f84147d8ffc5b8445c224f98b9591801d
Index: 88213443
IntegratedTime: 2024-03-29T03:14:07Z
UUID: 108e9186e8c5677afd21e3b0c1d4a7...9c
Body:
  DSSEObj:
    signatures:
      - verifier: |
          -----BEGIN CERTIFICATE-----
          ... GitHub Actions OIDC identity: release.yml@refs/heads/main ...
          -----END CERTIFICATE-----

The tell here is the timestamp. IntegratedTime reads 03:14 in UTC (Coordinated Universal Time, the global reference clock), and no build was scheduled then. A legitimate release lines up with a pipeline run you can point to. A signature that appears in the dead of night, or from a workflow reference you do not recognize in the embedded certificate, is a signing identity being abused. When you see that, you revoke the identity and treat everything signed inside the suspicious window as untrusted.

Tracing and cutting off a supply-chain compromise
1Detect
backdoored dep, breached builder, or misused signer
2Scope
SBOMs, provenance, and Rekor find every affected artifact
3Locate
registry and deploy records show where they run
4Contain
block the digests at admission, revoke the signer
5Eradicate
rebuild clean, rotate every reachable secret
6Verify + harden
re-scan, keep the deny rule, add the missing control

Contain: find where the poison runs, then shut the door

Containment means stopping the spread while you work, the way a recall pulls product off shelves before anyone has figured out the root cause. First, find where the affected images are running right now. You hunt by digest, the sha256 content hash that names one exact image. A tag like :latest can be moved to point somewhere else, but a digest cannot, so it is the precise thing to search for.

terminal
kubectl get pods -A \
  -o custom-columns='NS:.metadata.namespace,POD:.metadata.name,IMAGE:.spec.containers[*].image' \
  --no-headers | grep -E 'payments-api|notify-worker'
output
prod   payments-api-7d9c8b6f5-2xk4l    registry.acme.internal/payments-api@sha256:9b2c1e4f0ab3...f0
prod   payments-api-7d9c8b6f5-qp7zn    registry.acme.internal/payments-api@sha256:9b2c1e4f0ab3...f0
prod   notify-worker-5f7b9c4d8-h4m2p   registry.acme.internal/notify-worker@sha256:1a4e77b9c2d5...c3

Three pods across the two services. Now shut the door so no new copies start. In Kubernetes that door is admission control, the gate that inspects every pod before the cluster is allowed to run it. You may already run a policy there that verifies signatures, but that will not help in this incident, because the poisoned image is correctly signed. What you need is a plain deny rule on the specific compromised digests, your indicators of compromise (IOCs, the concrete fingerprints of the attack). Kyverno (a policy engine for Kubernetes that allows or denies workloads by rule) does it in a few lines.

quarantine-cve-2024-3094.yaml
apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: quarantine-cve-2024-3094
  annotations:
    policies.kyverno.io/description: >-
      Deny the known-compromised image digests from incident INC-2481.
spec:
  validationFailureAction: Enforce
  background: true
  rules:
    - name: deny-poisoned-digests
      match:
        any:
          - resources:
              kinds:
                - Pod
      validate:
        message: "Quarantined image digest (INC-2481 / CVE-2024-3094). Rebuild with a clean xz."
        foreach:
          - list: "request.object.spec.containers"
            deny:
              conditions:
                any:
                  - key: "{{ element.image }}"
                    operator: AnyIn
                    value:
                      - "*@sha256:9b2c1e4f0ab3*"
                      - "*@sha256:1a4e77b9c2d5*"
terminal
kubectl apply -f quarantine-cve-2024-3094.yaml
output
clusterpolicy.kyverno.io/quarantine-cve-2024-3094 created
A verified signature is not a clean bill of health
cosign passing and provenance verifying prove where an artifact came from and that no one tampered with it after signing. They do not prove its contents are safe. A backdoored dependency rides in through a build your pipeline trusts, so the malicious image is signed correctly by your real identity. And if the builder or a long-lived signing key was breached, the attacker can produce malicious artifacts that verify perfectly. Treat verification as necessary, never sufficient, and rotate every credential a breached builder could read, because it likely read all of them.

Eradicate by rebuilding, not by cleaning

You cannot scrub a backdoor out of a built artifact and trust the result, the same way you cannot pick contamination out of a finished cake. You rebuild. Pull clean source, pin the dependency to a version known good (for xz, that meant dropping back to the 5.4 line until a verified fix shipped), build fresh through the pipeline, and produce a new artifact with a new digest and new provenance. Then rotate secrets, and rotate them wider than feels comfortable. If the incident was a breached build server, assume the builder read every secret it could reach: registry push tokens, cloud credentials, and any long-lived signing keys. Short-lived keyless signing helps here, because there is no standing key to steal, but you still distrust anything signed during the compromise window. Revoke the affected signing identity or credentials, halt the pipeline path that produced the bad artifacts, and only then redeploy the verified rebuild.

Prove the fix, then keep the guard

Do not take remediation on faith. Verify it the same way you scoped it. Re-run the SBOM query against the rebuilt images and expect zero hits. Confirm no workload is still running the quarantined digests. And test the admission rule by trying to run the bad image on purpose.

terminal
kubectl run repro --image=registry.acme.internal/payments-api@sha256:9b2c1e4f0ab3...f0
output
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/default/repro was blocked due to the following policies
 
quarantine-cve-2024-3094:
  deny-poisoned-digests: 'Quarantined image digest (INC-2481 / CVE-2024-3094). Rebuild
    with a clean xz.'

Blocked, with your incident number in the message, which is exactly what you want the next person to see. Keep that deny rule in place after the fire is out. Those digests and the poisoned dependency version are permanent indicators of compromise, so feed them to your detection and leave the quarantine standing. Write the trace into the incident record too: which SBOMs matched, which digests, which namespaces, and which signing window you distrusted. The last step is the one that pays forward. Add the control that would have caught this earlier, whether that is failing the build when a dependency's checksum changes unexpectedly, or requiring provenance that names an approved source before an image can deploy, so the same class of attack cannot walk the same path twice.

Quick check
01Your SBOM sweep flags payments-api for liblzma5 5.6.1, and cosign verify-attestation on that same digest passes, printing certificate subject release.yml@refs/heads/main. What do you take from those two results together?
Incorrect — Both readings are true at the same time. The ingredient list tells you what went into the build, the attestation tells you who ran the build, and they answer different questions.
Correct — A green attestation covers where the artifact came from and that nobody altered it afterward. It says nothing about a poisoned library your pipeline pulled in believing it was fine.
Incorrect — Treat verification as necessary and never enough on its own. Promoting on a green check is precisely how a backdoored artifact reaches production with its paperwork in order.
Incorrect — Abuse of a signer is the other failure mode, and it shows up as signatures you cannot match to a real run. Here the subject names the workflow that was supposed to build this image.
02Your cluster already runs an admission policy that verifies image signatures, yet the poisoned payments-api pod schedules without a murmur. Why does that policy miss it, and what does the Kyverno rule you write next actually match on?
Incorrect — Admission is already where it runs. Timing is not the gap here; the gap is that a signature check can ask who built an image but never what is inside it.
Incorrect — Signatures do not lapse when a flaw becomes public. Signing the same poisoned bits a second time only puts your name on them again, and the rebuild is what fixes the contents.
Incorrect — Nothing here is malfunctioning. The webhook is doing its job flawlessly and admitting an image whose signature genuinely is valid, which is the entire problem.
Correct — A backdoor that rode in through a build you trust carries your real identity, so you block on the one property the attacker cannot change: the content hash, listed as your indicators of compromise.
03rekor-cli get on a suspect artifact returns IntegratedTime 2024-03-29T03:14:07Z, and nothing in your release history ran anywhere near that hour. How do you read it, and what happens next?
Incorrect — That integrated time is a notarized record rather than a casual local reading. A gap between it and your build calendar is evidence to act on, not measurement error to explain away.
Incorrect — The trailing Z means the time is already on the global reference clock, and a legitimate release still points back at a pipeline run you can name and open.
Correct — A signature with no build standing behind it is the fingerprint of a misused signer, and because the ledger only ever appends, that abuse stays on the record permanently.
Incorrect — Pages cannot be torn out of this ledger, which is exactly why the entry is worth reading. Comparing hashes also would not tell you who was signing at three in the morning.

Try this

Run for sbom in sboms/*.cdx.json; do on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

The trap worth remembering here: a verified signature is not a clean bill of health. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Continuous rescanning & VEXA DevSecOps maturity roadmap

Related

What CI is, and where Jenkins fitsJenkins foundations, done rightThe .gitlab-ci.yml modelSecure CI/CD with GitLabSupply-chain threat modelSoftware supply chain in depthCI/CD interview questionsInterview prep