Pinning, SBOMs, provenance and scanning

Digest pins and the bots that move them, BuildKit attestations, Syft, Trivy and Grype gates, and VEX.

An advisory lands on a Tuesday: the Werkzeug debugger lets an attacker run code (CVE-2024-34069), fixed in Werkzeug 3.0.3. Security asks two questions. Which images we run contain Werkzeug older than 3.0.3? And in which of them can anyone reach the debugger? If a software bill of materials (SBOM) is stored for every image digest, the first question is a search through files you already have. Without SBOMs it means pulling and unpacking every image in every registry. The second question no scanner can answer for you.

This lesson builds a deliberately outdated Flask image with BuildKit's SBOM and provenance attestations, generates SBOMs with Syft, scans with Trivy and Grype, gates on the result and records a "not affected" decision as VEX. Signing and promotion follow in "Signing, verifying and promoting by digest". Use the main lab VM. The lesson files go in ~/lab/provenance: app/ (Dockerfile, requirements.txt, app.py), install-tools.sh and vex.json. The lab needs internet access to GitHub, PyPI and the scanners' databases.

Install the tools

Syft, Trivy and Grype are static binaries. install-tools.sh downloads pinned releases from GitHub, checks each one against a SHA-256 value committed in the script itself and installs it into ~/lab/bin, which the cleanup deletes. Nothing is installed system-wide.

#!/bin/sh
# Usage: ./install-tools.sh TOOL...   (TOOL: cosign syft trivy grype)
# Downloads pinned release binaries from GitHub, checks each against the SHA-256 committed below,
# and installs them into $BIN (default ~/lab/bin). Nothing system-wide.
# The hashes live in this file, in your repository: a release asset replaced on GitHub after you
# pinned it fails the check. Bumping a version means updating its two hashes in a reviewed change.
set -eu
BIN=${BIN:-$HOME/lab/bin}
COSIGN=3.1.3 SYFT=1.54.1 TRIVY=0.75.0 GRYPE=0.120.1
case $(uname -m) in
  aarch64|arm64) A=arm64 T=ARM64 ;;
  x86_64)        A=amd64 T=64bit ;;
  *) echo "unsupported CPU $(uname -m)" >&2; exit 1 ;;
esac
sha() {  # sha ASSET: the pinned SHA-256 of a release asset
  case $1 in
    cosign-linux-arm64)                echo c5d324e091826b0d7a78eb16fef316450b4eb9aaec045611c08ba06f5e73220a ;;
    cosign-linux-amd64)                echo 4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71 ;;
    syft_1.54.1_linux_arm64.tar.gz)    echo dfdf0537610113edbefe1f1fc6548bc957b2d77439636ec824fcf0e10d46d054 ;;
    syft_1.54.1_linux_amd64.tar.gz)    echo c069905b391cc4c20a5ba65ad5c10be2a7ba074f8ea6ad203e24d14e303dad47 ;;
    trivy_0.75.0_Linux-ARM64.tar.gz)   echo a1ee9f6ffb7d112b64ff726a2a0717c21175c1114361391f4a132956751a13b3 ;;
    trivy_0.75.0_Linux-64bit.tar.gz)   echo c6e65abddb348e25f10549df887045629cf28cc72453cd1c63acb717316b3f3f ;;
    grype_0.120.1_linux_arm64.tar.gz)  echo 29f47391dc283aa79fcc38e65224cd61f64dec0ecfd0db7074128ebf8ff23514 ;;
    grype_0.120.1_linux_amd64.tar.gz)  echo 0a9ee97ef5ae2ee953b0a80098105052e846cdbe319a57d808b519c33cd1343d ;;
    *) echo "no pinned hash for $1" >&2; return 1 ;;
  esac
}
GH=https://github.com
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
mkdir -p "$BIN"
 
fetch() {  # fetch URL_DIR ASSET: download the asset and check it against the pinned hash
  want=$(sha "$2")
  curl -fsSL --retry 10 --retry-delay 5 --retry-all-errors -o "$tmp/$2" "$1/$2"
  (cd "$tmp" && echo "$want  $2" | sha256sum -c -)
}
 
for t in "$@"; do
  case $t in
    cosign) u=$GH/sigstore/cosign/releases/download/v$COSIGN
            fetch "$u" "cosign-linux-$A"
            install -m 0755 "$tmp/cosign-linux-$A" "$BIN/cosign" ;;
    syft)   u=$GH/anchore/syft/releases/download/v$SYFT
            fetch "$u" "syft_${SYFT}_linux_$A.tar.gz"
            tar -xzf "$tmp/syft_${SYFT}_linux_$A.tar.gz" -C "$BIN" syft ;;
    trivy)  u=$GH/aquasecurity/trivy/releases/download/v$TRIVY
            fetch "$u" "trivy_${TRIVY}_Linux-$T.tar.gz"
            tar -xzf "$tmp/trivy_${TRIVY}_Linux-$T.tar.gz" -C "$BIN" trivy ;;
    grype)  u=$GH/anchore/grype/releases/download/v$GRYPE
            fetch "$u" "grype_${GRYPE}_linux_$A.tar.gz"
            tar -xzf "$tmp/grype_${GRYPE}_linux_$A.tar.gz" -C "$BIN" grype ;;
    *) echo "unknown tool $t" >&2; exit 1 ;;
  esac
done

Each OK line is sha256sum -c accepting one download. Because the hashes live in the script, in your repository, a release asset that is replaced on GitHub after you pinned it fails the check. A checksum file downloaded from the same release cannot catch that, since whoever can publish the binary can publish a matching checksum file. The hashes still have to be right when you first pin them, so take them from a release whose signature you verified (the next lesson verifies the signature on Trivy's checksum list), and bump a version by changing its hashes in a reviewed commit, as "Docker in CI/CD: build to promote" (Docker in depth) does. Run the export again in every new shell. Each tool also ships as a container image; if you use one, pin it by tag@digest, as "Slimming images and layer hygiene" does for dive; Trivy's own images were part of the incident below, and only references by digest were safe.

A pinning policy

In March 2026 an attacker with stolen credentials force-pushed 76 of the 77 version tags of the aquasecurity/trivy-action GitHub Action to credential-stealing code and published a malicious Trivy release (CVE-2026-33634). Workflows that used the action by version tag ran the stealer. Trivy images referenced by digest were not affected, and neither were workflows pinned to a recent commit SHA. Pins to commits older than April 2025 were still exposed, because that code fetched a helper action by tag. A pin covers what it names, not what the pinned code pulls in. Every input to a build can be pinned:

The lab Dockerfile pins its base by index digest. Its requirements file pins Flask and Werkzeug to 2.2.2, released in 2022, so the scanners have something to find:

# python:3.14-slim, pinned to the index digest the tag pointed at when this lab was written
FROM python:3.14-slim@sha256:f85c5697265c178cc6887276c55fe16cf3d14ca35c3df6a5eab3b360534a55d2
WORKDIR /app
COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt
COPY app.py .
USER 10001
CMD ["python", "app.py"]

flask==2.2.2
werkzeug==2.2.2

A pin nobody moves becomes a frozen vulnerability that misses every Debian security update after the day it was written. Give pins to a bot. Renovate and Dependabot notice when a tag points at a new digest, or a newer tag exists, and open a pull request that changes the pin; the pipeline rebuilds, tests and scans it like any other change. Renovate's config:best-practices preset pins and updates digests for Docker images and GitHub Actions:

{
  "$schema": "https://docs.renovatebot.com/renovate-schema.json",
  "extends": ["config:best-practices"],
  "packageRules": [
    {
      "matchDatasources": ["docker"],
      "matchUpdateTypes": ["digest"],
      "automerge": true
    }
  ]
}

The package rule merges digest-only updates (same tag, rebuilt image) once the checks pass, while new tags wait for a human. Dependabot does the same from .github/dependabot.yml and updates the digest of a pinned FROM line together with its tag:

# version 2 is the dependabot.yml schema; do not confuse it with the obsolete Compose key
version: 2
updates:
  - package-ecosystem: "docker"
    directory: "/app"
    schedule:
      interval: "weekly"
  - package-ecosystem: "pip"
    directory: "/app"
    schedule:
      interval: "weekly"
  - package-ecosystem: "github-actions"
    directory: "/"
    schedule:
      interval: "weekly"

Either way, every pin has a bot that moves it, every move goes through build, test and scan, and nobody edits a digest by hand.

Attestations from the build

BuildKit can attach two attestations to an image: an SBOM (--sbom=true) and SLSA provenance, a record of how the image was built (--provenance). Both are in-toto statements, JSON documents that name the image by digest, stored in the attestation manifest that "Image anatomy: index, manifest, config and layers" in Docker in depth took apart. Provenance in min mode is on by default; the SBOM is not. The default docker driver can produce them because fresh Docker 29 installs use the containerd image store. A host still on the legacy store refuses with "Attestation is not supported for the docker driver"; use a docker-container builder there ("Buildx builders, outputs, cache and Bake" in Docker in depth).

Start a local registry and build with both attestations, provenance in max mode. --metadata-file writes the build result, including the digest, to a JSON file:

The resolve image config and docker-image:// steps fetch docker/buildkit-syft-scanner, the SBOM generator: an image with Syft inside that BuildKit pulls from Docker Hub. The generating sbom step near the end runs it against the final stage. Earlier stages of a multi-stage build are not scanned unless the Dockerfile declares ARG BUILDKIT_SBOM_SCAN_STAGE=true (BUILDKIT_SBOM_SCAN_CONTEXT does the same for the build context). containerimage.digest is the index digest, the value you sign, deploy and promote, and app.digest keeps it for the rest of the lesson. It changes on every build because the config records the build time.

The attestation manifest now carries two layers, one per attestation, each labelled with its predicate type:

The SPDX document is 2 MB because Syft lists every file it catalogued as well as every package. The provenance is about 10 kB. Read the SBOM through imagetools, which unwraps the in-toto statement:

The creators name the generator, Syft 1.51.0 bundled with BuildKit 0.33.1. Keep that with the SBOM, because generator versions differ in what they list. The SBOM covers the Python packages pip installed and the Debian packages of the base image. For a multi-platform image, select one platform with --format '{{ json (index .SBOM "linux/amd64").SPDX }}'.

The provenance answers "what was this built from":

resolvedDependencies lists the base image with the digest BuildKit used, and the scanner image, whose digest is in the full record. That works even for an unpinned FROM, so provenance tells you which base a running image came from. The last line, a build step, exists only in max mode. Build again with the default min mode (-q prints only the new digest) and compare:

min holds the materials, the frontend, the names of the local contexts and timestamps, about 1 kB. max adds every build step with its command and environment, the Dockerfile text and the layers each step produced. That detail makes it useful in an incident, and it is also why max publishes build-argument values next to the image ("Build-time secrets and how images leak them" shows the leak). max is safe to use once no credential appears in a build argument or anywhere on a RUN command line, such as a token in a pip install --index-url URL, because it publishes the Dockerfile and every command with the image. Internal hostnames and registry paths in those commands ship too.

SBOMs from Syft

Syft is the standalone form of the generator BuildKit used. You need it for images you did not build, such as vendor images and old releases, and for SBOM formats a customer or tool asks for. It reads the image from the registry and writes several formats in one pass:

SPDX (Linux Foundation) and CycloneDX (OWASP) describe the same packages in different schemas. Both identify a package by its package URL (purl), pkg:pypi/werkzeug@2.2.2, which is what scanners match against advisories, and Trivy and Grype read both. Besides registry: (plain HTTP here, as Docker allows for localhost registries), Syft reads the local Docker store, docker save archives and OCI layouts.

Syft 1.54.1 lists 102 packages; BuildKit's SBOM of the same image listed 121. The difference:

pip ships its own SBOM, pip/_vendor/bom.cdx.json, describing the libraries it bundles. The Syft inside BuildKit read it and added those packages; standalone Syft 1.54.1 with default settings did not. Neither is wrong, but a scan of an SBOM can only find what the SBOM lists, and the scans below show what that costs. Syft also scans source trees:

A directory scan sees only what the manifests declare: two direct dependencies, no Jinja2 or Click (installed as dependencies of Flask), no Debian packages, no Python. It is useful for checking a repository before anything is built, not as a replacement for the image SBOM.

Scanning with Trivy and Grype

A scanner matches packages against vulnerability databases. Trivy downloads its database on the first run (into ~/.cache/trivy, about 1.4 GB unpacked on this VM). Scan the image by digest, HIGH and CRITICAL only, and count the findings by fix status:

None of the 44 Debian findings has a fix: affected means Debian has not released a patched package, and fix_deferred means Debian decided not to fix it in this release. Rebuilding cannot clear them, so a gate that counts them fails every build until Debian acts or you change the base. The Python findings all have fixed versions. Gate on what the team can fix, and keep the rest in view: store the full report as a non-blocking artifact of the job, and review the unfixed HIGH and CRITICAL findings on a schedule with an owner. An unfixed CRITICAL that is being exploited is a reason to change the base image, not to wait for Debian.

--ignore-unfixed drops findings without a fix, and --exit-code 1 makes Trivy exit 1 when anything remains, which fails the pipeline step. The first three are in packages the Dockerfile installs. The last four have no path: Trivy found them in pip's bundled SBOM, so they are libraries inside pip. Now scan the SBOM that Syft wrote instead of the image:

Three findings, not seven, because Syft's SBOM does not list pip's bundled libraries. Scanning stored SBOMs is fast and needs no registry access, which suits a nightly rescan of every release you have shipped (the image does not change; the advisories do). Its coverage is the generator's coverage, though. Grype, Anchore's scanner, reads the same inputs. Its database is larger (about 3 GB unpacked) and the first download takes a few minutes:

Grype reports GitHub advisory IDs (GHSA-2g68-c3qc-8985 is CVE-2024-34069). --only-fixed is its --ignore-unfixed but keeps every severity, so Medium and Low rows appear, some for the Python interpreter itself. --fail-on high makes Grype exit 2 when a High or Critical finding remains. Its High findings are the same three on the image and the SBOM: Grype catalogs images with Syft and misses pip's bundled libraries too. EPSS is the Exploit Prediction Scoring System estimate that a vulnerability will be exploited in the next 30 days. Use one scanner for the gate; two in one pipeline mostly produce two lists to reconcile. In CI, cache the scanner database between jobs or point the scanner at a mirror of it, instead of downloading gigabytes on every run and running into rate limits.

Known is not exploitable

Every finding above is a known vulnerability, meaning that a package version matches an advisory. Whether it is exploitable in this image is a different question. CVE-2024-34069 needs the Werkzeug debugger, which this app never enables. The urllib3, msgpack and setuptools findings are in pip's bundled libraries, which run only when someone runs pip inside the container. Nothing in production should, and a multi-stage build that leaves pip out of the final image removes the findings together with the code. The Flask session-cookie finding applies only behind a caching proxy. EPSS adds the outside view of whether anyone is exploiting a vulnerability at all. None of that removes the need to patch; it decides what must block a release today and what can ride on next week's base-image bump.

VEX (Vulnerability Exploitability eXchange) records that decision in a machine-readable file: product, vulnerability, status (not_affected, affected, fixed, under_investigation) and, for not_affected, a justification from a fixed list. Scanners read it and stop reporting the finding, so the decision is written once, reviewed like code and stays with the artifact instead of living in a ticket. This OpenVEX document covers the debugger CVE:

{
  "@context": "https://openvex.dev/ns/v0.2.0",
  "@id": "https://example.test/vex/lab-app/2026-10-07",
  "author": "lab-app maintainers",
  "timestamp": "2026-10-07T00:00:00Z",
  "version": 1,
  "statements": [
    {
      "vulnerability": { "name": "CVE-2024-34069" },
      "products": [
        {
          "@id": "pkg:oci/lab-app?repository_url=localhost%3A5000%2Flab-app",
          "subcomponents": [ { "@id": "pkg:pypi/werkzeug@2.2.2" } ]
        }
      ],
      "status": "not_affected",
      "justification": "vulnerable_code_not_in_execute_path",
      "impact_statement": "The Werkzeug debugger is never enabled; app.py does not pass debug=True."
    }
  ]
}

The gate still fails, on the six findings that remain, and --show-suppressed lists what the VEX statement removed and why. Scope a statement as narrowly as the facts allow. The justification is a fact about this application, so the product is this image (pkg:oci/lab-app with its repository) and Werkzeug 2.2.2 is a subcomponent of it. VEX files get shared, published centrally and attached to base images, and a statement that named only pkg:pypi/werkzeug@2.2.2 would hide this RCE in every image with that version, including a service that does run the debugger. The package version in the subcomponent also makes the next Werkzeug upgrade end the statement instead of hiding a new problem. Trivy also reads CycloneDX VEX and CSAF; the next lesson attaches documents like this one to the image as signed attestations.

Clean up

Remove the registry, the images, the generated files and the tools with their databases (about 4.5 GB):

Try this

Work through “Clean up” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from pinning, sboms, provenance and scanning, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related