Trusting a minimal image: pin, SBOM, scan
Provenance for images with no package database.
A Docker tag is a sticky note. Somebody wrote acme/svc:1.0 on it and pressed it onto a box. The name points at whatever bytes were pushed under it most recently, and anyone with push access can peel that note off one box and stick it on another. The bytes underneath change. The name you deploy reads exactly the same. That gap, between a label anyone can move and the bytes it happens to point at tonight, is what you are defending against in this lesson.
Walk the poisoned-tag attack through once, slowly. An attacker gets write access to your registry namespace. Or you mistyped a public image name and pulled a convincing look-alike. Or the upstream base you build FROM was compromised before you ever saw it. They push a backdoored image over :1.0. Tonight your CI (continuous integration, the robot that rebuilds and ships your code on a schedule) resolves :1.0 to the new bytes and ships the backdoor with your pipeline's stamp of approval on it. A distroless base (an image carrying no shell and no package manager, only the libraries your program needs) with four packages in it does nothing to stop that. The image is still tiny. It is the wrong tiny image. Slimming an image cuts how much code an attacker can reach. It says nothing about who produced the bytes, or whether they are the bytes you actually read and approved.
Pin to the exact bytes
A digest is a fingerprint. More precisely, it is a SHA-256 hash (Secure Hash Algorithm, 256-bit: a short fixed-length value computed from content, where any change to the content produces a completely different value) taken over the image manifest, the small file listing everything the image contains. Flip one byte anywhere and the fingerprint changes. So FROM base@sha256:... binds every rebuild to the identical bytes you reviewed, instead of to whatever the tag resolves to tonight. A tag is a name you can move. A digest is the thing itself. You can catch a tag drifting under you by asking the registry what it currently points at, and at deploy time you read back what the node actually pulled.
# a tag is a pointer. resolve the digest it actually points at, right now:$ docker buildx imagetools inspect acme/svc:1.0 --format '{{.Manifest.Digest}}'sha256:9f2ac1e0d3b47a5f1c8e2b0a6d4f9c3e7b1a8f2d5c6e0b9a4f7d2c1e8b3a6f0d1# a week later. same tag, different bytes. someone moved the pointer.$ docker buildx imagetools inspect acme/svc:1.0 --format '{{.Manifest.Digest}}'sha256:aa31be09f7c2d4e6b8a0f1c3d5e7092b4a6c8e0d2f4b6a8c0e2d4f6b8a0c2e4d6# at deploy, read what the node actually pulled and compare to the digest you signed:$ docker inspect --format '{{index .RepoDigests 0}}' acme/svc:1.0acme/svc@sha256:aa31be09f7c2d4e6... # <- not the 9f2a you reviewed. drift. reject.
One tag with two different digests a week apart is the poisoned-tag signature, and it is not subtle once you look. So write the digest into the Dockerfile. Moving up to a newer patched base then becomes a line somebody edits and somebody else reviews, rather than a change a nightly rebuild swallows on its own.
# pin the base to exact bytes. "the latest patch" becomes a decision you make,# reviewed in a diff, not a surprise a rebuild silently pulls in.FROM gcr.io/distroless/base-debian12@sha256:d71f4f2a9c... AS baseCOPY --from=build /server /serverUSER nonrootENTRYPOINT ["/server"]
Build the parts list
An image with no package manager is a sealed box with no lid on the inside. Distroless ships no apt and no shell, so apt list has nothing to run in there. The parts list still exists, though. Distroless keeps its dpkg database (the Debian package database, the file recording which packages and versions were installed) inside the layers even after it strips out the tools that read it. That lets syft, which builds an SBOM (software bill of materials, a machine-readable parts list for a piece of software), work from the outside: it reads that database straight out of the layers and fingerprints the binaries it finds along the way. A scratch image holding one static binary has no database at all, and syft falls back to classifying that binary itself. Either way you end up with a real component inventory without ever starting the container. Write it out once in SPDX (Software Package Data Exchange, a standard file format for these parts lists) and every tool downstream reads the same list.
$ syft acme/svc@sha256:9f2ac1e0d3b4 -o spdx-json=sbom.spdx.json✔ Loaded image acme/svc@sha256:9f2ac1e0✔ Parsed image sha256:9f2ac1e0✔ Cataloged contents├── ✔ Packages [23 packages]└── ✔ File digests [812 files]$ jq -r '.packages[] | "\(.name) \(.versionInfo)"' sbom.spdx.json | head -3base-files 12.4+deb12u5libc6 2.36-9+deb12u7libexpat1 2.5.0-1
Scan the list, fail the build
A scanner takes that inventory and checks every line of it against public vulnerability feeds. CVE (Common Vulnerabilities and Exposures) is the shared catalogue of known flaws, each one carrying an ID like CVE-2024-45491 so that two different tools describing the same bug use the same name for it. trivy will scan either the image or the SBOM you generated a minute ago. The piece that turns a report into a gate is the exit code, the small number a command hands back when it finishes to say whether it was happy. --exit-code 1 tells trivy to return non-zero the moment it finds anything at or above your severity threshold, and a non-zero exit fails the CI job. --ignore-unfixed keeps the gate from blocking on flaws that have no patch available yet, so it only ever stops work you can actually do something about.
$ trivy image --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 \acme/svc@sha256:9f2ac1e0d3b4acme/svc (debian 12.5)======================Total: 1 (HIGH: 0, CRITICAL: 1)┌───────────┬────────────────┬──────────┬────────┬───────────────┬───────────────────┐│ Library │ Vulnerability │ Severity │ Status │ Installed Ver │ Fixed Version │├───────────┼────────────────┼──────────┼────────┼───────────────┼───────────────────┤│ libexpat1 │ CVE-2024-45491 │ CRITICAL │ fixed │ 2.5.0-1 │ 2.5.0-1+deb12u1 │└───────────┴────────────────┴──────────┴────────┴───────────────┴───────────────────┘$ echo $?1 # non-zero exit -> the CI gate fails the build
Sign the digest, verify before it runs
A signature is a tamper-evident seal that also carries the name of whoever applied it. Cosign keyless signing borrows your CI's short-lived OIDC (OpenID Connect, the protocol one system uses to prove to another system who it is) identity instead of a long-lived key you would have to store and guard forever, and it writes a record of the signature into a public transparency log called Rekor. You sign the digest and never the tag, so the seal is glued to exact bytes. At admission, the moment your platform decides whether an image is allowed to run, you check the seal and, more importantly, the identity behind it. This is where the poisoned tag finally hits a wall it cannot climb.
$ cosign sign --yes acme/svc@sha256:9f2ac1e0d3b4Generating ephemeral keys...Retrieving signed certificate from Fulcio (identity: [email protected])...tlog entry created with index: 154203847$ cosign verify \--certificate-identity-regexp '.*@acme.internal' \--certificate-oidc-issuer https://gitlab.acme.internal \acme/svc@sha256:9f2ac1e0d3b4 | jq -r '.[0].optional.Subject'# the swapped bytes fail here. they were never signed by that identity:$ cosign verify --certificate-identity-regexp '.*@acme.internal' \--certificate-oidc-issuer https://gitlab.acme.internal \acme/svc@sha256:aa31be09f7c2Error: no matching signaturesmain.go:74: error during command execution: no matching signatures
A seal does nothing about time. An image that cleared every gate on Monday keeps collecting fresh CVEs all week, because researchers keep publishing new flaws against packages it already ships. Nothing inside the image changed. What the world knows about it did. So rescan what is actually running, not only what CI built on release day, and rebuild against a patched base on a schedule instead of waiting for an incident to force the bump.
# scan what's actually running, not just what CI built at release time.$ docker ps --format '{{.Image}}' | sort -u | while read img; dotrivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$img" \|| echo "NEEDS REBUILD: $img"doneNEEDS REBUILD: acme/svc@sha256:9f2ac1e0d3b4 # a HIGH landed against it since Monday# then bump FROM ...@sha256 to the patched base, rebuild, rescan, re-sign, redeploy.
cosign verify and watching it exit zero feels like a pass. With keyless signing it is not one, unless you also pin --certificate-identity (or the regexp form) and --certificate-oidc-issuer. Leave those off and all you have confirmed is that somebody, somewhere, signed this image and wrote a record of it into Rekor. The attacker who pushed your poisoned :1.0 can sign their own bytes with their own OIDC identity in about a minute, and that signature verifies cleanly too. The identity flags are the entire point. They turn 'this is signed' into 'this is signed by our CI'. Verify by digest every time, and pin the identity every time.Tags move. Digests do not. Deploy by digest, or by a signature that binds a digest, so CI cannot quietly pick up a poisoned retag. Pair the pin with an SBOM and a vulnerability scan even when the image carries no package database, because a language binary still drags a list of dependencies in with it.
Make cosign, or whatever equivalent you run, the gate standing in front of production. An unsigned latest promoted off a shared runner is the usual way a supply-chain incident walks straight past the carefully minimal base image you spent a week choosing.
Minimal images change how you scan. They do not excuse you from scanning. Track where the base digest in your FROM line came from, and fail the build when that digest changes without a human having reviewed the change.
In production these same commands are your after-the-change-window check. Confirm the control is still on, paste the command and its output into the ticket, and refuse to close the change if the reading drifted from what you expected. Keep the scope as tight as the workload will tolerate. That habit compounds across every host and every pipeline you touch.
Try this
Pull an image by tag, write down the digest it resolved to, retag it locally to imitate the confusion an attacker would create, and watch a deploy pinned to that digest stay exactly where you put it while the tag name alone promises you nothing.
$ docker pull alpine:3.20$ DIGEST=$(docker inspect --format='{{index .RepoDigests 0}}' alpine:3.20)$ echo "$DIGEST"alpine@sha256:...$ docker run --rm $DIGEST uname -aLinux ...$ # in CI: pin FROM alpine@sha256:... and cosign verify before promote
Takeaway
Pin digests. Sign what you deploy. Build an SBOM even for distroless. Tags are sticky notes; digests and signatures are the lock.
FROM registry/base@sha256:d71f..., and CI rebuilds every night. An attacker with push access overwrites that base image's :latest tag with a backdoored build. What does tonight's rebuild get?:latest now points has no bearing on a digest-pinned FROM.apt and no shell, yet syft still lists its Debian packages without ever starting the container. How does it manage that?scratch, where syft classifies the lone binary instead.acme/svc:1.0 with backdoored bytes and signs those bytes using their own pipeline's OIDC (OpenID Connect) identity. At admission you run cosign verify and it exits zero. What actually keeps that image out?