Docker in CI/CD: build to promote

Build, test, scan, push, attest, sign, verify and promote by digest.

Intermediate18 min · lesson 24 of 24
Lesson files
The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (11 files, 5 KB): dockerci.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/docker-hard/dockerci.tar.gz && tar -xzf dockerci.tar.gz, which creates ~/lab/dockerci/. SHA-256: 1c893c393c9139b2cb778aaa4575f9c663cce9b6d0ebc01b0eae3e6f2f803692

A common release job builds the image twice: once with load: true so Trivy can scan it on the runner, and once with push: true for the registry. The scan passes, the push succeeds, and the image in the registry is not the one that was scanned. It is a second build that matches the first only if every layer came from cache, and with provenance attached its digest differs anyway. Then a cosign sign step signs whatever the tag points at by the time it runs. Each step looks right; the order and the identity of the artifact are wrong. This lesson builds a pipeline in which one image, identified by one digest, goes through every gate, and promotion to production requires a signature over that digest. It runs on the main lab VM, secopslog-docker, in ~/lab/dockerci with the lesson files, as a script you could run in any CI system, then shows the same steps as a GitHub Actions workflow.

The order

One commit, one digest, nine steps
1Build
once, with provenance
2Test
run that image
3Scan
fail on fixable HIGH/CRITICAL
4Push
to the build repository
5Resolve digest
must equal the build's
6Attest
SBOM, signed
7Sign
the digest
8Verify
signature + attestation
9Promote
by digest: staging, then prod
Nothing reaches a registry before test and scan pass. Signing needs a digest that exists in a registry, so it comes after the push. Each promotion verifies first.

The order follows from what each step needs. Test and scan must run on the image you will ship, so the build happens once and every later step refers to its output. The push comes after the gates, so a failing build never appears in any registry. Signing and attesting come after the push because Cosign signs a manifest digest and stores the signature in the registry next to it; before the push there is nothing to sign. The digest is checked against the build's own record, so the signature covers the bytes that were tested. Verification is a separate step that a different job repeats before every promotion.

Three repositories carry the result, and a push to one is not an approval:

Anyone with push rights to a registry can put an image there ("Tags, digests and promotion" showed a direct push overwriting a production tag). What makes a digest approved is a signature and an attestation that verify against your key or your CI identity. The policy belongs in the promotion job and, in Kubernetes, in an admission controller that verifies before it admits a pod.

Tools, keys and registries

The lab uses the same pinned tools as "Pinning, SBOMs, provenance and scanning" and "Signing, verifying and promoting by digest" in Advanced container security, which cover Cosign and Syft in depth: Cosign 3.1.3, Syft 1.54.1 and Trivy 0.75.0, downloaded from their GitHub releases into ~/lab/bin. Nothing is installed system-wide. The script that fetches them carries its own SHA-256 values:

install-tools.sh
#!/bin/sh
# Usage: ./install-tools.sh TOOL... (TOOL: cosign syft trivy grype)
# Downloads pinned release binaries from GitHub, checks each against the SHA-256 committed below,
# and installs them into $BIN (default ~/lab/bin). Nothing system-wide.
# The hashes live in this file, in your repository: a release asset replaced on GitHub after you
# pinned it fails the check. Bumping a version means updating its two hashes in a reviewed change.
set -eu
BIN=${BIN:-$HOME/lab/bin}
COSIGN=3.1.3 SYFT=1.54.1 TRIVY=0.75.0 GRYPE=0.120.1
case $(uname -m) in
aarch64|arm64) A=arm64 T=ARM64 ;;
x86_64) A=amd64 T=64bit ;;
*) echo "unsupported CPU $(uname -m)" >&2; exit 1 ;;
esac
sha() { # sha ASSET: the pinned SHA-256 of a release asset
case $1 in
cosign-linux-arm64) echo c5d324e091826b0d7a78eb16fef316450b4eb9aaec045611c08ba06f5e73220a ;;
cosign-linux-amd64) echo 4629c757b7618056f8ddd7e2625ae9fdd94c0372a65049520bc7d9df9efc7f71 ;;
syft_1.54.1_linux_arm64.tar.gz) echo dfdf0537610113edbefe1f1fc6548bc957b2d77439636ec824fcf0e10d46d054 ;;
syft_1.54.1_linux_amd64.tar.gz) echo c069905b391cc4c20a5ba65ad5c10be2a7ba074f8ea6ad203e24d14e303dad47 ;;
trivy_0.75.0_Linux-ARM64.tar.gz) echo a1ee9f6ffb7d112b64ff726a2a0717c21175c1114361391f4a132956751a13b3 ;;
trivy_0.75.0_Linux-64bit.tar.gz) echo c6e65abddb348e25f10549df887045629cf28cc72453cd1c63acb717316b3f3f ;;
grype_0.120.1_linux_arm64.tar.gz) echo 29f47391dc283aa79fcc38e65224cd61f64dec0ecfd0db7074128ebf8ff23514 ;;
grype_0.120.1_linux_amd64.tar.gz) echo 0a9ee97ef5ae2ee953b0a80098105052e846cdbe319a57d808b519c33cd1343d ;;
*) echo "no pinned hash for $1" >&2; return 1 ;;
esac
}
GH=https://github.com
tmp=$(mktemp -d); trap 'rm -rf "$tmp"' EXIT
mkdir -p "$BIN"
fetch() { # fetch URL_DIR ASSET: download the asset and check it against the pinned hash
want=$(sha "$2")
curl -fsSL --retry 10 --retry-delay 5 --retry-all-errors -o "$tmp/$2" "$1/$2"
(cd "$tmp" && echo "$want $2" | sha256sum -c -)
}
for t in "$@"; do
case $t in
cosign) u=$GH/sigstore/cosign/releases/download/v$COSIGN
fetch "$u" "cosign-linux-$A"
install -m 0755 "$tmp/cosign-linux-$A" "$BIN/cosign" ;;
syft) u=$GH/anchore/syft/releases/download/v$SYFT
fetch "$u" "syft_${SYFT}_linux_$A.tar.gz"
tar -xzf "$tmp/syft_${SYFT}_linux_$A.tar.gz" -C "$BIN" syft ;;
trivy) u=$GH/aquasecurity/trivy/releases/download/v$TRIVY
fetch "$u" "trivy_${TRIVY}_Linux-$T.tar.gz"
tar -xzf "$tmp/trivy_${TRIVY}_Linux-$T.tar.gz" -C "$BIN" trivy ;;
grype) u=$GH/anchore/grype/releases/download/v$GRYPE
fetch "$u" "grype_${GRYPE}_linux_$A.tar.gz"
tar -xzf "$tmp/grype_${GRYPE}_linux_$A.tar.gz" -C "$BIN" grype ;;
*) echo "unknown tool $t" >&2; exit 1 ;;
esac
done

A checksum file downloaded from the same release as the binary only catches a damaged download: whoever can replace the binary on the release page can replace the checksum file next to it. Hashes committed in your repository, taken when you reviewed the version, catch that too, the same way the hashed lock file in "Recipe: a Python web app" protects the Python dependencies. Moving to a new tool version is then a reviewed change of the version and its two hashes. On a flaky connection curl prints curl: (35) lines while it retries; the OK lines prove what arrived.

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ ./install-tools.sh cosign syft trivy export PATH="$HOME/lab/bin:$PATH" cosign version --json | jq -r '"cosign \(.gitVersion)"'; syft version | grep '^Version'; trivy --version | head -1
cosign-linux-arm64: OK syft_1.54.1_linux_arm64.tar.gz: OK trivy_0.75.0_Linux-ARM64.tar.gz: OK cosign v3.1.3 Version: 1.54.1 Version: 0.75.0
$ export COSIGN_PASSWORD=lab-only-test-password cosign generate-key-pair cosign signing-config create --out lab-signing-config.json
Private key written to cosign.key Public key written to cosign.pub
$ docker run -d --name lab-registry -p 127.0.0.1:5000:5000 registry:3 docker run -d --name lab-prod-registry -p 127.0.0.1:5001:5000 registry:3
bebf626ec88d6fde80f29106c5d506382286d5a0eb613fcda74eb5f0a2045364 ea499f5544b63a7adc2826b13b98091f9e59891f5dc42b2a876d514084b215b2

Locally, signing uses a key pair protected by a test password (COSIGN_PASSWORD, exported in the shell), and a signing configuration with no transparency log, so lab signatures are not published to the public Rekor log. Verification therefore passes --insecure-ignore-tlog=true. In CI the job signs keylessly instead, as the workflow below shows; "Signing, verifying and promoting by digest" (Advanced container security) explains both modes. Two registry:3 containers stand in for the build and production registries.

The pipeline script

ci.sh
#!/bin/sh
# Usage: ./ci.sh VERSION
# The pipeline one commit goes through. The CI job runs this same script; locally it uses
# cosign.key with COSIGN_PASSWORD, in GitHub Actions the sign step is keyless (see the workflow).
# Stops at the first failing stage. Nothing reaches the registry before test and scan pass.
set -eu
VERSION=$1
REPO=${BUILD_REPO:-localhost:5000/lab-build/app}
SIGN="--key cosign.key --signing-config lab-signing-config.json"
VERIFY="--key cosign.pub --insecure-ignore-tlog=true"
stage() { printf '\n== %s\n' "$*"; }
stage "1 build $REPO:$VERSION"
# the default (docker driver) builder: with the containerd image store, --load keeps the index
docker buildx build --builder default -q --provenance=mode=max --metadata-file build.json \
-t "$REPO:$VERSION" --load app
built=$(jq -r '."containerimage.digest"' build.json)
echo "built $built"
stage "2 test: start the image, call it, stop it"
docker run -d --name lab-ci-test -p 127.0.0.1:3900:3000 "$REPO:$VERSION" >/dev/null
trap 'docker rm -f lab-ci-test >/dev/null 2>&1' EXIT
for i in $(seq 1 30); do
[ "$(docker inspect -f '{{.State.Health.Status}}' lab-ci-test)" = healthy ] && break; sleep 1
done
curl -fsS http://127.0.0.1:3900/ | grep -q '"service":"payments"'
docker rm -f lab-ci-test >/dev/null
echo "test passed"
stage "3 scan: fail on HIGH or CRITICAL with a fix available"
trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$REPO:$VERSION"
echo "scan passed"
stage "4 push the image that was tested and scanned"
docker push -q "$REPO:$VERSION"
stage "5 resolve the digest"
digest=$(docker buildx imagetools inspect "$REPO:$VERSION" --format '{{.Manifest.Digest}}')
[ "$digest" = "$built" ] || { echo "registry has $digest, the build produced $built"; exit 1; }
echo "$digest" | tee "release-$VERSION.digest"
stage "6 attest: SBOM (signed) next to BuildKit's provenance"
syft scan -q "registry:$REPO@$digest" -o spdx-json=sbom.spdx.json
cosign attest --yes $SIGN --type spdxjson --predicate sbom.spdx.json "$REPO@$digest"
stage "7 sign the digest"
cosign sign --yes $SIGN "$REPO@$digest"
stage "8 verify before anyone may promote it"
cosign verify $VERIFY "$REPO@$digest" >/dev/null 2>&1
cosign verify-attestation $VERIFY --type spdxjson "$REPO@$digest" >/dev/null 2>&1
echo "VERIFIED $REPO@$digest"

Step 1 builds once with docker buildx build --load and --provenance=mode=max, and --metadata-file records the digest the build produced. The build names the default builder, which uses the docker driver: on Docker 29 with the containerd image store, --load from it keeps the image index with its provenance attestation, so the image in the local store is exactly what will be pushed. A docker-container builder, such as one left selected with docker buildx use, loads a single manifest without the attestation ("Buildx builders, outputs, cache and Bake"), and step 5 would then fail on a digest mismatch. Step 2 starts that image, waits for its health check and calls it. A real project also runs its unit tests, either before the build or in a test stage; this step tests the artifact. Step 3 scans the same local image and fails on HIGH or CRITICAL findings that have a fixed version. Step 5 asks the registry for the digest behind the pushed tag and refuses to continue unless it equals the build's. Steps 6 to 8 attach a Syft SBOM as a signed attestation, sign the digest and verify both.

Run 1: a vulnerable dependency

The app pins minimist 1.2.5, which has a known prototype-pollution vulnerability:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ grep -A1 '"node_modules/minimist"' app/package-lock.json
"node_modules/minimist": { "version": "1.2.5",
$ ./ci.sh 1.5.0-ci.1
== 1 build localhost:5000/lab-build/app:1.5.0-ci.1 sha256:211957f92245895eadb693314830fa1de53f3b5d0419468555ed60a1144c4c75 built sha256:211957f92245895eadb693314830fa1de53f3b5d0419468555ed60a1144c4c75 == 2 test: start the image, call it, stop it test passed == 3 scan: fail on HIGH or CRITICAL with a fix available ... Node.js (node-pkg) ================== Total: 1 (HIGH: 0, CRITICAL: 1) ┌─────────────────────────┬────────────────┬──────────┬────────┬───────────────────┬───────────────┬────────────────────────────────────────────┐ │ Library │ Vulnerability │ Severity │ Status │ Installed Version │ Fixed Version │ Title │ ├─────────────────────────┼────────────────┼──────────┼────────┼───────────────────┼───────────────┼────────────────────────────────────────────┤ │ minimist (package.json) │ CVE-2021-44906 │ CRITICAL │ fixed │ 1.2.5 │ 1.2.6, 0.2.4 │ minimist: prototype pollution │ │ │ │ │ │ │ │ https://avd.aquasec.com/nvd/cve-2021-44906 │ └─────────────────────────┴────────────────┴──────────┴────────┴───────────────────┴───────────────┴────────────────────────────────────────────┘
$ curl -s http://localhost:5000/v2/_catalog; echo
{"repositories":[]}

Build and test passed; the scan found CVE-2021-44906, CRITICAL, with fixed versions available, and --exit-code 1 stopped the script. The catalog of the build registry is empty: the image that failed the gate exists only on the build machine. --ignore-unfixed keeps the gate on findings you can act on today; findings without a fix still belong in a report and a risk decision, not in a blocked merge. "Pinning, SBOMs, provenance and scanning" (Advanced container security) covers severity policy and VEX.

Run 2: the fix

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ diff app/package.json fix/package.json cp fix/package.json fix/package-lock.json app/
7c7 < "minimist": "1.2.5", --- > "minimist": "1.2.8",
$ ./ci.sh 1.5.0-ci.2
== 1 build localhost:5000/lab-build/app:1.5.0-ci.2 sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe built sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe == 2 test: start the image, call it, stop it test passed == 3 scan: fail on HIGH or CRITICAL with a fix available Report Summary ┌─────────────────────────────────────────────────────────┬──────────┬─────────────────┬─────────┐ │ Target │ Type │ Vulnerabilities │ Secrets │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ localhost:5000/lab-build/app:1.5.0-ci.2 (alpine 3.24.2) │ alpine │ 0 │ - │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ app/node_modules/minimist/package.json │ node-pkg │ 0 │ - │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ app/node_modules/ms/package.json │ node-pkg │ 0 │ - │ ├─────────────────────────────────────────────────────────┼──────────┼─────────────────┼─────────┤ │ app/package.json │ node-pkg │ 0 │ - │ └─────────────────────────────────────────────────────────┴──────────┴─────────────────┴─────────┘ ... scan passed == 4 push the image that was tested and scanned localhost:5000/lab-build/app:1.5.0-ci.2 == 5 resolve the digest sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe == 6 attest: SBOM (signed) next to BuildKit's provenance Using payload from: sbom.spdx.json Signing artifact... == 7 sign the digest Signing artifact... Pushing signature to: localhost:5000/lab-build/app == 8 verify before anyone may promote it VERIFIED localhost:5000/lab-build/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe

Every stage passed. The digest printed by the build (built sha256:...) and the digest the registry returned after the push are the same value, and the script wrote it to release-1.5.0-ci.2.digest, the record the rest of the release uses. Digests on your machine differ: they hash content that includes build timestamps. The provenance attestation inside the pushed index records what the build used:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ docker buildx imagetools inspect localhost:5000/lab-build/app@$(cat release-1.5.0-ci.2.digest) \ --format '{{json .Provenance}}' | jq -r '.SLSA.buildDefinition.resolvedDependencies[].uri'
pkg:docker/node@24-alpine?digest=sha256:ebfe2f90462722a7a4de65e91990e97fe0d401c70e0e762c5b53302f905ec1c1&platform=linux%2Farm64

The base image by digest and platform. Because the signature covers the index digest, and the index lists the attestation manifest, the provenance is covered by the same signature. The SBOM is attached separately with cosign attest, signed with the same key, because many policy tools look for a signed SBOM attestation.

Promotion

promote.sh
#!/bin/sh
# Usage: ./promote.sh SOURCE_REPO@DIGEST TARGET_REPO:TAG
# Promotion job: verify the signature and the SBOM attestation on the exact digest, copy the image by
# digest and its signature artifacts (tag sha256-<hex>), verify both again in the target, and only then
# point the release tag at it. Being in the build registry is not an approval; only a digest that
# verifies is promoted, and a failed copy never leaves the release tag on an unverified image.
set -eu
src=$1 target=$2
VERIFY="--key cosign.pub --insecure-ignore-tlog=true"
case $src in *@sha256:*) ;; *) echo "REFUSED $src: promote a digest, not a tag"; exit 1 ;; esac
digest=${src#*@}; srcrepo=${src%@*}; dstrepo=${target%:*}
verified() { # verified REPO@DIGEST: the signature and the SBOM attestation both verify
cosign verify $VERIFY "$1" >/dev/null 2>&1 &&
cosign verify-attestation $VERIFY --type spdxjson "$1" >/dev/null 2>&1
}
verified "$src" || { echo "REFUSED $src: no valid signature and SBOM attestation"; exit 1; }
# On registries without the OCI referrers API (registry:3 here) Cosign keeps the signature and the
# attestation under this tag. A registry with that API has no such tag; copy with `oras cp -r` there.
sigtag=sha256-${digest#sha256:}
docker buildx imagetools inspect "$srcrepo:$sigtag" >/dev/null 2>&1 ||
{ echo "REFUSED $src: no $sigtag tag to copy"; exit 1; }
docker buildx imagetools create --progress=quiet --tag "$dstrepo@$digest" "$src"
docker buildx imagetools create --progress=quiet --tag "$dstrepo:$sigtag" "$srcrepo:$sigtag"
verified "$dstrepo@$digest" || { echo "FAILED $dstrepo@$digest does not verify in the target"; exit 1; }
docker buildx imagetools create --progress=quiet --tag "$target" "$dstrepo@$digest"
echo "PROMOTED $dstrepo@$digest as $target"

The promotion job takes a digest, never a tag. It verifies the signature and the SBOM attestation, then copies the image registry-side by digest with docker buildx imagetools create, which moves no tag, and copies the signature artifacts too. That second copy is necessary: imagetools create copies the image index and what it references, and Cosign's signature and attestation live in a separate artifact, tagged sha256-<hex of the digest> in this registry. It verifies both again in the target, and only then points the release tag at the digest. If a copy or the second verification fails, the release tag still names the previous release. Build to staging, then staging to production:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ ./promote.sh localhost:5000/lab-build/app@$(cat release-1.5.0-ci.2.digest) localhost:5000/lab-staging/app:1.5.0
PROMOTED localhost:5000/lab-staging/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe as localhost:5000/lab-staging/app:1.5.0
$ ./promote.sh localhost:5000/lab-staging/app@$(cat release-1.5.0-ci.2.digest) localhost:5001/lab-prod/app:1.5.0 curl -s http://localhost:5001/v2/lab-prod/app/tags/list; echo
PROMOTED localhost:5001/lab-prod/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe as localhost:5001/lab-prod/app:1.5.0 {"name":"lab-prod/app","tags":["1.5.0","sha256-2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe"]}
$ docker run -d --name lab-ci-prod -p 127.0.0.1:3901:3000 \ localhost:5001/lab-prod/app@$(cat release-1.5.0-ci.2.digest) >/dev/null sleep 2; curl -s http://127.0.0.1:3901/
Unable to find image 'localhost:5001/lab-prod/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe' locally localhost:5001/lab-prod/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe: Pulling from lab-prod/app Digest: sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe Status: Downloaded newer image for localhost:5001/lab-prod/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe {"service":"payments","uptime":"2s"}

Production holds the release tag and the signature tag, and the container runs from the production registry by digest. Deploy by digest, not by tag, so a later push to the tag cannot change what is running.

The sha256-<hex> tag is the fallback that Cosign 3 uses on registries without the OCI 1.1 referrers API, such as registry:3. On a registry that has the API, the signature and the attestation are stored as referrers of the digest and no such tag exists. In a test with zot 2.1, a registry with that API, Cosign 3.1.3 created no sha256- tag, so promote.sh stops at its tag check before it copies anything. For such registries, copy the image together with its referrers using oras cp -r (ORAS 1.3.4 copied them between zot and registry:3 in both directions in the same test, and cosign verify passed on the copies). cosign copy is deprecated in Cosign 3 and did not carry the new signature format.

Now the bypasses. A tag is refused outright, because it could point anywhere by the time the copy runs. A hotfix built on someone's machine and pushed straight into the build repository is refused because nothing signed it:

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ ./promote.sh localhost:5000/lab-build/app:1.5.0-ci.2 localhost:5001/lab-prod/app:1.5.0
REFUSED localhost:5000/lab-build/app:1.5.0-ci.2: promote a digest, not a tag
$ docker build -q --build-arg HOTFIX=1 -t localhost:5000/lab-build/app:hotfix app docker push -q localhost:5000/lab-build/app:hotfix ./promote.sh localhost:5000/lab-build/app@$(docker buildx imagetools inspect localhost:5000/lab-build/app:hotfix --format '{{.Manifest.Digest}}') \ localhost:5001/lab-prod/app:1.5.0
sha256:565f29d7038eb4ad2b88a6e88c46c4d953e41be708723ef20c0539d2e9f55f6d localhost:5000/lab-build/app:hotfix REFUSED localhost:5000/lab-build/app@sha256:565f29d7038eb4ad2b88a6e88c46c4d953e41be708723ef20c0539d2e9f55f6d: no valid signature and SBOM attestation
$ docker buildx imagetools inspect localhost:5001/lab-prod/app:1.5.0 --format '{{.Manifest.Digest}}'
sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe

The hotfix push succeeded; the registry accepts anything from someone with push rights. Promotion did not, and production still points at the signed digest. Because promotion checks the signature, the build registry can be shared without being trusted; an image being there proves nothing.

ubuntu@secopslog-docker:~/lab/dockerci · Docker 29.8.2
$ docker rm -f lab-ci-prod docker rm -f -v lab-registry lab-prod-registry docker image rm localhost:5000/lab-build/app:1.5.0-ci.1 localhost:5000/lab-build/app:1.5.0-ci.2 \ localhost:5000/lab-build/app:hotfix localhost:5001/lab-prod/app@$(cat release-1.5.0-ci.2.digest) rm -f cosign.key cosign.pub lab-signing-config.json build.json sbom.spdx.json release-*.digest rm -rf ~/lab/bin ~/.cache/trivy ~/.sigstore
... Untagged: localhost:5001/lab-prod/app@sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe Deleted: sha256:2a96468035986e727c89e3915707d4b28b102b7bcce20733ef6b11fa2bee8fbe

The same pipeline in GitHub Actions

The workflow below was linted, not run: the lab has no GitHub runner, and keyless signing needs a GitHub OIDC identity, so that part is described and not demonstrated. It shows the build job and the production promotion; a staging promotion is the same job with the staging repository and an environment of its own.

.github/workflows/release.yml
name: release
on:
push:
branches: [main]
permissions: {}
env:
BUILD_REPO: ghcr.io/acme/payments-api/build
PROD_REPO: registry.example.com/prod/payments-api
# The exact identity a signature must carry: this workflow file on main.
SIGNER: https://github.com/acme/payments-api/.github/workflows/release.yml@refs/heads/main
ISSUER: https://token.actions.githubusercontent.com
jobs:
build:
runs-on: ubuntu-24.04
permissions:
contents: read
packages: write # push to ghcr.io
id-token: write # OIDC token for keyless signing
outputs:
digest: ${{ steps.digest.outputs.digest }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
# Same Engine as the lab, with the containerd image store so --load keeps the index.
- uses: docker/setup-docker-action@2bf61fb9464cc67f0cbdeabed6aa0380accd1c70 # v5.5.0
with:
version: v29.8.2
daemon-config: '{"features": {"containerd-snapshotter": true}}'
- uses: docker/setup-buildx-action@f87e5991a6d7451dcb8d9637bfbc97413f497069 # v4.4.1
with:
driver: docker
- name: Install cosign, syft and trivy (pinned, checksum-verified)
run: BIN="$RUNNER_TEMP/bin" ./install-tools.sh cosign syft trivy && echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
- name: 1 build
id: build
uses: docker/build-push-action@c3c9e263c25d99ce0380d002d59b67737d91b0dc # v7.4.0
with:
context: app
load: true
provenance: mode=max
tags: ${{ env.BUILD_REPO }}:${{ github.sha }}
- name: 2 test
run: |
docker run -d --name test -p 127.0.0.1:3900:3000 "$BUILD_REPO:$GITHUB_SHA"
for _ in $(seq 1 30); do
[ "$(docker inspect -f '{{.State.Health.Status}}' test)" = healthy ] && break; sleep 1
done
curl -fsS http://127.0.0.1:3900/ | grep -q '"service":"payments"'
docker rm -f test
- name: 3 scan
run: trivy image -q --severity HIGH,CRITICAL --ignore-unfixed --exit-code 1 "$BUILD_REPO:$GITHUB_SHA"
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- name: 4 push and 5 resolve the digest
id: digest
env:
BUILT: ${{ steps.build.outputs.digest }}
run: |
docker push -q "$BUILD_REPO:$GITHUB_SHA"
d=$(docker buildx imagetools inspect "$BUILD_REPO:$GITHUB_SHA" --format '{{.Manifest.Digest}}')
[ "$d" = "$BUILT" ] || { echo "pushed $d, built $BUILT"; exit 1; }
echo "digest=$d" >> "$GITHUB_OUTPUT"
- name: 6 attest, 7 sign (keyless), 8 verify
env:
DIGEST: ${{ steps.digest.outputs.digest }}
run: |
IMAGE="$BUILD_REPO@$DIGEST"
syft scan -q "registry:$IMAGE" -o spdx-json=sbom.spdx.json
cosign attest --yes --type spdxjson --predicate sbom.spdx.json "$IMAGE"
cosign sign --yes "$IMAGE"
cosign verify --certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$IMAGE" > /dev/null
cosign verify-attestation --type spdxjson \
--certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$IMAGE" > /dev/null
promote:
needs: build
runs-on: ubuntu-24.04
# The production environment has required reviewers and allows only the main branch. Its own
# secrets (PROD_REGISTRY_USER/TOKEN) exist only for jobs that run in it, after approval.
environment: production
permissions:
contents: read
packages: read
env:
DIGEST: ${{ needs.build.outputs.digest }}
steps:
- uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7.0.1
- name: Install cosign (pinned, checksum-verified)
run: BIN="$RUNNER_TEMP/bin" ./install-tools.sh cosign && echo "$RUNNER_TEMP/bin" >> "$GITHUB_PATH"
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: ghcr.io
username: ${{ github.actor }}
password: ${{ secrets.GITHUB_TOKEN }}
- uses: docker/login-action@dbcb813823bdd20940b903addbd779551569679f # v4.6.0
with:
registry: registry.example.com
username: ${{ secrets.PROD_REGISTRY_USER }}
password: ${{ secrets.PROD_REGISTRY_TOKEN }}
- name: Verify, then promote by digest
run: |
SRC="$BUILD_REPO@$DIGEST"
cosign verify --certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$SRC" > /dev/null
cosign verify-attestation --type spdxjson \
--certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$SRC" > /dev/null
# Copy by digest, then the signature tag, verify in production, and only then tag.
# (A registry with the OCI referrers API has no sha256-<hex> tag: use `oras cp -r` there.)
sigtag="sha256-${DIGEST#sha256:}"
docker buildx imagetools create --tag "$PROD_REPO@$DIGEST" "$SRC"
docker buildx imagetools create --tag "$PROD_REPO:$sigtag" "$BUILD_REPO:$sigtag"
cosign verify --certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$PROD_REPO@$DIGEST" > /dev/null
cosign verify-attestation --type spdxjson \
--certificate-identity "$SIGNER" --certificate-oidc-issuer "$ISSUER" "$PROD_REPO@$DIGEST" > /dev/null
docker buildx imagetools create --tag "$PROD_REPO:${GITHUB_SHA::12}" "$PROD_REPO@$DIGEST"

Every third-party action is pinned to a full commit SHA, with the release in a comment for humans and for Dependabot, which updates both. A tag such as @v4 can be moved by whoever controls the repository, or by whoever steals their credentials: in March 2026, attackers force-pushed 76 of 77 tags of aquasecurity/trivy-action to commits that exfiltrated CI secrets (CVE-2026-33634). This workflow does not use a scanner action at all; it installs Cosign, Syft and Trivy with the same install-tools.sh as the lab, against the hashes committed in the repository, so CI and the lab run identical binaries.

docker/setup-docker-action gives the job Docker Engine 29.8.2 with the containerd image store, so load: true keeps the provenance index exactly as in the lab, and setup-buildx-action with the docker driver builds inside that engine. build-push-action builds once with load: true, and its digest output is compared with what the registry reports after docker push. The job grants itself only what it needs: packages: write to push to GHCR and id-token: write to request an OIDC token.

That token is what keyless signing uses. cosign sign --yes IMAGE@DIGEST with no key obtains a token from GitHub that names the repository, workflow file and ref; Sigstore's Fulcio issues a short-lived certificate for that identity; the signature and certificate are recorded in the Rekor transparency log. No long-lived signing key exists to leak. Verification must then pin the exact identity: --certificate-identity with the workflow file and ref, and the GitHub issuer. A regular expression such as ^https://github.com/acme/ accepts a signature from any workflow on any branch of any repository under that prefix, including a pull-request workflow an attacker can edit. The promote job runs in a GitHub environment with required reviewers, verifies that identity, and only then copies the digest and its signature to the production registry.

The reviewers protect production only if the production credentials are out of reach of every other job. Store PROD_REGISTRY_USER and PROD_REGISTRY_TOKEN as secrets of the production environment, not of the repository, and restrict the environment's deployment branches to main. A repository secret is readable by any workflow on any branch that someone with write access pushes, which could push to production without the promote job. Where the registry supports it, replace the token with OIDC federation, so that no static production credential exists.

A scan at build time describes the image on the day it was built. New CVEs are published against images that are already in production, so run a scheduled job that rescans the promoted digests (the signed SBOMs make that cheap: trivy sbom scans an SBOM file without pulling the image) and opens a rebuild when a fix appears. Shared runners also download the Trivy vulnerability database on every job; cache it between runs or mirror it, since the download is rate-limited.

Builders for untrusted code

A build runs the code it builds: every RUN line, every package install script. For pull requests from forks, treat the build as hostile.

Do not mount /var/run/docker.sock into a job, and do not run jobs on a self-hosted runner whose user is in the docker group. Access to the socket is root on the runner, read-only mount or not, and a job with root on a shared runner can read other jobs' secrets and replace the images they push. "The Docker socket and daemon hardening" (Advanced container security) explains why and what to use instead. Docker-in-Docker with --privileged is no better: a privileged container is root on the host.

Buildx itself is a client and isolates nothing. With the docker driver the build runs inside dockerd; with the docker-container driver it runs in a BuildKit container that is privileged by default ("Buildx builders, outputs, cache and Bake"). Options that do isolate: GitHub-hosted runners or other ephemeral VMs that are destroyed after each job; rootless BuildKit (the moby/buildkit:rootless image), which runs without root but needs relaxed seccomp and AppArmor settings for its user namespaces; remote builders reached with the remote or kubernetes Buildx drivers, so the runner holds no daemon at all; or rootless Docker on the runner ("Rootless Docker" in Advanced container security). Kaniko, the usual answer for years, was archived by Google in June 2025; a community fork is maintained by Chainguard. Whatever builds untrusted code must also hold no registry credentials and no signing identity: GitHub withholds secrets from fork pull requests, and pull_request_target workflows that check out the pull request's code undo that protection.

Next: Advanced container security. The Docker in depth course ends here. The advanced course starts from the kernel side: what namespaces, cgroups and capabilities actually enforce, how to harden the image, the container and the daemon, and how escapes work and are stopped.

Quick check
01A workflow builds with load: true, scans the local image with Trivy, then runs build-push-action again with push: true, and signs steps.push.outputs.digest. What is wrong?
Incorrect — Cosign signs any manifest digest in a registry; the output is fine to use.
Correct — The second build is a new artifact. Build once, push that image, and check that the pushed digest equals the build's.
Incorrect — A signature is over a digest that already exists in a registry, so signing comes after the push.
Incorrect — Trivy scans local engine images, tarballs and registry references.
02A developer pushes a hotfix image directly into the build repository, where the CI job also pushes. The promotion job is given the hotfix's digest. What should stop it reaching production?
Incorrect — A registry accepts any push from someone with push rights; the lab's hotfix push succeeded.
Incorrect — A clean scan says nothing about who built the image or whether it went through the pipeline.
Incorrect — A digest identifies content exactly, including content nobody approved.
Correct — Nothing signed the hotfix, so verification fails and the copy never runs.
03Production verification uses --certificate-identity-regexp '^https://github.com/acme/api/' with the GitHub OIDC issuer. Why is that weaker than --certificate-identity https://github.com/acme/api/.github/workflows/release.yml@refs/heads/main?
Correct — Pin the exact workflow file and ref that is allowed to sign releases.
Incorrect — Both forms check the log; they differ in which identities they accept.
Incorrect — They are supported, which is why the loose form is a real risk.
Incorrect — A signature proves who signed which digest, not what the image contains.

Try this

Work through “Builders for untrusted code” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from docker in ci/cd: build to promote, keep “Builders for untrusted code”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related