Signing, verifying and promoting by digest

Cosign v3 key-based and keyless signatures, verify-before-run, SBOM attestations and promotion by digest.

A production host deploys with docker pull registry.example.com/app:1.4 && docker run .... Nothing in that command can tell whether app:1.4 is the image the pipeline built, tested and scanned, or one somebody pushed with a leaked registry token an hour ago. The registry accepted both pushes without asking. A registry stores images; it does not approve them. Approval has to be something a deploy can check, such as a signature over the image digest, made after the image passed its tests by a key or identity the deploy trusts.

This lesson runs the whole release order end to end with Cosign v3 and two local registries, proves that a verify-before-run script refuses an unsigned image and a modified one, promotes the signed digest, and verifies real keyless signatures from public projects. Use the main lab VM. The lesson files go in ~/lab/signing: app/Dockerfile, install-tools.sh (from "Pinning, SBOMs, provenance and scanning") and verify-run.sh. The lab needs internet access to GitHub, gcr.io and Sigstore's public services.

The release order

Tests and scans run on the candidate before anything leaves the build machine, so a failing build never reaches a registry where someone could deploy it. After the push the registry holds the image under the digest it will be known by everywhere, and every later step uses that digest: attestations and the signature are bound to it, verification checks it and promotion copies it. Signing before the push, or signing a tag, signs whatever the name resolves to at that moment, which is not necessarily what was tested.

Use separate repositories for separate trust levels. The build registry receives every CI build and keeps it for a short time, and staging deploys from it by digest. Production pulls only from a registry (or repository) that the promotion job alone can write to, and every production host verifies before it runs. In the lab, localhost:5000 is the build registry and localhost:5001 is production. "Docker in CI/CD: build to promote" in Docker in depth puts this order into a CI workflow.

Verify, then run

Docker Engine runs any image it is asked to run. It has no step that checks a signature before docker run, so on plain Docker hosts verification is a script in front of the run. This one resolves the reference to a digest once, verifies a Cosign signature on that digest and runs exactly that digest:

#!/bin/sh
# Usage: ./verify-run.sh IMAGE[:TAG|@DIGEST] [COMMAND...]
# Resolves IMAGE to one digest, verifies a Cosign signature on that digest, and only then runs
# that digest. Exits 1 without running anything when verification fails.
# COSIGN_VERIFY is the trust policy, and there is no default: an unset policy is an error, so a copy
# of this script can never fall back to a weaker one. Examples:
#   keyless:  COSIGN_VERIFY="--certificate-identity=URI --certificate-oidc-issuer=URL"
#   key:      COSIGN_VERIFY="--key /etc/cosign/release.pub"   (an absolute path, never the working directory)
#   this lab: COSIGN_VERIFY="--key $HOME/lab/signing/cosign.pub --insecure-ignore-tlog=true"
set -eu
: "${COSIGN_VERIFY:?set COSIGN_VERIFY to the trust policy (see the comments in this script)}"
ref=$1; shift
name=${ref%%@*}
case ${name##*/} in *:*) name=${name%:*} ;; esac
digest=$(docker buildx imagetools inspect "$ref" --format '{{.Manifest.Digest}}')
image=$name@$digest
# COSIGN_VERIFY is a list of flags, so it is expanded unquoted on purpose.
if ! err=$(cosign verify $COSIGN_VERIFY "$image" 2>&1 >/dev/null); then
  echo "REFUSED  $image"
  printf '%s\n' "$err" | grep -m1 '^Error' | sed 's/^/         /'
  exit 1
fi
echo "VERIFIED $image"
exec docker run --rm "$image" "$@"

Running the digest instead of the name closes the gap between check and use: a tag can move between cosign verify and docker run, a digest cannot, and Docker checks pulled content against the digest. COSIGN_VERIFY is the trust policy, and the script has no default: with the variable unset it stops with an error, so a copy on another host can never fall back to a weaker policy. For keyless signatures the policy names a certificate identity and issuer; for a key it names the public key by an absolute path, so a cosign.pub in whatever directory the script runs from is never trusted by accident. The lab sets a policy for its own key in the next section, and that policy skips the transparency log because the lab signs without one.

Tools, registries and a key pair

Install Cosign, Syft and Trivy with the same checksum-verified script as in the previous lesson:

If the connection to GitHub is reset during a download, curl prints a curl: (35) line and --retry in the script tries again. A file that still arrived damaged would fail its checksum line.

Start the build registry on port 5000 and the production registry on port 5001. Both are plain registry:3 containers; what makes one of them production is who may push to it.

A key pair is the simplest trust anchor: whoever holds the private key can sign, and anyone with the public key can verify. The export COSIGN_VERIFY=... line in the next command sets the trust policy verify-run.sh uses for the rest of the lesson; run it again in any new shell, together with the PATH and COSIGN_PASSWORD exports. cosign generate-key-pair encrypts the private key with the password in COSIGN_PASSWORD, here an obvious test value. In CI the password comes from the secret store, or the key lives in a KMS and never touches disk (--key awskms://..., gcpkms://, azurekms://, hashivault://).

Cosign v3 takes the Sigstore services it uses (Fulcio, Rekor, a timestamp authority) from a signing config. The default comes from Sigstore's TUF repository and uploads even key-based signatures to the public Rekor transparency log. lab-signing-config.json lists no services, so the lab publishes nothing about throwaway keys and a localhost registry. The price is that verification must skip the log (--insecure-ignore-tlog=true) and nothing independent records when a signature was made. In production, sign keyless (below) or keep a log, public or a private Rekor. The older --tlog-upload=false is deprecated, and v3 refuses it while a signing config is in use.

Build, test and scan the candidate

# alpine:3.22, pinned to the index digest the tag pointed at when this lab was written
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
ARG BUILD=1
RUN echo "lab-app build $BUILD" > /build.txt
USER 10001
CMD ["cat", "/build.txt"]

lab-app:candidate exists only in the local image store. The test compares the output with what this build must print, and the scan applies the gate from the previous lesson. Trivy downloads its database on this first run. The Alpine base has no fixable HIGH or CRITICAL findings today, so the gate passes; on another day it may not, and then the release stops here.

Push and resolve the digest

The 1.0: digest: line of the push output is the digest the registry stored for the tag, the index digest on Docker 29, and sed saves it in app.digest. Do not ask the registry what 1.0 points to after the push: another job could push the same tag in between, and you would sign its image. With docker buildx build --push the digest comes from --metadata-file, and docker/build-push-action returns it as the digest output.

Attest and sign the digest

An attestation is a signed statement about the image: an in-toto statement whose subject is the digest and whose predicate is a document, here the SBOM. BuildKit's attestations from the previous lesson sit inside the image index and are not signed; cosign attest signs one with your key, so a verifier knows who vouches for it. Generate the SBOM from the pushed digest, attest it, then sign:

Cosign v3 stores each signature and attestation as a Sigstore bundle, an OCI artifact that refers to the image digest. A registry with the OCI 1.1 referrers API returns such artifacts when asked for the referrers of a digest. registry:3 lacks that API, so Cosign used the fallback the OCI specification defines: a tag named after the digest, sha256-<hex>, whose index lists both bundles, the signature (cosign/sign/v1) and the SBOM attestation (spdx.dev/Document). The image index and its digest are unchanged.

Verify

cosign verify checked each signature against cosign.pub and that the signed claims name this digest. It lists every bundle this key signed for the digest, so the attestation appears next to the signature; type tells them apart. The summary says the log was "verified offline" although the warning above says it was skipped; trust the warning. verify-attestation returns the signed statement, which decodes to the SBOM for this digest. A deploy policy that requires an attestation of a given type (an SBOM, a scan result, a VEX document) is built on this command.

Now the script. It runs the signed image:

Someone builds a "hotfix" and pushes it straight to the production registry, bypassing the pipeline. The push works, because the registry checks credentials, not provenance. The script does not run it:

A second case: someone rebuilds the image, pushes it over 1.0 in the build registry and signs it with a key of their own, so the tag now points at a signed image:

The tag resolved to the new digest, whose only signature was made with other.key. Cosign v3 says "no matching attestations" for signature bundles too; the important part is "Found: 0, Expected 1", no signature from the trusted key. A check for "any valid signature" would have passed, which is why a signature means nothing without the key or identity you require. The tested image is still reachable by the recorded digest, and that still verifies:

Docker had no local image under that digest reference, so it pulled one; every layer was already in the local store, and the content was checked against the digest.

Promote by digest

Promotion copies the verified digest from the build registry to production with imagetools create, as in "Tags, digests and promotion" in Docker in depth. Then verify in production:

The image arrived with the same digest, but the signature did not. imagetools create copies the index and everything it references; the signature bundles reference the image, not the other way round, so nothing in the index leads to them. Copy them as well. With the tag fallback they are one more tag to copy:

The signature covers the digest, not a registry name, so it verifies wherever the digest and its signature artifacts are. Registries with the referrers API have no fallback tag to copy; use a tool that copies referrers with the image, such as oras copy -r or regctl image copy --referrers (cosign copy is deprecated in v3, and its help points to oras copy -r). The other common pattern is to sign again after promotion with a production key that only the promotion job holds, so the second signature means "approved for production" rather than "built by CI".

Keyless signing

Keys have to be stored, rotated and revoked. Keyless signing replaces the key with an identity. The CI job requests an OpenID Connect (OIDC) token from its platform, a signed statement such as "workflow release.yaml in repository org/app, running for tag v1.4". Cosign generates a throwaway key pair, and Fulcio, Sigstore's certificate authority, issues a ten-minute certificate binding that public key to the identity. Cosign signs the digest, Rekor records the signature with a timestamp in a public append-only log, and the private key is discarded. A verifier checks that the certificate chains to Fulcio's root (distributed through TUF), that the signature was logged while the certificate was valid, and that identity and issuer match what it expects. Public Fulcio accepts tokens only from configured issuers (GitHub Actions, GitLab.com, Google and others); a self-hosted GitLab needs a private Sigstore deployment.

Keyless signing needs a real OIDC identity, so the lab verifies signatures that public projects made. Google's distroless images are signed with a Google service account identity, documented in the distroless README:

Verification is by digest again. Subject and issuer come from the Fulcio certificate, and the Rekor entry shows when the signature was logged. With a different expected identity the same signature is rejected, and the error names the identity it found. Pin the identity exactly: a loose --certificate-identity-regexp '.*' accepts a signature from anyone the issuer will vouch for, which for GitHub Actions means any workflow in any repository. If you need a pattern, anchor it (^https://github\.com/org/app/\.github/workflows/release\.yaml@refs/tags/v).

Signatures from GitHub Actions carry the workflow file and Git ref as identity. Trivy's release checksums are signed that way, which also completes the install from the previous lesson: the checksum file the hashes in install-tools.sh were taken from can itself be verified:

The identity says which workflow on which ref signed, not that the code was benign. In the March 2026 Trivy compromise the attacker pushed a commit that altered the release workflow and tagged it v0.69.4, and the project's own release pipeline built and published the malicious release from that tag. Protected tags and branches and review of workflow changes decide whether a ref deserves trust. In CI, signing is cosign sign --yes IMAGE@DIGEST with no key, plus permission to request an OIDC token (id-token: write on GitHub Actions); "Docker in CI/CD: build to promote" in Docker in depth shows the workflow.

Where verification can be enforced

Docker Engine itself never enforces a signature. Docker Content Trust, the Notary v1 mechanism behind DOCKER_CONTENT_TRUST=1 and docker trust, was removed from the Docker CLI in 29.0:

The variable is silently ignored and the pull succeeds. On Docker hosts the enforcement points are a wrapper like verify-run.sh, a deploy job that verifies before it calls Docker, and a production registry that only a verifying promotion job can write to. Engine authorization plugins can veto API calls, but none that ships with Docker checks Sigstore signatures. Kubernetes has the step Docker lacks: an admission controller sees every pod before it is created, and Kyverno or Sigstore's policy-controller reject pods whose images lack a valid signature or attestation from the required identity.

Notation, from the CNCF Notary Project, is the other current signing tool. It signs digests with X.509 certificates from your own PKI (often a cloud key vault), stores signatures as OCI referrers and verifies against a trust policy file. Cosign fits teams that want keyless CI identities and a public log; Notation fits organisations that already run certificate authorities. Both sign digests, both need verification wired into the deploy path, and Kubernetes admission controllers support both.

Clean up

Remove the registries, the images (including the digest references the runs created), the keys and generated files, the tools and their caches:

Try this

Work through “Clean up” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from signing, verifying and promoting by digest, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related