Tags, digests and promotion

Mutable tags, immutable digests and promoting by digest.

QA signs off on app:1.0 in staging at two in the afternoon. At three, a developer rebuilds a quick fix and pushes it under the same tag, because the version number "has not shipped yet". At four, the release job copies app:1.0 from staging to production. Production now runs an image nobody tested, and every dashboard says 1.0. Nothing in that story is a bug in Docker. A tag is a name that anyone with push rights can move; the fix is to record and promote the digest, which cannot move. This lesson reproduces the incident with a local registry, then does the promotion properly. It runs on the main lab VM, secopslog-docker, with the lesson files (a Dockerfile and a small check script) in ~/lab/tagsdigests.

Pinning the base image

Start where most images start, the FROM line. The lab Dockerfile pins alpine by tag and digest:

# alpine:3.22, pinned to the index digest the tag pointed at when this lab was written
FROM alpine:3.22@sha256:5291449c3df73caf6ed85e649dec1b9e818b39a5d8c871e97afc13e9cd5e8fa8
ARG BUILD=1
RUN echo "lab-app build $BUILD" > /build.txt
CMD ["cat", "/build.txt"]

In name:tag@sha256:... the digest decides what is used and the tag is only there for people reading the file. Compare the pin with what the tag points at today:

--format '{{.Manifest.Digest}}' prints the digest of whatever the tag resolves to, which for alpine is its index ("Image anatomy: index, manifest, config and layers" explains the index). When this lab ran, the two values matched. The next time the Alpine maintainers rebuild 3.22 for a security fix, the tag moves and the lines differ, while builds from this Dockerfile keep using the pinned bytes until someone changes the pin on purpose. Moving pins forward on a schedule with Renovate or Dependabot is covered in "Pinning, SBOMs, provenance and scanning" in Advanced container security.

Pin the index digest, the one the tag resolves to. Every builder then picks its own platform from the index, so the same line works on an amd64 laptop and an arm64 CI runner. The digest of one platform's manifest (an entry inside the index) pins that architecture only; an arm64 builder given an amd64 manifest digest builds for amd64 or fails. Use a platform digest only when that restriction is intended, and write the reason next to it.

One tag, two builds

Start a registry:3 container on a lab network, published on the VM's loopback address. Docker trusts localhost registries over plain HTTP, so no TLS setup is needed for the lab:

Build the image, push it to the staging repository, and record its digest. This is the build QA tests:

The push ends with 1.0: digest: sha256:..., and the lookup prints the same value: the index digest, because BuildKit attached a provenance attestation and made the result an index. tee keeps it in tested.digest, playing the part of the release record. Now the afternoon fix arrives under the same tag:

Same tag, new digest. The registry accepted the overwrite without complaint, and only the alpine layer and the empty attestation config were reused. Anyone who pulls 1.0 from now on gets build 2, and anyone holding the digest still gets build 1:

The third command uses name:tag@digest. The tag says 1.0, which is build 2, and the container still prints build 1: when a reference carries a digest, Docker resolves the digest and ignores the tag. That is how Compose files and Kubernetes manifests should name production images. Most of the layers were already present, because build 1's content was still in the local content store and only the tag had moved off it, so the pull finished almost at once.

Image ID, RepoDigests and platform digests

Deploy scripts often compare "the image we meant" with "the image we got". Know which fields hold what:

On the containerd image store, the default on fresh Docker 29 installs, the image ID is the index digest, so the ID and the registry digest agree, as here. On hosts still using the legacy graph-driver store the ID is the digest of the image config, which no registry reports ("Inspecting, exporting and cleaning up images" shows it). RepoDigests is the portable field: repository@digest for every registry this image is known to have come from, on both stores. A running container records the ID in docker inspect --format '{{.Image}}' CONTAINER. Compare digests from the registry or from RepoDigests, not raw IDs, if your fleet mixes old and new hosts.

The digest you recorded names an index. Inside it are the platform manifest and its attestation:

On this arm64 VM the only runnable entry is linux/arm64; built on amd64 it would be linux/amd64, and a multi-platform build ("Multi-platform images") lists both. Record and promote the index digest, never the inner manifest digest, or the attestation and any other platforms are left behind.

Promotion by digest

The lab registry has two repositories standing in for two environments: lab-staging/app and lab-prod/app. In a real setup they are usually separate registries or projects with separate credentials. Promotion copies the tested digest to production; it never rebuilds. A rebuild from the same commit produces a different digest anyway, because the config records the build time, so "the same source" is not "the same image". docker buildx imagetools create copies an image between repositories registry-side, without pulling it into the local store:

The two copying lines are the manifests the index lists, the arm64 image and its attestation, each with the blobs it references; the last line pushes the index itself under the new tag. The image never passed through the local store. The check script turns "is production running what we tested" into an exit status a pipeline can act on:

#!/bin/sh
# Usage: ./check-digest.sh IMAGE:TAG DIGEST_FILE
# Exits 0 only when IMAGE:TAG currently resolves to the digest recorded in DIGEST_FILE.
want=$(cat "$2")
have=$(docker buildx imagetools inspect "$1" --format '{{.Manifest.Digest}}')
if [ "$have" = "$want" ]; then
  echo "OK    $1 -> $have"
else
  echo "FAIL  $1 -> $have"
  echo "      tested image is $want"
  exit 1
fi

A reachable registry is not an approval

Promotion by digest only protects you if it is the only way into production. The registry has no idea which images were tested. Anyone who can reach it with push rights can overwrite the production tag, here with the untested build 2:

The push succeeded and production silently changed; only the check noticed. The lesson is the order of controls, not the script. Give production push rights to the promotion job alone, turn on immutable tags where the registry offers them (Harbor, Amazon ECR, Azure Container Registry and Google Artifact Registry do; registry:3 does not), deploy by digest so a moved tag cannot change a running service, and verify a signature made over the tested digest before anything runs ("Signing, verifying and promoting by digest" in Advanced container security, and the full pipeline in "Docker in CI/CD: build to promote"). Network access to a registry is a transport, not a decision.

Put production back on the tested image. The plain docker tag and docker push route works too, as long as the source is the digest and not the tag:

Nothing was uploaded, since the repository already had every blob, and the pushed digest equals the tested one, attestation included. With the containerd store, pushing a pulled image keeps its index intact; check the printed digest anyway. A host on the legacy store keeps only the platform image, not the index, so a re-push from there cannot reproduce the original digest; imagetools create avoids the question by never going through the local store. Clean up:

Try this

Work through “A reachable registry is not an approval” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from tags, digests and promotion, keep “A reachable registry is not an approval”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related