Build-time secrets and how images leak them

ARG, ENV, history, cache exports and provenance, and the mount that avoids them.

A pull-request build log contains this line: #6 [build 3/4] RUN printf '//npm.example.test/:_authToken=%s\n' "npm_FAKE_lab_token_0000" > .npmrc. The Dockerfile never contained the token. It was passed with --build-arg, used only in a builder stage, and the final image is clean. The log is one of six places the value ended up anyway. This lesson finds all of them with a fake token, then shows the build where the same credential ends up in none.

Use the main lab VM. The lesson files go in ~/lab/multistage: three Dockerfiles, a find-token.sh helper and a BuildKit config for the cache section. How stages, --target and RUN --mount work is the subject of "BuildKit builds: stages, cache and mounts" in Docker in depth; this lesson assumes that syntax and asks a different question: once a value enters a build, where does it go? The token npm_FAKE_lab_token_0000 is not a credential for anything. Use a value like it whenever you test for leaks, never a real one.

A build argument in a builder stage

Dockerfile.arg is the pattern found in many repositories. The builder stage takes the registry token as an ARG, writes an .npmrc, fetches dependencies (a stand-in step here, so the lab needs no registry) and produces an artifact. The final stage copies only that artifact.

# Unsafe on purpose: shows where a token passed with --build-arg ends up.
FROM alpine:3.22 AS build
ARG NPM_TOKEN
WORKDIR /src
RUN printf '//npm.example.test/:_authToken=%s\n' "$NPM_TOKEN" > .npmrc
RUN test -s .npmrc && echo "dependencies fetched" && mkdir -p /out && echo "app v1" > /out/app.txt
 
FROM alpine:3.22
COPY --from=build /out/app.txt /app/app.txt
CMD ["cat", "/app/app.txt"]

#!/bin/sh
# find-token.sh IMAGE STRING: where does STRING appear in a local image?
img=$1 tok=$2
tmp=$(mktemp -d)
docker save "$img" | tar -x -C "$tmp"
echo "history lines: $(docker history --no-trunc --format '{{.CreatedBy}}' "$img" | grep -c -- "$tok")"
echo "config env:    $(docker image inspect --format '{{json .Config.Env}}' "$img" | grep -c -- "$tok")"
layers=0 other=0
for b in "$tmp"/blobs/sha256/*; do
  if gzip -t "$b" 2>/dev/null; then
    gzip -dc "$b" | grep -aq -- "$tok" && layers=$((layers + 1))
  else
    grep -aq -- "$tok" "$b" && other=$((other + 1))
  fi
done
echo "layer blobs:   $layers"
echo "json blobs:    $other"
rm -rf "$tmp"

find-token.sh checks the four places a local image can carry a string: the history (docker history --no-trunc), the environment in the image config, the layer blobs and the other JSON blobs (config, manifests, attestations) of a docker save export. Build the image and check it:

Step #6 prints the RUN command with $NPM_TOKEN already replaced by its value. Every CI system that stores build logs now stores the token, readable by anyone who can open the job. The build also ended with a SecretsUsedInArgOrEnv warning, cut here; the ENV section below shows that check in full.

The final image itself is clean: zero hits in history, config and blobs. Multi-stage did its job for the image you ship. Everything that follows is about what else the build produced.

The builder stage is one --target away

Teams build the builder stage on its own all the time, to debug a failing build or to run tests in CI. Build it the way they would:

The history has three hits. BuildKit records the ARG instruction with its value, and every RUN after it as RUN |1 NPM_TOKEN=... /bin/sh -c ...: the |1 means one build argument was in scope, followed by its value. That is how BuildKit keeps the cache key correct, and it is why an ARG is visible to anyone who can read the image. The layer blob hit is the .npmrc file itself; the extract loop pulls it out of the saved image with nothing but tar. A file written into a layer stays in that layer (the reason is in "Layers and the build cache" in Docker for beginners), and a later rm in another step would not change this result.

Build records keep the arguments

BuildKit also keeps a record of every build: its status, logs, materials and the request that started it. docker buildx history reads them:

The first line is the record the lookup found. BuildKit names a record after the context directory and the Dockerfile, so the awk pattern only matches when the files are in a directory called multistage; with NOT FOUND there, adjust the pattern to your directory name. The record holds the argument in clear text, together with the full log from the previous section. Anyone with access to the Docker daemon can read the records, and Docker Desktop shows them in its Builds view. BuildKit keeps them until its history limits drop old records; docker buildx history rm deletes them sooner.

ENV goes further

Dockerfile.env copies a token into the environment, a pattern often used to make a build value available at run time:

# Unsafe on purpose: ENV bakes the value into the image config.
FROM alpine:3.22
ARG API_TOKEN
ENV API_TOKEN=$API_TOKEN
RUN echo "configured"
CMD ["sh", "-c", "echo app started"]

Now the value is in the config of the final image (config env: 1), in its history and in every container started from it, where docker inspect, /proc/1/environ and every child process can read it. "Runtime secrets, done right" covers the run-time side. For the build side, run the checks on their own; --check evaluates the Dockerfile without building and exits non-zero when a rule fires:

SecretsUsedInArgOrEnv is one of BuildKit's build checks: it flags ARG and ENV names that look like credentials (token, password, secret, key and similar). Exit status 1 makes --check usable as a CI gate before the build step, and a # check=error=true line at the top of a Dockerfile turns check warnings into build failures for every build of that file. The rule looks at names, never at values, so a credential in an innocently named argument passes it. It is a net for the obvious cases.

Provenance attestations with mode=max

On Docker 29 every BuildKit build attaches a provenance attestation to the image ("Image anatomy: index, manifest, config and layers" in Docker in depth shows where it sits in the index). The attestation travels with the image to every registry. Its detail depends on the mode: min, the default, or max, which adds the full build request. Push the same build both ways to a local registry:3:

The default attestation has no trace of the token. The max attestation has four: the build arguments of the request (twice, the request and its root) and the environment of the two RUN operations in the build definition. This is the final image, the clean one from the first section, and anyone who can pull it can read its provenance. The last check shows the same attestation inside a local docker save export, so image tarballs carry it too.

Docker's documentation states it directly: mode=max exposes the values of build arguments, and secret mounts are never included. Two settings make this common in practice. Supply-chain guidance often recommends mode=max because it records more about the build. And the docker/build-push-action GitHub Action adds mode=max provenance by default when the repository is public. A pipeline that passes a token with --build-arg in a public repository publishes that token with every image. "Pinning, SBOMs, provenance and scanning" covers what provenance is for; the fix is never to turn it down, but to keep secrets out of build arguments.

Exported build cache

CI runners share build cache by exporting it, to a directory or to a registry reference. The default docker driver can export cache as long as the containerd image store is in use, which is the default on fresh Docker 29 installs, this VM included. This section uses a docker-container builder anyway, because that is what most CI setups run and it also works on hosts still on the legacy store ("Buildx builders, outputs, cache and Bake" in Docker in depth covers drivers and cache export). The builder reads buildkitd.toml, which pulls Docker Hub images through mirror.gcr.io and talks plain HTTP to the lab registry:

mode=min exports only the layers of the final result: four blobs, none with the token. mode=max exports the intermediate layers of every stage, so the builder's .npmrc layer is in it. The registry export is the same blob under a cache reference, and the loop above read it with plain HTTP requests, as anyone with pull access to that repository can. mode=max is what makes exported cache useful for multi-stage builds, so treat a file written in a builder stage as readable by everyone who can pull the cache.

Secret and ssh mounts

Dockerfile.secret does the same work with the two mounts BuildKit provides for credentials. The npm configuration is mounted from a file for one RUN; the ssh mount forwards the client's ssh agent socket, so a git clone over ssh can authenticate while the key stays in the agent. The syntax and options (env=, required, mode, --ssh default) are in "BuildKit builds: stages, cache and mounts" in Docker in depth.

FROM alpine:3.22 AS build
WORKDIR /src
RUN --mount=type=ssh \
    test -S "$SSH_AUTH_SOCK" && echo "ssh agent socket at $SSH_AUTH_SOCK"
RUN --mount=type=secret,id=npmrc,target=/src/.npmrc,required=true \
    test -s .npmrc && echo "dependencies fetched" && mkdir -p /out && echo "app v1" > /out/app.txt
 
FROM alpine:3.22
COPY --from=build /out/app.txt /app/app.txt
CMD ["cat", "/app/app.txt"]

Create the inputs: the npm configuration with the same fake token, and a throwaway ssh key. Then build the builder stage, the worst case from before, with mode=max provenance and a push, and export a mode=max cache from the same Dockerfile:

The build log shows the RUN lines with their mount flags and no value. The ssh step saw only a socket path inside the build container. Now search every place that leaked before, for the token and for a line of the private key file:

Zero everywhere: history, image config, layers, the attestation, the exported cache, the build record and its log. The record: line confirms that the last two counts come from the record of this build, not from an empty lookup. (The second search string is the third line of the private key file, which holds key material.) The max provenance records that a secret called npmrc and an ssh agent called default were used, which is useful for auditing, and nothing else. The secret file is mounted on a tmpfs for the duration of the RUN and is not part of the layer that step produces, so no cache mode can export it.

One limit remains. A mount keeps the value out of the image, but the command that runs inside the step can still put it somewhere. RUN --mount=type=secret,id=npmrc,target=/src/.npmrc cp .npmrc /tmp/ writes it into the layer, and a step that echoes the value prints it into the build log. Mounts protect what BuildKit records, not what your commands do.

Where each value ends up

Every cell is a result from this lab; a dash is a case the lab did not test. An ENV value is worse than all of these, since it sits in the config of the final image. If a real token has already gone through one of these channels, rotate it first. Deleting a tag, a cache reference or a build record afterwards limits further spread but does not undo copies that runners, mirrors and pull caches already hold.

Clean up

Remove the builder (which deletes its cache and records), the registry, the images and this lesson's build records from the default builder:

Try this

Work through “Clean up” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from build-time secrets and how images leak them, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related