Multi-stage builds, in depth

Leave every compiler and package manager behind.

A Go web service needs roughly 900 MB of toolchain to compile and about 8 MB to actually run. Ship all 900 MB and the first person who finds an RCE (remote code execution, meaning they can make your app run commands of their choosing) inherits a compiler to build a payload, git to push stolen data back out over a port your firewall already allows, a package manager to fetch more tools, and a shell to tie it all together. None of that serves a single request. It rode along because the build needed it, and nobody took it back out.

A multi-stage build takes it back out for you, and the shape of it is a workshop and a display case. You cut, weld and sand in the workshop, then put only the finished piece behind glass. The saws stay in the workshop. In a Dockerfile (the text file that lists the steps for building an image) that means one stage carrying the full toolchain that compiles your artifact, then a second, nearly empty stage that copies out the finished binary and nothing else. Docker keeps only the last stage. Everything in the builder is thrown out before a single byte reaches a registry (the server your images get pushed to and pulled from).

# syntax=docker/dockerfile:1
FROM golang:1.23 AS build              # full toolchain: compiler, git, shell (~900MB)
WORKDIR /src
COPY go.mod go.sum ./
RUN go mod download
COPY . .
RUN CGO_ENABLED=0 go build -ldflags="-s -w" -o /out/server ./cmd/server
 
FROM gcr.io/distroless/static:nonroot  # ~2MB base: no shell, no package manager
COPY --from=build /out/server /server
USER nonroot
ENTRYPOINT ["/server"]

Build it both ways, then weigh it

Here is the same app built two ways so the gap is visible. The fat build is that Dockerfile with the second stage deleted, so the golang base ships as-is with every tool it carries. The slim build is the file above. Three small details are carrying the weight. CGO_ENABLED=0 tells Go to produce a fully static binary, one that carries everything it needs inside itself instead of calling out to libc (the shared C library nearly every Linux program leans on for basics like opening a file). That independence is what lets the final base be almost empty. The -ldflags="-s -w" pair strips the symbol table and the debug information, so the binary is smaller and far more annoying to reverse-engineer. And USER nonroot on a distroless base (an image holding your program and its runtime files and essentially nothing else, no shell, no package manager) starts the process as an unprivileged user with no ladder back up, because there is no su and no shell to run it in.

$ docker build -t svc:fat -f Dockerfile.fat .
$ docker build -t svc:slim .
$ docker images svc
REPOSITORY   TAG    IMAGE ID       CREATED          SIZE
svc          slim   9c7b1e4a2f08   6 seconds ago    8.21MB
svc          fat    3f2a9d5c1b77   38 seconds ago   1.16GB
# same binary inside; the slim image is ~140x smaller because the toolchain never ships

Copy the artifact, never the stage

The discipline that makes this pay off is copying the narrowest thing you can with COPY --from. One binary. One directory of static assets. Never COPY --from=build /, which hauls the whole toolchain back in and quietly cancels the point of the exercise. Naming your stages with AS buys you something extra as well. Running docker build --target build stops at the builder, so one file gives you a tiny production image and a full-toolchain debugging image that are identical, byte for byte, up to that stage. Two outputs, one file, no drift between what you poke at and what you ship.

Prove the final image shipped nothing

Small is nice. What you actually want is proof that the attack surface went away, and you can measure that directly. Two readings tell you. There is no shell to drop into, and a vulnerability scanner finds no operating-system packages that could carry a CVE (Common Vulnerabilities and Exposures, the public catalogue of known security flaws, each one with an ID and a write-up anybody can read). Run both against the slim image, then against the fat one. The contrast is the whole lesson in two commands.

$ docker run --rm --entrypoint sh svc:slim
docker: Error response from daemon: failed to create task for container:
failed to create shim task: OCI runtime create failed: runc create failed:
unable to start container process: error during container init:
exec: "sh": executable file not found in $PATH
 
$ trivy image --scanners vuln svc:fat
svc:fat (debian 12.8)
Total: 187 (UNKNOWN: 0, LOW: 92, MEDIUM: 71, HIGH: 21, CRITICAL: 3)
 
$ trivy image --scanners vuln svc:slim
svc:slim (debian 12)
Total: 0 (UNKNOWN: 0, LOW: 0, MEDIUM: 0, HIGH: 0, CRITICAL: 0)
# nothing named sh to exec, and 187 OS-package CVEs drop to 0 by severing the build stage

The leak multi-stage does not fix

Builder stages never reach the registry, which makes them look like a fine hiding place for a private token, or for an SSH (Secure Shell, the encrypted protocol Git uses to prove who you are) key so go mod download can reach a private repo. Resist that. A secret you COPY into a builder, or hand over as a build ARG, gets written into that stage's layers while the build runs. Those layers stay behind in your local build cache, and in any shared CI (continuous integration, the automated build service that runs your pipeline) cache, long after the stage was 'thrown away.' Anyone who can read the cache can read the secret. Recent Docker versions do warn you, since the BuildKit linter (BuildKit is the engine that actually executes your Dockerfile) prints a SecretsUsedInArgOrEnv warning, but that line scrolls past in a wall of build output, so do not lean on it. Here is how you catch the leak after the fact, and how you close it.

# a builder that took the token as a build ARG (the common mistake)
$ docker build --target build --build-arg NPM_TOKEN=$NPM_TOKEN -t svc:builder .
$ docker history --no-trunc svc:builder | grep -io 'NPM_TOKEN=[^ ]*'
NPM_TOKEN=npm_9f3xQK2mZp7v...     # <-- the live token, read straight from layer metadata
# the stage is 'discarded' from the final image, but its layers are cached and readable

# syntax=docker/dockerfile:1
FROM node:22 AS build
WORKDIR /app
COPY package*.json ./
RUN --mount=type=secret,id=npmrc,target=/root/.npmrc \
    npm ci                       # token exists only for this RUN, in a tmpfs (in-memory) mount
COPY . .
RUN npm run build
 
# build it, then confirm the secret never landed in a layer:
$ docker build --secret id=npmrc,src=$HOME/.npmrc -t svc:1.0 .
$ docker history --no-trunc svc:1.0 | grep -i npmrc
$                                 # empty output: nothing to recover from any layer or cache

Multi-stage gets you a small image. How small, and how safe, comes down entirely to the base of that last stage. Build multi-stage onto ubuntu:latest and you still hand an attacker a shell and apt to work with. The next lessons pick that final base. distroless, scratch and static binaries are what turn 'smaller' into 'nothing left to work with.'

Keep the split in your head as workshop and display case. Compilers, package managers and git belong in the workshop stage. The final stage gets the artifact and the runtime files it genuinely needs, and nothing beyond that. Every extra binary in there is a tool you decided to hand to whoever breaks in.

The failures that get people are the ones that look tidy. You COPY a whole /app directory forward and it still holds node_modules build caches, a .git directory with your full history, and test fixtures with sample credentials in them. Name your COPY paths explicitly, one at a time. Pin base images by digest (the sha256 fingerprint of one exact image) so yesterday's builder cannot quietly turn into a different builder overnight.

A gate in your pipeline should refuse any image whose runtime stage still contains apk, apt, gcc or cargo when that stage was meant to be minimal. Size on its own is a weak signal. A bloated base can compress well, and a small one can still carry a package manager. Content is the signal. Assert on what is present, not on megabytes.

Treat the check as part of the change, not a favour you do afterwards. Once a Dockerfile edit ships, take the two readings again, paste the command and its output into the ticket, and refuse to close the change if the numbers moved. A control you verified once, months ago, is a belief rather than a control.

Sooner or later the final stage needs one more thing: a timezone database, a CA (certificate authority) bundle so outbound TLS (Transport Layer Security, the encryption behind https) can check who it is talking to, a directory of templates. Copy the file. Do not install the package that would have provided it, because the package drags a package manager and a dependency tree in behind it. One explicit COPY line beats one apt-get install every single time.

The first objection you will hear is that nobody can debug a container with no shell in it. Correct, and that is the point. You debug it from the outside instead. Attach an ephemeral debug container that carries the tools and shares the target's namespaces, or rebuild with --target build and run that image locally against the same inputs. The production image stays empty. Your tools live somewhere an intruder cannot reach them.

Shared build cache is a trust boundary, and almost nobody treats it as one. If your pipeline pushes cache to a registry and pulls it back on every run, then whoever can write to that location can influence what your builds produce. Scope the credentials for it tightly, keep it in a repository you control, and never point it somewhere an untrusted fork's pipeline can write.

If you ever find a live token in docker history, treat it as spent. Rotate it first, before you clean anything up, because the cleanup takes time and the credential stays readable the whole while. Then clear the build cache on every host and every runner that ever pulled that layer, the CI fleet included. Deleting the image changes nothing on its own. The layer is a separate object with its own life.

A slim image also makes the boring paperwork easier. When an auditor asks what is inside the thing you deployed, a distroless image with one static binary has a short answer and an SBOM (software bill of materials, a machine-readable list of every component in a build) that fits on one screen. The fat image has 187 OS packages worth of answer, and each package is a question you get to answer again every time a new flaw is published.

Not every stack collapses down to one file. A Python or Node service drags an interpreter and a dependency tree into the final stage however you slice it. The habit still pays off. Build wheels or run npm ci in the builder, copy the resolved dependency directory forward, and leave the compilers, headers and git behind. You will not land at 8 MB. You will still drop the tools that turn one bug into a foothold.

Give yourself one number to watch per service: how many executables sit in the final image. Take the reading while the image is healthy, write it down in the repo next to the Dockerfile, and compare on every change. A count that drifts upward always has a reason, and the reason is usually somebody who needed a tool for an afternoon and never took it back out.

Try this

Build a two-stage Go image (or the equivalent in whatever language you use) and then go looking for the compiler in the final stage. Run docker history against the builder and against the final image and compare what each one carries. Your program should be in there. Any way to compile another one should not.

$ cat > Dockerfile <<'EOF'
FROM golang:1.22-alpine AS build
WORKDIR /src
RUN printf 'package main\nimport "fmt"\nfunc main(){fmt.Println("ok")}\n' > main.go
RUN CGO_ENABLED=0 go build -o /out/app main.go
FROM alpine:3.20
COPY --from=build /out/app /app
ENTRYPOINT ["/app"]
EOF
$ docker build -t multi:demo .
$ docker run --rm multi:demo
ok
$ docker run --rm --entrypoint sh multi:demo -c 'command -v go || echo no-go; ls -l /app'
no-go
-rwxr-xr-x    1 root     root       ... /app

Takeaway

Multi-stage builds are how you stop shipping the forge along with the sword. Keep the toolchain inside a named builder stage, COPY only the finished artifact forward, and check the final image for a compiler or a package manager before you trust it in production.

Related