Layers and the build cache

Why order matters, what busts the cache, what image sizes mean and where layers live.

The order of the instructions in a Dockerfile decides how much of each rebuild BuildKit can skip. Get it wrong and a pipeline reinstalls every dependency on every commit, even for a one-line README fix; get it right and a code change rebuilds one small layer. This lesson measures both, using two ideas: an image is a stack of layers, and BuildKit reuses a cached layer only while everything that went into it is unchanged. It also shows what the image sizes mean and why deleting a file in a later step does not remove it.

Use the main lab VM and unpack the lesson files into ~/lab/layers. They contain the small Express service from "Writing a Dockerfile" (renamed payments-api), its committed package-lock.json, a .dockerignore, the Dockerfile in the right order, Dockerfile.bad in the wrong order, and a third Dockerfile for the last section.

FROM node:24-alpine
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
USER node
EXPOSE 3000
CMD ["node", "server.js"]

An image is a stack of layers

Each Dockerfile instruction that changes files (COPY, ADD, RUN, and WORKDIR when it creates the directory) produces a layer: a set of file additions, changes and deletions relative to the layers below it. Instructions such as ENV, USER, EXPOSE and CMD only change the image configuration and add no files. FROM adds no layer of its own; it brings in the base image's layers. Layers are identified by the digest of their content, so two images built FROM node:24-alpine store the base layers once and share them. When a container starts, Docker adds a thin writable layer on top, private to that container, as described in "Images, containers and the core commands".

Build once and list the layers

The first build uses --no-cache so every step runs, as it would on a fresh machine. As in "Writing a Dockerfile", the plain progress format is shown and npm's update notice is cut.

docker history lists the layers of an image, newest first, with the instruction that created each and the size it added.

The top seven rows are this Dockerfile; everything from CMD ["node"] down came from the base image, which was built two weeks before this run. Your sizes and ages will differ. The configuration-only instructions (CMD, EXPOSE, USER, ENV) show 0B. RUN npm ci added 9.42MB, the largest layer of your own. The bottom row shows the Alpine root filesystem for aarch64, because the lab VM is an arm64 machine; on an amd64 host the same tag resolves to the amd64 variant and that row names an x86_64 archive. <missing> in the IMAGE column only means those intermediate layers have no image ID of their own; it is not an error. The COMMENT column shows that BuildKit built every layer, including the base image's.

How the cache decides

On each build, BuildKit walks the instructions in order and asks whether it already has a result for this exact step on top of this exact parent. For RUN, the key is the command text; BuildKit does not look at what the command would download. For COPY and ADD, the key includes a checksum of the files being copied. The first instruction whose key changed is executed again, and so is every instruction after it, because each one builds on the result of the one before.

Append a comment to server.js (with the current time, so the change is new every time you try this) and build again.

WORKDIR, the manifest copy and npm ci come back CACHED because none of their inputs changed. Only COPY . . ran, since server.js is part of what it copies. The plain output lists steps in the order BuildKit finished resolving them, which is not always file order; the [n/5] numbers give the Dockerfile order.

The wrong order

Dockerfile.bad copies everything first and installs afterwards.

FROM node:24-alpine
WORKDIR /app
COPY . .
RUN npm ci --omit=dev
USER node
EXPOSE 3000
CMD ["node", "server.js"]

Build it once, change server.js again, and rebuild.

This time COPY . . sits below the install, so the edited file changes its key, and npm ci runs again for a change that has nothing to do with dependencies. Here that costs about a second, because the app has a single dependency. A real service with a few hundred packages pays minutes per build, on every commit. The rule follows directly: copy what changes rarely (dependency manifests, then the install) before what changes often (the code). The same pattern exists for every ecosystem: requirements.txt before pip install, go.mod and go.sum before go mod download, pom.xml before the Maven dependency step.

A cached step is reused however old its result is. A RUN apt-get update && apt-get install -y curl step keeps its cached result until something below it changes, however old the package lists get. Pin versions where repeatability matters, and rebuild with --no-cache (or --pull to also check for a newer base image) when you deliberately want fresh results, for example in a scheduled rebuild that picks up security fixes.

What the sizes mean

All three images report 249MB of DISK USAGE, but they do not use 747MB between them. DISK USAGE counts every layer an image needs, including the shared base, and the images here differ only in their top layers. CONTENT SIZE (64MB) is the compressed size of the layers, roughly what a pull or push transfers. docker system df totals what is actually stored, counting shared layers once; the image count and the totals include everything on the VM, so yours will differ.

The Build Cache row is BuildKit's store of step results, including results no current image uses. It is kept until BuildKit's garbage collection trims it or you remove it with docker builder prune. That command clears cache for every project on the host, so on a shared machine leave it to whoever looks after the host.

Where the layers live

On a fresh Docker 29 install, image content is stored by containerd, not by Docker's older graph drivers: the storage driver reports overlayfs with driver-type io.containerd.snapshotter.v1, layer content sits under /var/lib/containerd, and /var/lib/docker has no overlay2 directory. Hosts upgraded from older versions keep the classic overlay2 driver until they migrate. "Where Docker keeps data" in Docker in depth explains both layouts and how to tell which one a host uses.

A deleted file is still in the image

Layers only add. A later instruction that deletes a file records the deletion in its own layer and hides the file from the final filesystem, but the earlier layer still contains it and anyone with the image can extract it. The third Dockerfile copies a stand-in key file, uses it, and deletes it.

FROM alpine:3.22
COPY fake-key.txt /tmp/fake-key.txt
RUN wc -c /tmp/fake-key.txt
RUN rm /tmp/fake-key.txt

The COPY layer still holds the 3 KB file (12.3kB on disk, rounded up to filesystem blocks), and the rm layer did not shrink anything; it added a few kilobytes of its own. Keep secrets out of build instructions entirely. "Build-time secrets and how images leak them" in Advanced container security shows how to extract such a file and the build features that avoid the problem.

Clean up

Try this

Work through “Clean up” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from layers and the build cache, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related