Slimming & layer hygiene

Measure the image, then cut every wasted byte.

Delete a file from a Docker image and the bytes stay put. The file only stops being visible from the top. Here is the shape of it: an image gets built like a stack of clear plastic sheets, each one drawn on and then laid over the last. Write a secret on one sheet, lay an opaque sheet over it, and the ink is hidden, not gone. Lift the top sheet and it still reads fine. Docker calls those sheets layers, and layers are append-only. Every COPY and every RUN seals a new read-only sheet on top of the one before. When you rm a file in a later step (rm is the Unix remove command), Docker writes a small "whiteout" marker, a note that says treat this path as missing. The original bytes never move. They sit in the earlier layer, and they ride along to every registry the image is pushed to and every laptop that pulls it. This is not theory. Researchers who bulk-scanned public Docker Hub images have pulled live AWS (Amazon Web Services) keys, database passwords, and private SSH (Secure Shell) keys out of layers the authors were certain they had cleaned.

Layers add, they never subtract

A finished image is an overlay filesystem, OverlayFS in Linux terms: a pile of read-only layers, with one thin writable layer dropped on top only when a container actually runs. Each RUN, COPY and ADD seals one more read-only layer. Once a file lands in any layer it stays reachable in the finished image, whether or not a later layer marks it deleted, because "deleted" is one more entry that says pretend this path is empty. That makes the cleanup rule narrow, and worth memorizing. Removing a file shrinks the image, or scrubs a secret, only when the removal happens inside the same RUN that created the file. Split the create and the delete across two RUN lines and the secret is baked in for good.

FROM node:20-alpine
WORKDIR /app
# a deploy key, "just so npm can reach our private registry"
COPY deploy_key /root/.ssh/id_ed25519
RUN chmod 600 /root/.ssh/id_ed25519 \
 && npm ci                         # uses the key to pull a private package
RUN rm -f /root/.ssh/id_ed25519     # "cleanup" in a SEPARATE layer, too late
COPY . .                            # drags .git, .env, everything into a layer

$ docker build -t app:leaky .
$ docker history --no-trunc --format 'table {{.Size}}\t{{.CreatedBy}}' app:leaky
SIZE      CREATED BY
12MB      COPY . . # buildkit
0B        RUN /bin/sh -c rm -f /root/.ssh/id_ed25519 # buildkit
38MB      RUN /bin/sh -c chmod 600 /root/.ssh/id_ed25519  && npm ci # buildkit
399B      COPY deploy_key /root/.ssh/id_ed25519 # buildkit
0B        WORKDIR /app
...
# that 399B COPY layer still holds the key. unpack the image and prove it:
$ docker save app:leaky -o app.tar && mkdir lay && tar -xf app.tar -C lay
$ for b in lay/blobs/sha256/*; do tar -tf "$b" 2>/dev/null \
    | grep -q id_ed25519 && echo "leak in $b" \
    && tar -xOf "$b" root/.ssh/id_ed25519 | head -1; done
leak in lay/blobs/sha256/7b41c2f0d9...c9
-----BEGIN OPENSSH PRIVATE KEY-----

Scan for it, don't eyeball it

docker history is a hint, not proof. Reading it is like guessing what is in a shopping bag from the receipt: usually close, sometimes wrong. On real images the CreatedBy column gets truncated, and images that were squashed or built by BuildKit (the build engine Docker uses by default now) can hide the instruction that wrote a file. So do not trust your eyes. Use a scanner that walks every layer and matches known secret patterns. Trivy does that in one pass, which is why it still flags the key even though a later layer "deleted" it. Wire it into your pipeline and fail the build on any hit. Then cut off the most common leak at the source with a .dockerignore file, so a lazy COPY . . cannot sweep your .git directory (the whole commit history, sometimes tokens) or a local .env file into a layer in the first place.

.git
.env
*.pem
deploy_key
node_modules
test/fixtures
# nothing here can be COPY-ed in by accident or bloat the build context

$ trivy image --scanners secret app:leaky
2026-07-16T10:22:14Z  INFO  Secret scanning is enabled
app:leaky (alpine 3.21.3)
 
root/.ssh/id_ed25519  (secrets)
Total: 1 (HIGH: 1)
 
HIGH: AsymmetricPrivateKey (private-key)
════════════════════════════════════════
 Asymmetric Private Key
────────────────────────────────────────
 root/.ssh/id_ed25519:1
────────────────────────────────────────
   1 [ -----BEGIN OPENSSH PRIVATE KEY-----*****
────────────────────────────────────────

Same mechanic, wasted megabytes

The behavior that traps secrets also wastes disk. Run apt-get update in one layer (apt-get is Debian's package tool), install in the next, and rm -rf /var/lib/apt/lists/* in a third, and those downloaded package lists sit in the first two layers, shipped everywhere, useful to nobody. The rm in the third layer paints a whiteout over them and nothing more. Chain the whole sequence into a single RUN, add --no-install-recommends (or --no-cache with Alpine's apk) so the extras never arrive at all, and delete the lists before the layer seals. dive is the tool that scores how much of an image is dead weight like this. Run it in CI (continuous integration, the automated build that fires on every push) with --ci and an efficiency threshold, and a bloated build fails on its own instead of quietly shipping 40 wasted megabytes.

# one layer: update, install, and purge the lists before it seals
RUN apt-get update \
 && apt-get install -y --no-install-recommends curl ca-certificates \
 && rm -rf /var/lib/apt/lists/*

# the anti-pattern: install and cleanup in different layers, so the lists never leave
$ docker build -t demo:bloat - <<'EOF'
FROM debian:12-slim
RUN apt-get update
RUN apt-get install -y --no-install-recommends curl ca-certificates
RUN rm -rf /var/lib/apt/lists/*
EOF
$ dive --ci --lowestEfficiency=0.95 demo:bloat
  efficiency: 62.3149 %
  wastedBytes: 39821312 bytes (40 MB)
  userWastedPercent: 100.0000 %
Inefficient Files:
Count  Wasted Space  File Path
    2         31 MB   /var/lib/apt/lists/deb.debian.org_debian_dists_bookworm_main_binary-amd64_Packages
    2        6.4 MB   /var/lib/apt/lists/deb.debian.org_debian_dists_bookworm-updates_main_binary-amd64_Packages
Results:
  FAIL: lowestEfficiency: 0.62 is below threshold 0.95
  FAIL: highestUserWastedPercent: 1.00 exceeds threshold 0.10
  SKIP: highestWastedBytes: rule disabled
Result:FAIL [Total:3] [Passed:0] [Failed:2] [Warn:0] [Skipped:1]

The fix: keep the secret out of every layer

Two clean fixes, one blunt one. Clean fix one is a BuildKit secret mount. You hand a credential to a single RUN, BuildKit mounts it into that one command, and it never lands in a layer. It works like lending a neighbor your door key for one afternoon and taking it straight back, with no copies cut. Clean fix two is a multi-stage build. Anything sensitive lives in a builder stage that gets thrown away, the way a carpenter leaves the sawdust and offcuts in the workshop, while the final stage copies out the finished piece and nothing else. The blunt option is docker build --squash, which flattens every layer into one so files deleted along the way really do vanish. It belongs to the legacy builder (BuildKit, the default now, does not do it), it is still marked experimental, and it wrecks layer-cache reuse, so every push re-uploads the whole image. It also cannot do a thing about a secret you already shipped. Reach for secret mounts and multi-stage first.

# syntax=docker/dockerfile:1
FROM node:20-alpine AS build
WORKDIR /app
COPY package*.json ./
# key is mounted for THIS run only; it never lands in a layer
RUN --mount=type=secret,id=deploy_key,target=/root/.ssh/id_ed25519,mode=0600 \
    npm ci
COPY . .
RUN npm run build
 
FROM gcr.io/distroless/nodejs20-debian12   # tiny, no shell, no key
COPY --from=build /app/dist /app           # only the artifact crosses over
WORKDIR /app
CMD ["server.js"]

$ docker build --secret id=deploy_key,src=./deploy_key -t app:clean .
$ docker save app:clean -o clean.tar && mkdir c && tar -xf clean.tar -C c
$ for b in c/blobs/sha256/*; do tar -tf "$b" 2>/dev/null \
    | grep -q id_ed25519 && echo "found in $b"; done
$ echo "(no 'found in' line above means the key is in no layer)"
(no 'found in' line above means the key is in no layer)
$ trivy image --scanners secret app:clean
2026-07-16T10:31:02Z  INFO  Secret scanning is enabled
app:clean (debian 12.8)
Total: 0
# the builder stage held the key for one RUN, then got discarded. nothing shipped.

In code review the tell is cheap to spot. Any Dockerfile where one instruction brings a credential in and a separate RUN takes it out later is leaking, however tidy the comment above it reads. Ask for a secret mount instead. The second tell is a bare COPY . . near the top of a file with no .dockerignore sitting beside it, which quietly sweeps the entire working directory, .git history included, into a layer.

Gate this rather than remember it. A secret scan on the built image and a dive efficiency threshold both exit with a non-zero status when they fail, so both can stop a pipeline with no glue code. Run them against the image you are about to push, not against the source tree. The source tree never shows you what a build wrote into a layer.

Write the rotation step down before you need it. When a key does reach a pushed layer, the clock started at push time, not at the moment you noticed, and the only question that matters is how fast your team can issue a replacement and retire the old one. Teams that can answer in minutes treat a leaked deploy key as a chore. Teams that cannot treat it as an incident.

Build context deserves the same care as the image. Everything in the directory you build from is uploaded to the Docker daemon (the background service that does the building) before the first instruction runs, so a fat context slows every build and hands COPY far more than it should ever see. Keep test fixtures, local environment files and vendored dependency directories out of it. Where a build genuinely needs a package cache, mount it for the RUN that needs it, so it speeds the build up without settling into a layer.

After any change to a base image or a build stage, run the same two checks and paste the output into the change ticket: the secret scan on the new digest (the fingerprint that names one exact build), and the efficiency score. If either reading has drifted from the last known good one, refuse to close the change. Keep the tightest build that still produces a working artifact. That habit compounds across every image and every pipeline you own.

Try this

Build the trap yourself, because watching a whiteout fail to delete anything lands harder than reading about it. Write a fake secret into an image in one layer, remove it in the next, then prove from the outside that it is still there, either through docker history or by saving the image and searching the layer tarballs.

$ cat > Dockerfile <<'EOF'
FROM alpine:3.20
RUN echo 'AKIA_EXAMPLE_SECRET' > /tmp/key && cp /tmp/key /opt/key
RUN rm -f /tmp/key /opt/key
EOF
$ docker build -t leak:demo .
$ docker run --rm leak:demo sh -c 'ls /opt /tmp; echo clean-from-top'
clean-from-top
$ docker history --no-trunc leak:demo | head -5
IMAGE ... CREATED BY ...
... RUN /bin/sh -c rm -f ...
... RUN /bin/sh -c echo 'AKIA_EXAMPLE_SECRET' ...
$ # history still shows the secret in the earlier layer command — treat layers as forever

Takeaway

Never take rm as evidence that an image is clean. Keep credentials out of the layers from the start with BuildKit secret mounts and multi-stage builds, collapse any create-then-delete pair into one RUN, and read the layer history before you push instead of after somebody else does.

Related