Distroless & scratch

No shell, no package manager, nothing to pivot with.

Advanced14 min · lesson 6 of 25

Most container break-ins are the start of the job, not the end of it. Someone finds a remote code execution bug (RCE, meaning they can run commands of their choosing on your server by talking to your app), fires the payload, and then goes shopping. Attackers call the next few minutes living off the land: use the tools the victim already installed. Spawn a shell. Run curl to pull down the next stage of the malware. Use cat and find to hunt for database passwords. Every one of those moves needs a program that is already sitting inside your image. Distroless and scratch win by taking the programs away.

A burglar who gets through the window is far less dangerous in an empty apartment than in one with a fully stocked garage. Your image's filesystem is that apartment. A normal debian or ubuntu base arrives furnished, with a toolbox on every shelf. scratch is the bare lot before anything was built: it is Docker's reserved do-nothing base, zero files and zero bytes, so the finished image holds only what you COPY into it. Distroless sits one small step above that. Google's gcr.io/distroless/* images carry what your program needs in order to run and nothing a person would reach for: glibc (the standard C library, the code that lets a program open files and talk to the network) in the base variant, CA root certificates (certificate authority, the list of signers your program trusts when it makes an HTTPS call), a timezone database, and an /etc/passwd holding one nonroot user. No shell. No package manager. No ls, no cat, no find.

Prove the shell is gone

terminal
# no shell to exec, whether you start one fresh or attach to a running container
$ docker run --rm gcr.io/distroless/static-debian12:nonroot sh
docker: Error response from daemon: failed to create task for container:
... exec: "sh": executable file not found in $PATH: unknown.
$ docker exec -it myapp-prod /bin/bash
OCI runtime exec failed: exec failed: unable to start container process:
exec: "/bin/bash": stat /bin/bash: no such file or directory: unknown

That error message is the feature. When the attacker's code calls execve("/bin/sh"), the system call that means "replace this process with that program", the kernel goes looking for the file inside the container's mount namespace. A mount namespace is the container's private view of the filesystem, a room with a one-way mirror that shows it nothing but itself. The file is not in the room, so the call comes straight back with ENOENT (error number for "no such file or directory"). The reverse-shell one-liner people paste off a cheat sheet dies on line one. There is no curl or wget, so fetching stage two fails. There is no apt or apk, so the attacker cannot install a replacement. Whatever happens next has to come out of your own binary.

terminal
# you can't `ls` inside a shell-less image, so inspect its rootfs from the host
$ cid=$(docker create gcr.io/distroless/static-debian12:nonroot)
# is there a shell, a downloader, or a package manager anywhere in the rootfs?
$ docker export "$cid" | tar -tf - | grep -E '/(sh|bash|busybox|curl|wget|apt|apk)$'
# (no output: none of them are in the image)
# how many executables ship before you COPY your own binary in?
$ docker export "$cid" | tar -tf - | grep -cE '(^|/)s?bin/[^/]+$'
0
$ docker rm "$cid" >/dev/null
# Zero tools. ~2,000 files DO ship, but they're almost all the timezone database
# and CA certs. A debian:12 rootfs, by contrast, carries over a thousand executables.

What actually gets smaller

Two things shrink at the same time. The first is the attacker's toolkit, which you deleted a moment ago. The second is your patch queue. A debian:12 base ships roughly a hundred operating system packages, and each one is a steady drip of CVEs (Common Vulnerabilities and Exposures, the public catalogue of known security bugs) that somebody on your team has to read and rule on, even for code your app never calls. A distroless static image ships close to zero of those packages, and most of that noise leaves with them. The nonroot variant runs as uid 65532 (uid is user ID, the number Linux uses instead of a name) rather than root, so any file the attacker does manage to write lands as an unprivileged user.

One honest caveat keeps distroless from sounding like magic. If your app runs on an interpreter, a distroless python or java image still has to ship that interpreter, and an interpreter does everything a shell does. python -c will open a socket and read files all day long. So the shell-less win is sharpest behind a compiled static binary, Go or Rust, where everything the program needs is baked into a single file and that file is the only executable code in the image. On interpreted stacks distroless still strips the operating system attack surface, which is worth having, but write the runtime down in your threat model as a tool the attacker gets for free.

Debugging a shell-less container: pick a door
Prod distroless container misbehaving
no shell to docker exec into
best
Ephemeral debug container
kubectl debug / docker debug shares the target's namespaces; the running image stays inert
host-side
nsenter + /proc/$pid/root
run host tools against the container's namespaces; nothing added to the image
last resort
:debug image tag
BusyBox shell baked in; only for the moment you need it, never left in prod
The first two never touch the running image, so the shell-less guarantee survives the debugging session.

Debugging without a shell

Losing docker exec sh costs you something real, and pretending otherwise helps nobody. The trick is to carry the tools to the container instead of storing them there, the way a mechanic drives a toolbox out to a broken-down car rather than keeping a full set of spanners in every glovebox. An ephemeral debug container (kubectl debug in Kubernetes, docker debug on Docker Desktop) attaches a throwaway toolbox that shares the target's process and network namespaces and never touches its filesystem. On a plain Docker host you can do the same by hand, because the kernel publishes every container's filesystem and namespaces under /proc, the virtual directory where Linux exposes live process state as ordinary-looking files.

terminal
# debug a shell-less container using tools from the HOST; the image is untouched
$ pid=$(docker inspect -f '{{.State.Pid}}' myapp-prod)
# read the container's filesystem via /proc, using the host's own `ls`
$ sudo ls -l /proc/"$pid"/root/
-rwxr-xr-x 1 65532 65532 8462112 Jul 16 09:12 server
drwxr-xr-x 2 65532 65532 6 Jan 1 1970 etc
# run the host's `ss` inside only the container's network namespace
$ sudo nsenter -t "$pid" -n ss -tln
State Recv-Q Local Address:Port
LISTEN 0 *:8080

Scratch asks for a fully self-contained binary

With no package manager, and on scratch no filesystem at all, everything your program needs at runtime has to be inside the artifact before it ever starts. The classic trap is the dynamic linker. A glibc-linked binary does not contain libc. It carries a note asking the loader ld-linux.so to fetch libc at startup, the way a recipe lists ingredients it expects to find already in the kitchen. On scratch there is no kitchen and no loader, so the container dies with "no such file or directory" while pointing at your binary, which is sitting right there in front of you. The missing file is the loader, not the program. Two more scratch-only gaps catch teams out. With no CA certificates, every HTTPS call fails with an x509 error (x509 is the certificate format TLS uses). With no /etc/passwd, USER has to be a bare number, and any getpwnam lookup (the C function that turns a username into an account record) comes back empty.

terminal
# a dynamically linked binary on scratch: misleading "no such file" (it's the loader)
$ docker run --rm myapp:scratch
exec /server: no such file or directory
$ ldd server
linux-vdso.so.1 (0x00007ffc9b3fe000)
libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007f2a41a00000)
/lib64/ld-linux-x86-64.so.2 (0x00007f2a41c00000)
# rebuild fully static, then it runs on a zero-file scratch base
$ CGO_ENABLED=0 go build -ldflags="-s -w" -o server ./cmd/server
$ ldd server
not a dynamic executable
A :debug tag left in prod quietly undoes all of this
The :debug variants bundle a BusyBox shell (BusyBox is one small binary that impersonates dozens of Unix commands at once) so you can poke at a container that has gone wrong. That is genuinely useful. The failure mode is forgetting to take it back out. A FROM gcr.io/distroless/static-debian12:debug-nonroot typed during a 3am incident, or a latest-debug that nobody ever reverted, puts a working shell back into production and hands the next attacker their toolkit. Make the build catch it for you. In CI (continuous integration, the pipeline that builds and tests every commit), export the finished image and fail the job if a shell turns up anywhere in the tree. Distroless debug images park busybox under /busybox rather than /bin, so match on the filename and not the directory: docker export $(docker create $IMG) | tar -tf - | grep -qE '(^|/)(sh|bash|busybox)$' && exit 1. Now the shell-less guarantee is something your pipeline enforces instead of something you have to remember.

Starving the living-off-the-land playbook is the whole strategy. No shell means no interactive pivot. No package manager means no "apk add curl" after an RCE. The bill comes due in debugging, so pay it on purpose: better logs and metrics up front, a break-glass debug tag you have actually rehearsed, and ephemeral debug containers instead of a permanently fat image you keep around in case.

Scratch wants a binary that genuinely needs nothing. distroless/static and distroless/base cover the common middle ground, where you want two or three files present (certificates, a timezone table, a libc) and still no tools. Read the notes on the specific image you pick. "Distroless" names a family, not one filesystem, and the variants differ on which glibc and openssl pieces come along for the ride.

An attacker who expects bash will stall out. An attacker who compiled a fully static implant and drops it straight into memory may not care that your image is empty. Treat distroless as necessary rather than sufficient, and keep the other layers switched on: non-root, a read-only root filesystem, and a seccomp profile (seccomp decides which kernel system calls a process is allowed to make at all).

That export-and-grep check earns its keep after every change window, not only on the day you build the image. Run it, confirm the control is still on, paste the command and its output into the ticket, and refuse to close the change if the reading has drifted since last time. Pick the tightest image that still lets the workload run. That habit compounds across every host and every pipeline you own.

Try this

Put a fat alpine image next to a distroless one and try to get a shell in each. Watch the pivot tools disappear.

terminal
$ docker run --rm alpine sh -c 'command -v sh; command -v wget; echo alpine-ok'
/bin/sh
/usr/bin/wget
alpine-ok
$ docker run --rm --entrypoint sh gcr.io/distroless/static:nonroot -c 'echo hi' 2>&1 | head -3
docker: Error response from daemon: failed to create task for container: ...
failed to find path /bin/sh / ... executable file not found
$ # expected: no shell to exec — that is the control working

Takeaway

Distroless takes away the toolbox an attacker expects to find after an RCE. You pay for that in harder debugging, so keep one controlled way in and rehearse it. Then keep stacking the rest: non-root, read-only root filesystem, syscall filters. An empty image is not a sandbox by itself.

Quick check
01An attacker lands remote code execution inside a container built FROM gcr.io/distroless/static-debian12:nonroot that runs a static Go binary. Which next move is actually open to them?
Incorrect — Neither curl nor sh exists in a distroless static image, so this line dies before it does anything.
Correct — With no shell, no downloader and no package manager in the image, the attacker is boxed into what your program can already do.
Incorrect — There is no package manager here. The debian12 in the name only says where a couple of libraries came from, not that apt shipped with them.
Incorrect — The image has no shell, so an exec of /bin/bash comes straight back with 'no such file or directory'.
02The lesson names one case where distroless does NOT give you the full shell-less benefit. Which one?
Incorrect — No. UID 65532 is an ordinary unprivileged account, so the nonroot variant lowers privilege rather than keeping root's powers.
Incorrect — No. No distroless image ships a package manager, so there is no apt to install anything with.
Correct — Yes. An interpreter does a shell's job, so the shell-less win is sharpest for compiled static binaries and weaker for interpreted stacks.
Incorrect — No. Distroless carries no shell at all, which is exactly why an exec of sh reports 'no such file'.
03A production container built FROM gcr.io/distroless/static-debian12:nonroot is misbehaving, and docker exec ... sh fails because there is no shell in the image. Which approach lets you investigate WITHOUT weakening the running container's shell-less guarantee?
Incorrect — No. A debug image left in production puts a shell straight back and hands post-exploitation its toolkit again.
Correct — Yes. The tools come to the container through shared namespaces, so the running image stays inert and shell-less.
Incorrect — No. Writing a shell into the live container changes it and destroys the exact property you were trying to keep.
Incorrect — No. The image holds no shell whichever user you run as, and a restart tells you nothing about the live process you wanted to inspect.

Related