Minimal bases: distroless, scratch, static and Alpine

Choose a runtime base and still be able to debug it.

docker exec lab-probe sh fails with exec: "sh": executable file not found in $PATH. On a distroless image that is the design working: there is no shell, no package manager and no downloader for an attacker who gets code execution to reach for, and no apt-get install to bring them back. The cost is that the image also lacks the small runtime files a program quietly expects, such as CA certificates, a timezone database or a user entry, and that debugging needs a different method. This lesson builds one Go program on three bases to see exactly which files matter, proves static versus dynamic linking, compares Alpine and Debian, and debugs the shell-less container from a sidecar.

Use the main lab VM. The lesson files go in ~/lab/distroless. The distroless images come from gcr.io, not Docker Hub, so they do not count against Docker Hub's pull limit.

What a distroless image contains

Google's distroless project publishes images built from Debian 13 packages: static-debian13 for programs that need no C library, base-debian13 which adds glibc and OpenSSL, cc-debian13 which adds the C++ runtime, and language images such as python3-debian13, nodejs24-debian13 and java21-debian13. Each comes in four tags: latest (runs as root), nonroot (UID 65532), and debug or debug-nonroot, which add a BusyBox shell. Pull the ones this lesson uses (the loop at the end pulls alpine:3.22 and debian:trixie-slim for the size comparison if an earlier lesson has not) and read the config of the static image:

The config sets a user, an environment and a working directory, and no Cmd or Entrypoint; your image supplies those. That is why docker create on the bare base fails with no command specified. To look inside such an image, give docker create any command at all; it is never run:

The 1400 entries include a few that docker export adds for every container (.dockerenv, etc/hosts, etc/resolv.conf). Of the rest, 1247 are timezone files. There are no ELF executables at all, no shell and no package manager. What is there is exactly the runtime support a static program needs: a passwd file with root, nobody and nonroot, and a CA certificate bundle. The same check, with the grep for shells and package managers, is a useful CI gate on your own final images.

One program on three bases

probe is a small Go program that reports its user and tries the four things that typically break on a minimal base: a user lookup, a timezone conversion, a DNS lookup and an HTTPS request. Run with serve, it is an HTTP server on port 8080 for the debugging section.

// probe: checks the runtime files a minimal image may or may not have.
package main
 
import (
	"fmt"
	"net"
	"net/http"
	"os"
	"os/user"
	"time"
)
 
func check(name string, err error, ok string) {
	if err != nil {
		fmt.Printf("%-5s FAIL  %v\n", name, err)
		return
	}
	fmt.Printf("%-5s ok    %s\n", name, ok)
}
 
func main() {
	if len(os.Args) > 1 && os.Args[1] == "serve" {
		http.HandleFunc("/", func(w http.ResponseWriter, r *http.Request) { fmt.Fprintln(w, "probe ok") })
		fmt.Println("listening on :8080")
		fmt.Println(http.ListenAndServe(":8080", nil))
		return
	}
	fmt.Printf("uid=%d gid=%d\n", os.Getuid(), os.Getgid())
 
	u, err := user.Current()
	check("user", err, func() string { if u != nil { return u.Username + " " + u.HomeDir }; return "" }())
 
	loc, err := time.LoadLocation("Europe/Berlin")
	check("tz", err, func() string { if loc != nil { return time.Date(2026, 7, 1, 12, 0, 0, 0, time.UTC).In(loc).Format("15:04 MST") }; return "" }())
 
	addrs, err := net.LookupHost("mirror.gcr.io")
	check("dns", err, fmt.Sprintf("%d addresses", len(addrs)))
 
	c := &http.Client{Timeout: 10 * time.Second}
	resp, err := c.Get("https://mirror.gcr.io/v2/")
	if err == nil {
		resp.Body.Close()
	}
	check("tls", err, func() string { if resp != nil { return resp.Status }; return "" }())
}

The Dockerfile builds it once with CGO_ENABLED=0 and copies the binary onto three bases: plain scratch, scratch plus three runtime files, and distroless static. Note the numeric USER on the scratch stages, and that the builder and the distroless base are pinned by digest, so a rebuild next month starts from the same files; a bot moves the pins, as "Pinning, SBOMs, provenance and scanning" describes. Dockerfile comments must start a line; a comment after FROM, COPY or USER breaks the instruction.

FROM golang:1.27-alpine@sha256:8a5910f31396cd4d89662f56c68b3ae31d374308270a1c3bd96672ee5ed43414 AS build
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=0 go build -trimpath -ldflags="-s -w" -o /out/probe .
RUN echo 'app:x:10001:10001:app:/nonexistent:/sbin/nologin' > /out/passwd
 
FROM scratch AS scratch-bare
COPY --from=build /out/probe /probe
USER 10001:10001
ENTRYPOINT ["/probe"]
 
FROM scratch AS scratch-full
COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
COPY --from=build /usr/local/go/lib/time/zoneinfo.zip /usr/share/zoneinfo.zip
ENV ZONEINFO=/usr/share/zoneinfo.zip
COPY --from=build /out/passwd /etc/passwd
COPY --from=build /out/probe /probe
USER 10001:10001
ENTRYPOINT ["/probe"]
 
FROM gcr.io/distroless/static-debian13:nonroot@sha256:e2e927ec666bae08560abb3c55d0659eceabb657f56b6782ab500a9fc7f555e3 AS distroless
COPY --from=build /out/probe /probe
ENTRYPOINT ["/probe"]

On bare scratch three of the four checks fail, each for a missing file:

scratch-full copies those three things from the build stage and passes everything; distroless static passes because it ships them. The next check shows the other reason for a numeric USER: Docker resolves a user name against the image's /etc/passwd when the container starts, and on scratch there is no file to resolve against.

A numeric UID needs no entry. That is also why distroless/static:nonroot and the scratch stages above work with volume ownership set by number; "Run as non-root" covers that part.

Static or dynamic: prove it

Go binaries are static only when cgo is off. Go enables cgo by default for native builds when it finds a C compiler; golang:*-alpine has none, so cgo was off above, while the Debian-based golang images include gcc and produce binaries linked against glibc for code that uses net or os/user. Dockerfile.cgo turns cgo on deliberately (after installing gcc and the musl headers) and puts the result on distroless base, which does have a C library, but glibc:

FROM golang:1.27-alpine@sha256:8a5910f31396cd4d89662f56c68b3ae31d374308270a1c3bd96672ee5ed43414 AS build
RUN apk add --no-cache gcc musl-dev
WORKDIR /src
COPY go.mod main.go ./
RUN CGO_ENABLED=1 go build -trimpath -o /out/probe .
 
FROM gcr.io/distroless/base-debian13:nonroot@sha256:a0d70d6a97cd697d9362bc2aae4a6560dd65817e365d0043b07325a97975dc91
COPY --from=build /out/probe /probe
ENTRYPOINT ["/probe"]

no such file or directory names /probe, which is plainly there. The missing file is the program interpreter, the dynamic loader the kernel must start first. file and ldd on the two binaries show it:

The static build is statically linked and ldd says not a dynamic executable. The cgo build names its interpreter, /lib/ld-musl-aarch64.so.1, and needs libc.musl-aarch64.so.1. The VM's glibc ldd substitutes its own loader for that interpreter line and cannot find the musl C library at all. A distroless or Debian base has glibc's loader, not musl's, so a binary linked on Alpine does not run there, and the reverse is equally true. On an amd64 machine the names read x86-64, /lib/ld-musl-x86_64.so.1 and libc.musl-x86_64.so.1. Put file on your build artifact into CI and fail on dynamically linked when the target base is scratch or static.

Building C with gcc -static against glibc does not give the same guarantee. glibc's user and host lookups (getpwnam, getaddrinfo) go through the Name Service Switch, which loads libnss_* modules at run time even in a statically linked program, so the binary starts on scratch and then fails on its first lookup. The linker warns about this when it links those functions statically. For static C or Rust, link against musl (for Rust the x86_64-unknown-linux-musl or aarch64-unknown-linux-musl target); for Go, CGO_ENABLED=0.

Alpine, musl and Debian slim

These are disk-usage figures on this VM (unpacked layers plus stored content, as explained in "Layers and the build cache" in Docker for beginners). alpine:3.22 is a tenth of debian:trixie-slim, and distroless static is smaller still. Size is not the whole decision. Alpine uses musl instead of glibc, and the differences show up in production:

The musl behaviour is documented on the musl wiki; the binary incompatibility you saw in the lab. Choose Alpine for static or musl-native programs and for small utility images. Choose Debian slim (or distroless base or cc) when the service pulls in native extensions, vendor binaries or deep recursion in threads, or when a DNS difference would be expensive to discover in production. For a static Go or Rust binary, distroless static or scratch with the three files above beats both.

Debugging a container without a shell

Start the probe as a server. docker exec has nothing to run:

Bring the tools to the container instead. A debug container started with --pid container:NAME joins the target's PID namespace and with --network container:NAME its network namespace, so its tools see the target's processes and its localhost, while the target's image stays untouched:

PID 1 is /probe serve running as 65532, the listener on 8080 is visible, and wget reaches the app over the shared loopback. Reading the target's filesystem through /proc/1/root needs more: the kernel requires ptrace access to another process's root, and Docker's default capability set does not include CAP_SYS_PTRACE. Grant it to the debug container only, for the session:

A debug container sharing the target's PID namespace can also signal and trace the target, so treat it like an interactive root session: start it with --rm, keep it short, and use an image you trust (busybox:1.37 here). On Kubernetes, kubectl debug creates the same kind of ephemeral container. docker debug exists too, but it is a Docker Desktop feature and not part of Docker Engine. The basic no-shell tools (inspect, logs, cp) are in "Inspect, logs and exec" in Docker for beginners.

The :debug images are for reproducing a problem on your own machine:

It is the same base with a BusyBox of 382 applets under /busybox, outside the usual PATH directories. A :debug tag that ships to production undoes the point of the base, which is what the export check from the first section catches: it matches busybox and sh by name, wherever they are.

Clean up

The distroless base images stay in the image store for the next lessons; remove them with docker image rm if you need the space.

Try this

Work through “Clean up” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from minimal bases: distroless, scratch, static and alpine, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related