Runtime secrets, done right

Deliver secrets as files, not environment variables, and close the run-time leaks.

A password passed with -e DB_PASSWORD=... is readable by anyone who can reach the daemon, whatever else you did to the container. "Environment variables and configuration" (Docker for beginners) already warned that environment values are stored in the container config and read back by tooling; this lesson shows exactly where a secret passed that way turns up, moves it to a file instead, and names the ways a file-based secret can still leak at run time. Everything runs on the main lab VM (secopslog-docker), in ~/lab/runsecrets. All values are fake.

Where an environment secret shows up

Start one container with a secret in its environment, then read it back from outside the container and from the host:

docker inspect prints the value to anyone in the docker group, which "The Docker socket and daemon hardening" shows is the same as root on the host. /proc/<pid>/environ is the environment block the process was handed at start, read from its memory and readable by host root; unsetting the variable in code later does not clear that block. And every child process inherits it. Two of those three reads came from outside the container entirely, so the in-container user and a read-only root filesystem change nothing. There are two more exits most teams forget: a crash that dumps the environment, and the log it lands in.

The process printed its configuration to stderr on the way down, Docker wrote that to the container's JSON log file, and the secret is now sitting on disk in the clear, counted once by grep. An error tracker that attaches the process environment to every exception report forwards it the same way. Before changing how you deliver secrets, find the ones already running. One sweep over every container's environment finds the secret-shaped variables:

Both containers carry the fake password; on a real host this list routinely turns up live API keys. Rotate what it finds, then fix the delivery so the replacement does not land in the same places.

Deliver it as a file

A mounted secret file never enters the environment, so docker inspect, /proc/<pid>/environ and child processes have nothing to show. Compose does this directly: you declare a secret backed by a file on disk, list it on the service, and Compose mounts it read-only under /run/secrets/. Many official images already expect a file through the *_FILE convention, so you point the image at the path instead of giving it the value:

name: lab-runsecrets
 
services:
  db:
    image: postgres:18-alpine
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password   # the image reads the file, once, at first start
    secrets: [db_password]
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
      interval: 2s
      retries: 15
 
  report:
    image: alpine:3.22
    user: "10001:10001"
    depends_on:
      db: { condition: service_healthy }
    secrets:
      - source: db_password
        target: db_password
        uid: "10001"
        gid: "10001"
        mode: 0400
    command: ["sh", "-c", "wc -c /run/secrets/db_password && exec sleep 3600"]
 
secrets:
  db_password:
    file: ./secrets/db_password.txt   # plain text on the host: keep it out of Git and backups

The db service gets POSTGRES_PASSWORD_FILE rather than POSTGRES_PASSWORD; the report service reads the same secret as a non-root user. Create the backing file and bring it up:

The report container could not read the file: its process runs as UID 10001, and the mounted file is owned by the host user that created it. That is the first thing to understand about Compose file secrets outside Swarm. The long-syntax uid, gid and mode keys are Swarm-only and are ignored with a warning (visible in the previous step), so the file arrives with its host ownership and mode, and the reader's UID must match. Look at what actually got mounted and how it reads from outside:

The secret is a read-only bind mount of the host file, not an encrypted object: "not in the environment" is what you bought, not "encrypted at rest", so the backing file needs the same care as any credential (keep it out of Git). The db container's config carries only POSTGRES_PASSWORD_FILE, a path, and the host's view of the postgres process environment holds no password: the entrypoint reads the file, exports the value into its own short-lived init process, and unsets it before the long-running server starts. That last part is image-specific. The postgres image clears it; "Recipe: PostgreSQL, MySQL and MongoDB" (Docker in depth) notes that MySQL's server keeps MYSQL_ROOT_PASSWORD in /proc/1/environ even when you pass it through _FILE. The _FILE convention moves the secret off the command line and out of your Compose file; whether it stays out of the running process's environment depends on the image.

Fix the ownership so the non-root reader can open it, and the report service reads the secret:

Read at startup, and what rotation needs

An image that reads a secret only at first initialisation will not notice a changed file. Query the database with the current secret, then change the backing file and restart the database. The database still has the old password, so a client that uses the file's new contents is rejected:

The first query authenticated; after rotating the file and restarting, a client using the file's new contents is rejected, because postgres set its password from the file only when it first initialised its data directory. Changing a secret file is not rotation on its own. Real rotation needs the credential changed at the source (the database, the cloud IAM system) and the consumer to re-read, which is why the stronger pattern keeps the source of truth outside the container: a secrets manager such as HashiCorp Vault issues short-lived credentials, and a sidecar writes each fresh one into the file the app already reads, on a tmpfs so it never reaches the disk. The credential expires on its own, every fetch is logged, and the container never holds a long-lived secret.

The export anti-pattern

A file-based secret stays out of the environment right up until a startup script copies it back in. This one line, common in hand-rolled entrypoints, undoes the whole thing:

After export DB_PASSWORD="$(cat ...)", the config stays clean, but /proc/<pid>/environ carries the value again and every child inherits it. Have the application open the file itself. If a tool genuinely needs an environment variable, set it inline on that one process's exec line rather than exporting it into the shell, and know it is still readable in that single process's environ.

Keep it out of the image, too

A secret written into the container filesystem at run time lives in the writable layer, and docker commit captures that layer. A tmpfs mount does not, because it never touches the layer:

The first container wrote a token to its root filesystem; docker diff shows the change, and the committed image carries the file for anyone who pulls it. The second wrote to a tmpfs at /run/secrets: docker diff shows only the mountpoint /run/secrets being created (C /run, A /run/secrets), not the token, and the committed image's /run/secrets is empty. Write secrets that a process fetches at run time to a tmpfs. Compose file secrets outside Swarm are not on one: they are read-only bind mounts of a host file, as the mount table above showed, so that file on the host disk is where the secret lives at rest and what you protect (Swarm secrets do land on a tmpfs). Keep build-time secrets out of image layers with BuildKit's --mount=type=secret, which "Build-time secrets and how images leak them" covers. A distroless image with an env-delivered secret still leaks it, because stripping the image does nothing for a password in the container config that docker inspect prints.

Try this

Work through “Keep it out of the image, too” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from runtime secrets, done right, keep “Keep it out of the image, too”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related