Runtime secrets, done right

Get secrets into a read-only container without env leaks.

A password you pass with -e DB_PASSWORD=... is not a secret. It is a variable with a scary name, and anyone on the box can read it. Environment variables (small named values the operating system hands a program when it starts) were built to be inherited by child programs and read back by tooling. They were never built to hide anything, and Docker treats them exactly the way they were designed. So get the delivery wrong and everything else you did to this container stops counting: the non-root user, the read-only root filesystem, the distroless base (an image stripped down so far it has no shell and no package manager), the dropped capabilities (Linux splits root's power into roughly 40 separate keys, and you handed most of them back). All that hardening is now standing guard over a password sitting in plain view.

Watch how exposed it really is. One docker run, and the same value lands in three readable places before your app has printed its first log line.

# pass a secret as an env var, then read it back three different ways
$ docker run -d --name app -e DB_PASSWORD=S3cr3t myapp:1.0
9f3c1a2b7e04
 
# 1) docker inspect: readable by anyone in the docker group (root-equivalent)
$ docker inspect app --format '{{json .Config.Env}}'
["PATH=/usr/local/sbin:/usr/local/bin:/usr/bin","DB_PASSWORD=S3cr3t"]
 
# 2) the live process environment, exposed to root on the host
$ sudo cat /proc/$(docker inspect -f '{{.State.Pid}}' app)/environ | tr '\0' '\n' | grep DB_
DB_PASSWORD=S3cr3t
 
# 3) every child process inherits it
$ docker exec app sh -c 'env | grep DB_'
DB_PASSWORD=S3cr3t

Only the third read needed docker exec. The other two got in from outside. docker inspect prints that value to anyone in the docker group, and belonging to that group is the same thing as being root on the host, because the daemon (the background Docker service) runs as root. So "they only have Docker access" and "they have your database password" are the same sentence. The second read is /proc/<pid>/environ (PID means process ID, and /proc is a fake filesystem the kernel exposes so you can read live process state as though it were files). That file is a photograph of the environment the process was handed the moment it started. Unsetting the variable in your code later does not touch the photograph. Nothing here leaked because someone slipped up further down the line. It leaked at process start, and no in-container user and no read-only filesystem changes that.

Find the leaks you already shipped

Before you change how you deliver secrets, find out where the old way is still running. One loop across your running containers pulls out every environment variable that looks like a credential. The list is usually longer than you would guess.

# sweep every running container for secret-shaped env vars
$ for c in $(docker ps -q); do \
    docker inspect "$c" --format '{{.Name}} {{json .Config.Env}}'; \
  done | grep -iE 'pass|secret|token|api[_-]?key'
/app     ["DB_PASSWORD=S3cr3t","REDIS_URL=redis://cache:6379"]
/billing ["STRIPE_API_KEY=sk_live_4eC39HqLyjWDarjtT1zdp7dc","LOG_LEVEL=info"]

That Stripe key starts with sk_live_, so it moves real money. Every account on that host can read it, and so can every child process the billing container spawns. Rotate it now. Then fix how it gets delivered, so the replacement does not land in exactly the same place.

Hand it over as a file, not a variable

An environment variable is like a label taped to the outside of a parcel. Anyone walking past the shelf reads it. A mounted secret file is like a note slid under a locked door, into a room only the app process can enter. The app opens the file once at startup and gets on with its work. The value never enters the environment, so docker inspect has nothing to print, /proc/<pid>/environ has nothing to hold, and the app's children inherit nothing. That is the whole win, and it holds no matter which disk the file physically lives on.

Docker Compose (the tool that runs a group of containers from one YAML file) does this out of the box. You point the app at a path. Compose mounts the file read-only under /run/secrets/, and the container config stores that path instead of the value. Most official images already expect this and look for a *_FILE variable: set DB_PASSWORD_FILE=/run/secrets/db_pw and the image opens the file itself rather than pulling a value out of the environment.

services:
  app:
    image: myapp:1.0
    read_only: true
    environment:
      DB_PASSWORD_FILE: /run/secrets/db_pw   # app reads the FILE; the value is never in env
    secrets: [db_pw]
secrets:
  db_pw:
    file: ./secrets/db_pw.txt                # plaintext on disk, so keep this path out of git

$ docker compose up -d
$ docker compose exec app cat /run/secrets/db_pw
S3cr3t                                             # the app can read it
 
$ docker compose exec app mount | grep /run/secrets
/dev/vda1 on /run/secrets/db_pw type ext4 (ro,relatime)   # read-only bind mount
 
# the same three checks that leaked before now come up clean:
$ docker inspect myapp-app-1 --format '{{json .Config.Env}}'
["PATH=/usr/local/bin:/usr/bin","DB_PASSWORD_FILE=/run/secrets/db_pw"]   # a path, not the value
$ sudo cat /proc/$(docker inspect -f '{{.State.Pid}}' myapp-app-1)/environ | tr '\0' '\n' | grep -i db
DB_PASSWORD_FILE=/run/secrets/db_pw                # harmless: it points at the file

Same secret. The app still reads it. inspect and /proc/environ come up empty. The config now carries a path, which tells an attacker where the app looks but not what it found there. There is one caveat, and it deserves to be said plainly: Compose secrets bind-mount a plaintext file off your disk. What you bought is "not in the environment", not "encrypted at rest". That source file still needs the same handling as any other credential, starting with keeping it out of git.

Keep them out of the image too

Runtime is not the only place secrets escape. A secret you COPY into an image lives in that layer forever and travels to everyone who can pull the image. A build-time ARG (an argument you pass to docker build) is worse than it looks, because the value gets written in clear text into the image history, the stored record of how the image was built. Deleting the file in a later step buys you nothing. Layers stack, they never subtract, so the earlier layer still holds the value. BuildKit, the build engine Docker uses by default, gives you --mount=type=secret, which hands one build step the secret and writes it into no layer at all.

# give one build step the secret without writing it into any image layer
$ export DOCKER_BUILDKIT=1
$ docker build --secret id=npm,src=$HOME/.npmrc -t app:1.0 .
# Dockerfile line:  RUN --mount=type=secret,id=npm,target=/root/.npmrc  npm ci
 ...
 => exporting to image
 => => naming to docker.io/library/app:1.0
 
# detection: grep the finished image's history; the secret isn't in it
$ docker history --no-trunc app:1.0 | grep -i npmrc
$                                          # (no output: nothing was baked in)
 
# a leaky ARG build, by contrast, records the value forever:
$ docker history --no-trunc leaky:1.0 | grep -io 'ARG_TOKEN=[^ ]*'
ARG_TOKEN=ghp_R2d2c3po9a8b7c6d5e4f3g2h1i

Files fix exposure. They do nothing about age. A file is static, so the same value sits there until a human remembers to rotate it. The stronger pattern keeps the source of truth outside the container altogether. A secrets manager such as HashiCorp Vault issues short-lived credentials, and a sidecar (a small helper container that runs alongside your app and shares its lifecycle) such as Vault Agent writes each fresh one into the file the app already reads, ideally onto a tmpfs (a scratch filesystem that lives in RAM and never touches a disk). A stolen credential expires on its own. Rotation happens in one place. Every fetch shows up in an audit log. The container never holds a long-lived secret at all.

Whatever platform you run on, the rule holds. Environment variables leak through docker inspect, /proc, child processes, crash dumps and CI logs (CI means continuous integration, the pipeline that builds and tests your code). Deliver through a tmpfs-mounted secret, your orchestrator's secret store, or Docker secrets on Swarm (Docker's built-in clustering mode), injected at runtime into a read-only path that only the service's UID (user ID) can open.

Rotating means replacing the secret material, not hoping nobody kept a copy of the old environment. If a value ever sat in an env var on a shared host, treat it as burned. And keep secrets out of image layers entirely. You can always issue a new credential, but you cannot recall the copies already sitting on every registry mirror and every laptop that pulled the tag.

Distroless plus an env-delivered secret is still a leak. Stripping the image down and locking the filesystem does nothing for a password parked in the container's config blob, one docker inspect away.

In production this is a check you repeat, not a cleanup you do once. After a change window, re-run the inspect sweep across the hosts you touched, paste the command and its output into the ticket, and refuse to close the change if the reading drifted. Give each workload the narrowest access that still lets it start. That habit compounds across every host and every pipeline you own.

Try this

Run the two deliveries back to back on your own machine and compare what inspect prints. The first container takes the password through -e and shows it off. The second reads the same value from a file under /run/secrets, and its Config.Env never mentions the password at all. Both get cleaned up at the end.

$ docker run -d --name envleak -e DB_PASSWORD=S3cr3t alpine sleep 120
$ docker inspect -f '{{.Config.Env}}' envleak
[DB_PASSWORD=S3cr3t PATH=...]
$ docker rm -f envleak
$ mkdir -p /tmp/secretdemo && printf 'S3cr3t' > /tmp/secretdemo/db_password
$ docker run -d --name fileok --read-only --tmpfs /run/secrets \
  -v /tmp/secretdemo/db_password:/run/secrets/db_password:ro alpine sleep 120
$ docker inspect -f '{{.Config.Env}}' fileok
[PATH=...]
$ docker exec fileok cat /run/secrets/db_password; echo
S3cr3t
$ docker rm -f fileok

Takeaway

Deliver secrets as runtime files on a tight tmpfs or a platform secret mount, never as environment variables and never baked into image layers. If docker inspect can print your database password, you do not have a secret.

Related