Secrets & configs in Swarm

Delivering sensitive data to services.

Passing a database password in an environment variable is like taping the alarm code to the front of the monitor. Anyone who walks past reads it. In container terms, "anyone who walks past" means every process running inside the container, every person who can run docker inspect, and every crash log that dumps the environment on its way out. Swarm (Docker's built-in clustering mode, where several machines behave as one) has a better answer. A Docker secret is stored encrypted in the Raft log (the shared state store that the manager nodes keep in sync with each other), sent only to the nodes running a service you granted it to, and delivered into the container as a file in memory under /run/secrets. It never lands on the node's disk. It never gets baked into an image. docker inspect will not print it. You create it once, then hand it out one service at a time.

$ printf 'S3cr3tP@ss' | docker secret create db_password -
q7w8e9r0t1y2u3i4o5p6a7s8d
$ docker secret ls
ID                          NAME          DRIVER    CREATED          UPDATED
q7w8e9r0t1y2u3i4o5p6a7s8d   db_password             13 seconds ago   13 seconds ago
$ docker service create --name db \
    --secret db_password \
    -e POSTGRES_PASSWORD_FILE=/run/secrets/db_password \
    postgres:16
image postgres:16 could not be accessed on a registry to record
its digest. Each node will access postgres:16 independently,
possibly leading to different images being run.
xk2m9p4v7q1nz8b3c6d0f5g7h
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service db converged

Where the app actually reads it

So where does the app look? The secret shows up as a read-only file at /run/secrets/<name>. Only inside containers you granted it. Only on the nodes where those tasks happen to be running. The mount is a tmpfs, a filesystem that lives in the machine's memory instead of on its drive, so the value sits in RAM (the memory that empties when power goes away) and never touches disk. Two things fall out of that. The value is not in the process environment, so running env inside the container leaks nothing. And it vanishes the moment the task stops. Most official images already expect this and accept a variable whose name ends in _FILE pointing at the path, which is why the Postgres service above got POSTGRES_PASSWORD_FILE instead of a plain POSTGRES_PASSWORD.

$ CID=$(docker ps -qf name=db)
$ docker exec $CID cat /run/secrets/db_password
S3cr3tP@ss
$ docker exec $CID sh -c 'mount | grep /run/secrets'
tmpfs on /run/secrets/db_password type tmpfs (ro,relatime)
$ docker exec $CID sh -c 'env | grep -i pass'
POSTGRES_PASSWORD_FILE=/run/secrets/db_password

That last line is the whole trick. The environment carries the path, not the password. The grant model matters just as much. A secret is not ambient data drifting around the cluster for anyone to pick up. It reaches a node only when a task that was granted it gets scheduled there, and it is wiped from that node once the task stops. Move the service and the secret follows the tasks. Break into a worker, then, and you get the secrets for the services running on that worker at that moment. Nothing else in the cluster.

A full stack, wired together by a secret

One-off docker service create commands are fine for a demo. Real deployments describe the whole thing in a Compose file (a YAML text file listing every service and how they connect) and ship it with docker stack deploy, so the state you want lives in version control and can be rebuilt on any cluster. Here is a small stack: Postgres reading its password from a secret, with adminer, a lightweight web front end for poking at databases, sitting in front of it. The secret is loaded from a local file at deploy time. A harmless tuning file rides along as a config. You declare and attach a config the same way you declare and attach a secret. What a config skips is the encryption and the in-memory mount, because there is nothing in it worth hiding.

services:
  db:
    image: postgres:16
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: shop
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    configs:
      - source: pg_tuning
        target: /etc/postgresql/conf.d/tuning.conf
  adminer:
    image: adminer:4
    ports:
      - "8080:8080"
 
secrets:
  db_password:
    file: ./db_password.txt
 
configs:
  pg_tuning:
    file: ./tuning.conf

$ printf 'sh0pP@ss99' > db_password.txt
$ printf 'log_min_duration_statement = 500\n' > tuning.conf
$ docker stack deploy -c stack.yml shop
Creating network shop_default
Creating secret shop_db_password
Creating config shop_pg_tuning
Creating service shop_db
Creating service shop_adminer
$ docker stack services shop
ID             NAME           MODE         REPLICAS   IMAGE          PORTS
a1b2c3d4e5f6   shop_adminer   replicated   1/1        adminer:4      *:8080->8080/tcp
g7h8i9j0k1l2   shop_db        replicated   1/1        postgres:16
$ docker exec $(docker ps -qf name=shop_db) \
    sh -c 'PGPASSWORD=$(cat /run/secrets/db_password) psql -U app -d shop -c "select current_user, current_database();"'
 current_user | current_database
--------------+------------------
 app          | shop
(1 row)
$ docker exec $(docker ps -qf name=shop_db) cat /etc/postgresql/conf.d/tuning.conf
log_min_duration_statement = 500

Rotate without downtime

Secrets are immutable. You cannot edit one in place, and that is deliberate. Every version is its own object with its own ID, so the change leaves a trail somebody can audit months later. To rotate, create the new value under a new name, then swap it in one update: remove the old grant, add the new one, and point it at the same target path the app already reads. The path never moves, so the application needs no reconfiguring at all. Swarm replaces the tasks one at a time, each replacement starting life with the fresh file, and the service keeps answering the whole way through.

$ printf 'N3wP@ss2026' | docker secret create db_password_v2 -
m3n4o5p6q7r8s9t0u1v2w3x4y
$ docker service update \
    --secret-rm db_password \
    --secret-add source=db_password_v2,target=db_password \
    db
db
overall progress: 1 out of 1 tasks
1/1: running   [==================================================>]
verify: Service db converged
$ docker secret rm db_password
db_password

Files in memory, not leftovers in the environment

Swarm delivers small blobs to tasks as files under /run/secrets, and only to the tasks that were granted them. That kills two bad habits at once: baking passwords into images, and the reflex to type -e PASSWORD=… on a run line, which then sits in docker inspect output forever. Configs are the non-sensitive twin for ordinary configuration files. A running task never picks up a new secret version on its own, so treat the redeploy as part of the rotation rather than an afterthought.

A Swarm secret is not a full vault. The manager nodes hold encrypted copies of everything for the whole cluster, which makes a manager a far richer target than a worker. It still beats a plaintext password committed to a compose file in Git by a wide margin. The trade you are making is Swarm's simplicity against the central rotation and audit trail a dedicated secret manager gives you. Use Swarm secrets for credentials that belong to this cluster and nowhere else. Move to Vault or your cloud provider's secret manager when someone starts asking who read which credential and when.

Never bake a secret into an image at build time. Keep it runtime-only, grant it to the specific services that need it and no others, and keep it out of application logs. Pair it with a non-root user and a read-only root filesystem wherever the app tolerates that.

Rehearse the rotation before you need it under pressure: create the new secret version, update the service to reference it, watch the tasks actually restart, then retire the old secret. Write down how your app reacts when the file changes. Some processes read the file once at startup and need a signal or a full restart. Others reopen it on every request.

Keep a short record of each rotation: the old and new secret IDs that docker secret ls printed, which node you ran the update from, and the time on the converge line. If the new password turns out to be wrong, that note is what tells you which object to grant back. The rollback is one docker service update with --secret-rm and --secret-add pointing at the previous secret, and it takes seconds if you wrote the old name down, considerably longer if you deleted it and now have to reconstruct the value from memory.

Try this

Run these on a lab engine with Swarm mode turned on (Docker 24 or newer is fine). Read the sample output first so you know what a healthy result looks like before you rely on any of it in production.

$ echo 's3cret' | docker secret create db_password -
$ docker service create --name db --secret db_password -e POSTGRES_PASSWORD_FILE=/run/secrets/db_password postgres:16
$ docker service ls
NAME   REPLICAS   IMAGE
db     1/1        postgres:16
# inspect does not print the secret body — only the name
# STATUS: READY — password delivered as a file to the task

Takeaway

If you carry two habits out of this lesson, make them the _FILE convention and a version number in every secret name. The first keeps the password out of every environment dump, crash log and inspect output. The second is what stops a stack redeploy from quietly serving yesterday's password while telling you everything went fine.

Related