Stacks, secrets and configs

Deploy a Compose file to a swarm, with private images, secrets and configs that rotate.

This Compose file runs with docker compose up on a single host, the way "Multi-container apps with Compose" (Docker for beginners) built such files:

services:
  web:
    build: ./web
    image: lab-private:5000/shop-web:1
    container_name: shop-web
    restart: unless-stopped
    ports:
      - "8080:80"
    depends_on:
      db:
        condition: service_healthy
  db:
    image: lab-reg:5000/postgres:18-alpine
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - db_password
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "postgres"]
  tools:
    image: lab-reg:5000/alpine:3.22
    profiles: [debug]
secrets:
  db_password:
    file: ./db_password_v1.txt

The lab uses the simulated cluster from "Swarm clusters and services", rebuilt with swarm-lab.sh from the lesson files, with both workers labelled disk=ssd:

Handed to that swarm (after creating the password file it refers to), the Compose file does not get past validation:

docker stack deploy and docker stack config read the file with the older Compose file format 3 schema, not the Compose Specification that docker compose uses. The long form of depends_on with a condition is rejected, and so is profiles. Remove both and the file passes, but deploying it shows what else a stack silently drops:

There is no startup ordering in a swarm. Services start in parallel and are restarted until they stay up, so an application must retry its database connection instead of relying on depends_on. docker stack config -c file prints the file as Swarm will read it, with variables substituted and paths made absolute, and is the cheapest check to run in CI before a deploy.

The stack file

A shop stack: two replicas of a web image from a private registry, with its nginx configuration delivered as a Swarm config, and Postgres 18 with its password delivered as a Swarm secret. Postgres 18 keeps its data under /var/lib/postgresql (the image sets PGDATA=/var/lib/postgresql/18/docker), so that is the path the volume mounts; "Recipe: PostgreSQL, MySQL and MongoDB" covers the image itself.

services:
  web:
    image: lab-private:5000/shop-web:1
    ports:
      - "8080:80"
    configs:
      - source: web_conf
        target: /etc/nginx/templates/default.conf.template
    networks: [backend]
    deploy:
      replicas: 2
      update_config:
        parallelism: 1
        delay: 5s
        monitor: 15s
        failure_action: rollback
  db:
    image: lab-reg:5000/postgres:18-alpine
    environment:
      POSTGRES_USER: app
      POSTGRES_DB: shop
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets:
      - source: db_password
        target: db_password
        uid: "70"
        gid: "70"
        mode: 0400
    volumes:
      - pgdata:/var/lib/postgresql
    networks: [backend]
    healthcheck:
      test: ["CMD", "pg_isready", "-U", "app", "-d", "shop"]
      interval: 5s
    deploy:
      placement:
        constraints: ["node.labels.disk == ssd"]
 
networks:
  backend:
    driver: overlay
    attachable: true
 
volumes:
  pgdata:
 
secrets:
  db_password:
    name: shop_db_password_v1
    file: ./db_password_v1.txt
 
configs:
  web_conf:
    name: shop_web_conf_v1
    file: ./web.conf

Secrets and configs get explicit name: values with a version suffix. Without name:, Swarm calls them <stack>_<key> (shop_db_password), and the immutability section below shows why a version in the name matters. The long secret syntax sets uid, gid and mode for the file inside the container: 70 is the postgres user in the Alpine-based image, and 0400 makes the file readable by that user only. The placement constraint is deliberately wrong, as the volume section shows.

Private images and --with-registry-auth

The private registry lab-private:5000 requires a login. Its password file comes with the lesson files; it was created with htpasswd from apache2-utils:

htpasswd -Bbn -C 10 deploy lab-only-password > auth/htpasswd

The registry speaks plain HTTP, and the nodes accept that only because swarm-lab.sh lists the lab subnet under --insecure-registry. With a password on top, that sends the credentials in clear text on every pull; it is tolerable on a private lab bridge only. A real registry needs TLS, and a password-protected registry never belongs in insecure-registries. The login warning is accurate too: without a credential helper the CLI keeps the password base64-encoded in ~/.docker/config.json ("Registries, tags and sharing images" in Docker for beginners covers helpers). The images were built on mgr1 and pushed, then the local copies removed, so every node, including mgr1, has to pull them. First deploy without passing credentials:

The deploy itself succeeds: it only writes the desired state. The tasks are rejected on the nodes with no basic auth credentials, because docker login stored the credentials in the CLI's config on the VM, and the nodes pull for themselves. --with-registry-auth makes the CLI send them along with the service spec:

All three tasks run, and the web service answers through the routing mesh. The credentials were not used once and forgotten. The manager stores them in the service spec in the Raft log and hands them to every node that runs a task of the service. To see that, log out, change the image tag in the file and deploy without the flag:

Both new shop-web:2 tasks were pulled from the private registry, on nodes that had never pulled that tag, with no credentials on the client: the update kept the stored ones. They are not part of what the Engine API returns: docker service inspect, which prints the API's answer, shows no registry credentials at all. They still travel to every node that runs a task of the service, and anyone who controls a manager (root on it, with its Raft data and keys) can recover them. The deploy also printed a warning it did not print with credentials: without them the CLI could not ask the registry for the digest of shop-web:2, so the spec holds the bare tag and each node resolves it on its own (the pinning from "Swarm clusters and services" is lost until a deploy with credentials). Use a dedicated, read-only, pull-only registry token for deployments, not a personal account, rotate it, and redeploy with --with-registry-auth after rotating so the spec carries the new one.

Secrets inside the task

Inside the database container the secret is a file on a read-only tmpfs at /run/secrets/db_password, owned by postgres with mode 0400, as the long syntax asked; without those fields it would be root-owned with mode 0444, readable by every user in the container. Swarm applies these fields when it mounts the secret, which is the behaviour "Recipe: PostgreSQL, MySQL and MongoDB" relies on when it contrasts stacks with plain Compose. The environment holds only the path in POSTGRES_PASSWORD_FILE, which the official image reads at initialisation. docker secret inspect shows the metadata but never the value; there is no command that reads a secret back out of the cluster. Managers keep secrets encrypted in the Raft log (autolock, from "Swarm clusters and services", protects the key), and a worker receives a secret only while it runs a task that was granted it. What can still leak a secret once it is in a container (an entrypoint that exports it, logs, core dumps) is covered in "Runtime secrets, done right" (Advanced container security).

Configs work the same way for non-sensitive files, with two differences: they are mounted directly into the container filesystem rather than on a tmpfs, and anyone with access to a manager's API can read them back (docker config inspect --pretty prints the content). Both secrets and configs are limited to 500 KB.

A local volume lives on one node

pgdata is a local volume, and in a swarm that means local to whichever node runs the task. The constraint node.labels.disk == ssd matches two workers. Write a row, then drain the node the database runs on:

The scheduler did what the constraint allowed: it started the database on the other ssd worker. That node had no shop_pgdata volume, so Docker created an empty one, the Postgres entrypoint initialised a brand-new database in it, and the table is gone from the service's point of view. Both workers now hold a shop_pgdata volume with different contents. Nothing failed; the service is green and empty. The data is still on the first node, which is why pinning the service back to exactly that node brings it back:

Pin a stateful service with local volumes to one node, by a label only that node carries (pgdata=true here) or node.hostname == <host>. If that node is down, the task stays Pending, which is the outcome you want for a database: an alert instead of an empty replacement. The alternatives are a volume driver backed by shared or replicated storage, or Swarm cluster volumes (CSI plugins, available since Engine 23), or running the database outside the swarm. Replication between databases is the database's job, not Swarm's.

Secrets and configs are immutable

A new password goes into the secret file and the stack is redeployed:

The deploy stops with only updates to Labels are allowed. A secret's data cannot change after it is created; the CLI tried to update shop_db_password_v1 in place, the manager refused, and nothing in the stack changed. Configs behave the same way. The way to rotate is a new object under a new name, which is what the _v1 suffix is for. Here the new file becomes version 2, and the stack points the same target at it, so the path inside the container does not change:

Changing the secret in the service spec replaced the database task, and the new task has the new file. The database password, though, did not change:

The Postgres entrypoint reads POSTGRES_PASSWORD_FILE only when the data directory is empty; on every later start it skips initialisation, and the log shows that line for both restarts on wrk1 (after pinning and after the secret change). The secret and the database now disagree. Setting the role's password from the new secret file, through the trusted local socket inside the container, brings them back together:

In production, do it in the order that avoids the outage: change the credential in the database first (or create a second role or password that both versions accept), then roll out the new secret to the clients, then remove the old one. Swarm replaces tasks when a secret reference changes, so an application that reads its secret file only at startup gets the new value on that restart. An application that keeps running across a change never sees it, because the mounted file belongs to the old object. The old secret stays in the cluster until you remove it, and Swarm refuses to remove a secret that a service still uses.

A config follows the same rules. Editing web.conf in place fails; a new name rolls the web tasks with the new file:

Updating and removing a stack

Running docker stack deploy again with the same file and stack name is the update; Swarm compares each service with its spec and changes only what differs, using each service's update_config. Removing a service from the file does not remove it from the cluster unless you add --prune. The deploy also returns as soon as the desired state is written (the "Since --detach=false was not specified" message); --detach=false makes it wait for convergence, which is what a CI job usually wants.

docker stack rm removed the services, the network, and the secrets and configs that carry the stack's label. Both shop_pgdata volumes are still on their workers: a stack never removes volumes, and on a multi-node swarm they are spread over every node that ever ran the task. Removing them is a per-node docker volume rm, after you have checked which one holds the data you care about. The lab removes everything with the cluster:

Try this

Work through “Updating and removing a stack” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from stacks, secrets and configs, keep “Updating and removing a stack”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related