Docker Compose in production: the parts that bite

Restart policies, health-gated dependencies, resource caps, and secrets — what to fix before Compose meets real traffic.

A compose.yaml that works on a laptop has usually been shaped by three habits that do not survive a production host: depends_on: [db] (which waits for the database container to start, not for the database to accept connections), credentials in .env (which docker inspect prints to anyone with socket access), and :latest tags (which make every restart a surprise upgrade). Compose can run a small production stack on one VM for years; the file just has to be written for reboots, crashes and the person who will inspect it, and not for the demo.

The file, production-shaped

services:
  db:
    image: postgres:17-alpine
    restart: unless-stopped
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/db_password
    secrets: [db_password]
    volumes: [pgdata:/var/lib/postgresql/data]
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 10s
      timeout: 5s
      retries: 5
      start_period: 20s
    logging:
      options: { max-size: "20m", max-file: "5" }
 
  api:
    image: ghcr.io/acme/api:1.4.2
    restart: unless-stopped
    depends_on:
      db: { condition: service_healthy }
    healthcheck:
      test: ["CMD", "/app", "healthcheck"]
      interval: 15s
      retries: 3
    deploy:
      resources:
        limits: { cpus: "1.0", memory: 512M }
    read_only: true
    tmpfs: [/tmp]
    secrets: [db_password, api_token]
    logging:
      options: { max-size: "20m", max-file: "5" }
 
secrets:
  db_password:
    file: ./secrets/db_password.txt
  api_token:
    file: ./secrets/api_token.txt
 
volumes:
  pgdata:

Each line above answers a specific failure. condition: service_healthy makes Compose wait for the database healthcheck instead of the container start, so the API does not race the database on every reboot; in the run the api container's StartedAt was half a second after the db's first passing probe. start_period gives Postgres time to initialise on first boot before failed probes count. restart: unless-stopped brings services back after a crash or a host reboot but respects a deliberate docker compose stop during maintenance, which always does not. A crash means the process exiting on its own: when the fixture's api exited 3, Compose's policy had it running again within a second with RestartCount 1; after docker kill it stayed exited with code 137, because a kill from the socket counts as a stop. deploy.resources.limits applies cgroup limits under plain Compose too (the api container carried Memory=536870912 and NanoCpus=1000000000 in docker inspect), so a memory leak in the API is an OOM kill of one container rather than of the host. read_only with a tmpfs for /tmp and log rotation on every service are the two settings people add after the incident instead of before; touch /marker in the api container answered Read-only file system and /tmp/marker worked.

Secrets as files, and what Compose will not do for them

A value in environment: appears in docker inspect, in /proc/<pid>/environ, in crash dumps and in any child process that inherits the environment. Compose secrets mount the value as a file under /run/secrets/<name> instead, and most images that matter accept a _FILE variant of their credential variable (POSTGRES_PASSWORD_FILE above). How far to trust this depends on where the secret comes from, and three things need keeping apart. What the Compose file reference says: the secret is mounted read-only, the long-syntax mode defaults to 0444, and uid, gid and mode are implemented only when the secret's source is environment; with a file source Compose uses a bind mount and the three attributes are, in the documentation's words, silently ignored. What Docker Compose v2.40.3 did in the run: a file: secret was a read-only bind mount of the host file with no mode of Compose's own, and the three attributes on it were ignored with a warning printed at up rather than silently; an environment: secret was a file Compose wrote into the container, 0444 root-owned by default, 0400 and 1000:1000 when asked. What the platform added: the run happened on a macOS host whose Linux VM presents shared files as root-owned, so the file secrets showed up as uid=0 gid=0 mode=644; on a native Linux filesystem (checked once in a Docker-in-Docker daemon, outside the fixture) the container sees the host file's exact owner and mode, so a service running as 1000:1000 read a 600 file it owned and could not read a root-owned 640 one. The conclusion is the same on both: for file-backed secrets the host file is the permission model. Protect it there (chmod 600 and the right owner, on an encrypted disk), give a service that runs unprivileged a uid or group that can read it, and treat this as a single-host arrangement: a multi-host deployment wants an external secret store.

Operating it: upgrade, verify, roll back

Pinned tags mean upgrades happen when you change the file, which is the point: the change is a commit, the rollout is pull then up -d, and the rollback is the previous commit. docker compose config renders the merged file and catches YAML and interpolation mistakes before anything restarts (a ${API_URL:?API_URL must be set} left unset came back as required variable API_URL is missing a value, exit 1); docker compose ps shows health, not just running state, and is the check to run before declaring the upgrade done. One check the runbook should not skip on first deploy: that the _FILE variable was read at all. The postgres image trusts connections from inside its own container, so a psql there proves nothing about the password; the fixture logged in from another container on the Compose network with the file's value (scram, accepted) and with a wrong one (password authentication failed for user "app", exit 2).

The rollback in that last line only holds if the new version did not migrate the database schema forward. A release that runs migrations needs its own backward-compatibility plan (expand-then-contract, or a backup taken before up -d), because git revert restores the old binary against a schema it may not understand. That is a property of the application, not of Compose, but Compose gives you nothing to hide behind: there is no rolling update and no second replica to keep serving while the first one fails.

When an upgrade or a secret change goes wrong: recovery in order of preference

Related posts

Quick reference