HEALTHCHECK and graceful shutdown in containers

Tell the orchestrator when your container is actually healthy, and handle SIGTERM so deploys don't drop requests.

HEALTHCHECK options, defaults, and what each one counts

Up in docker ps means the main process has not exited; it says nothing about whether the process can answer a request, reach its database or has finished loading its model. HEALTHCHECK is the engine's way of asking the container that question on a schedule, and the table is most of what there is to know about it, with two details that catch people: the first check runs one interval after start unless a start period is set (5.1 s after StartedAt with --interval=5s on the engine used here), and a check that hangs is killed at the timeout and counted as a failure, so a probe that opens a database connection can mark a healthy service unhealthy when the database is slow.

A probe that reports on this process, not on the world

FROM node:22-alpine
# ...
HEALTHCHECK --interval=15s --timeout=3s --start-period=30s --start-interval=2s --retries=3 \
  CMD wget -qO- http://127.0.0.1:8080/healthz || exit 1
 
STOPSIGNAL SIGTERM
ENTRYPOINT ["node", "server.js"]      # exec form: node is PID 1 and receives the signal

/healthz should answer for the process itself: the listener is up, the worker pool is alive, the last startup step finished. Whether the database is reachable is a different question with a different consequence; a container marked unhealthy because its dependency is down gets restarted by a restart policy or an orchestrator and comes back to the same dependency being down, now with a restart storm on top. Dependency health belongs in a readiness signal that removes the instance from traffic without killing it, the job of Kubernetes readiness probes and of a load balancer's target health.

Shutdown: the signal, the grace period, and the shell that eats both

docker stop and every orchestrator do the same thing: send the stop signal (SIGTERM unless STOPSIGNAL says otherwise) to PID 1, wait a grace period, then SIGKILL. Everything that is graceful about a shutdown happens in that window and is the application's job: stop accepting connections, finish what is in flight, flush, exit 0. The shell form of CMD or ENTRYPOINT (CMD node server.js) runs the command under /bin/sh -c, so the shell is PID 1, and a shell does not forward SIGTERM to its child. The application never hears the signal, the grace period expires, and every deploy ends in a SIGKILL that drops whatever was in progress. The exec form in the Dockerfile above is necessary and, on its own, not sufficient: the kernel does not deliver a default-action signal to PID 1, so a process with no SIGTERM handler (sleep, and many small daemons) is killed after the grace period even in exec form. Node installs a handler and exits on SIGTERM; a process that does not needs --init (or tini as the entrypoint), which adds a minimal PID 1 that forwards signals and reaps children.

let healthy = true;                        // read by GET /healthz
const server = app.listen(8080);
 
process.on('SIGTERM', () => {
  healthy = false;                          // /healthz now returns 503: stop attracting new work
  server.close(() => process.exit(0));      // stop accepting; exit when in-flight requests finish
  setTimeout(() => process.exit(1), 8000).unref();   // give up before the platform's SIGKILL would
});

services:
  api:
    image: registry.acme.dev/shop/api:1.4.0
    stop_grace_period: 12s               # longer than the 8 s drain above; default is 10 s
    depends_on:
      db:
        condition: service_healthy       # wait for db's HEALTHCHECK, not just its start
  db:
    image: postgres:17
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U app -d app"]
      interval: 5s
      timeout: 3s
      retries: 10
      start_period: 20s

The drain timeout in the process, the stop_grace_period in Compose and terminationGracePeriodSeconds in Kubernetes are one number seen from three places, and they need to agree: the application gives up slightly before the platform would kill it, so the exit is clean and logged rather than abrupt. depends_on with condition: service_healthy is the other half of the Compose story; plain depends_on orders container start, and an API that starts before Postgres is ready spends its first thirty seconds crash-looping, which the healthcheck on db prevents.

When the healthcheck itself is the outage: recovery in order of preference

The two halves meet at deploy time: blue-green and canary switches assume the old pods finish their requests when their endpoints are removed, and Kubernetes probe design is the same readiness-versus-liveness distinction with the platform's names. A Compose file for production is where the healthcheck, the grace period and the restart policy have to be set together rather than discovered one outage at a time.

Related posts

Quick reference