BlogCI/CD

HEALTHCHECK and graceful shutdown in containers

Tell the orchestrator when your container is actually healthy, and handle SIGTERM so deploys don't drop requests.

Aug 27, 2024·4 min readIntermediate·By the SecOpsLog team · command-tested

A container in state Up only means the main process has not exited — not that it can serve traffic, connect to the database, or finish warming a cache. HEALTHCHECK gives the engine (and Compose, Swarm, or orchestrators that honor it) a periodic probe so unhealthy containers leave load balancers before users hit timeouts. The other half of reliable deploys is graceful shutdown: on SIGTERM, stop accepting work, drain in-flight requests, then exit — otherwise every rollout hard-kills mid-request.

You will add a Dockerfile HEALTHCHECK that hits a real readiness endpoint, use exec-form ENTRYPOINT so signals reach your app, implement SIGTERM handling in the process, and align stop grace periods with drain time. Kubernetes probes differ in naming but share the same ideas — container runtime depth is in Docker in depth.

Deploy drain from healthy to stopped

Healthchecks gate startup; SIGTERM handling gates shutdown. Both must be correct or rollouts hurt users.

1Container startprocess up, not ready2HEALTHCHECKprobe /healthz3Status healthyreceive traffic4Deploy signalSIGTERM to PID 15Stop acceptserver.close()6Drain inflightwait for requests7Exit 0before SIGKILL

Tell the platform you are healthy

Point HEALTHCHECK at an endpoint that reflects this process being ready — not every downstream dependency (that belongs in readiness logic inside the app, or a separate probe in Kubernetes). Keep interval, timeout, and retries conservative enough that brief blips do not flap status, tight enough that a wedged process is marked unhealthy within a minute.

Dockerfile
HEALTHCHECK --interval=15s --timeout=3s --start-period=30s --retries=3 \
CMD wget -qO- http://127.0.0.1:8080/healthz || exit 1
# distroless: use a static health binary or HTTP check from orchestrator
bash — health surfaces in statuslive
docker ps --format "table {{.Names}} {{.Status}}"
api Up 2 minutes (healthy)
docker inspect --format="{{.State.Health.Status}}" api
healthy
unhealthy containers can be restarted by orchestration policy

Shut down gracefully

On stop or deploy, the runtime sends SIGTERM, waits stop_grace_period (Compose stop_grace_period, Kubernetes terminationGracePeriodSeconds), then SIGKILL. Your app must catch SIGTERM, close the listening socket, wait for workers to finish, flush buffers, and exit 0. Set grace period longer than your p99 request time plus cleanup.

server.js (signal handling)
const server = app.listen(8080);
process.on('SIGTERM', () => {
console.log('SIGTERM received, draining...');
server.close(() => process.exit(0));
setTimeout(() => process.exit(1), 25000).unref();
});
// Dockerfile: ENTRYPOINT ["node", "server.js"] — exec form, PID 1

Compose: health-gated dependencies

Plain depends_on waits for start, not readiness — your API races the database on boot. Use condition: service_healthy so downstream services start only after HEALTHCHECK passes. Same pattern prevents cascading restarts from marking dependents healthy too early. Set stop_grace_period on the service to match your app's drain budget so Compose sends SIGTERM early enough to finish before the default ten-second kill.

Shell form vs exec form
CMD node server.js
Runs under /bin/sh
PID 1 is shell
SIGTERM may not reach app
Hard-kill on every deploy
ENTRYPOINT ["node", "server.js"]
App is PID 1
Receives SIGTERM directly
Drain works as coded
Required for graceful stop
Shell-form CMD swallows signals
CMD node server.js (shell form) runs under sh as PID 1, which often ignores or mishandles SIGTERM. Use exec form ENTRYPOINT/CMD so your application receives the signal and can drain. Every deploy otherwise drops in-flight requests at the grace-period wall.

Where this goes next

Health and drain discipline carry into Kubernetes as liveness, readiness, and preStop hooks — and into blue-green cutovers where the load balancer must stop sending traffic before SIGTERM. Combine HEALTHCHECK with non-root users, read-only roots, and minimal base images so reliability and security reinforce each other. Docker in depth covers signals, cgroups, and production container behavior end to end.

Go deeper in a courseDocker in depthHealthchecks, signals, Compose production patterns, and engine internals.View course

Related posts