Kubernetes liveness, readiness, startup probes done right

Stop routing traffic to pods that aren't ready and restarts that make outages worse — the three probes explained.

The kubelet knows a container's process is running. It does not know whether the process can serve a request, whether it is deadlocked, or whether it is thirty seconds into a ninety-second cache warm-up. Probes answer those questions, and each of the three answers has a different consequence, which is the only thing that matters when choosing which check goes where.

Three probes, three consequences

The failure that turns a blip into an outage

A liveness handler that checks the database looks thorough and is a fleet-wide restart waiting to happen: the database pauses for a failover, every replica's liveness fails three times, the kubelet restarts all of them at once, and the service that would have recovered in ten seconds now spends two minutes rebuilding connection pools while the incident channel fills with a Kubernetes-shaped explanation of a database event. Readiness is the probe that may look outward, because its failure mode is stop sending me traffic, which is exactly right while a dependency is down. Liveness may only look inward.

readinessProbe:
  httpGet: { path: /ready, port: 8080 }   # checks dependencies and warm-up
  periodSeconds: 5
  failureThreshold: 3
livenessProbe:
  httpGet: { path: /healthz, port: 8080 } # answers only: is this process responsive
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
  terminationGracePeriodSeconds: 20       # a wedged process gets 20 s, not the Pod's 60 s
startupProbe:
  httpGet: { path: /healthz, port: 8080 }
  periodSeconds: 5
  failureThreshold: 36                    # up to 180 s to come up before anything else runs

The startup probe is what makes a slow-booting JVM or .NET service survivable without loosening liveness for its whole life: during the failureThreshold × periodSeconds window only the startup probe runs, and once it succeeds the tight liveness settings take over. The run put two Pods with a 20-second boot next to each other under the same liveness probe (three failures, two seconds apart): the one without a startup probe was restarted after 9 s, before it had served anything, and was sitting in CrashLoopBackOff with two restarts a little later; the one with a startup probe was Ready after 12 s with zero restarts. The per-probe terminationGracePeriodSeconds on the liveness and startup probes shortens the shutdown for a process the kubelet is restarting because it is unresponsive; the Pod-level grace period still applies to ordinary terminations, where the process is expected to drain. That number is visible in the restart timing: with terminationGracePeriodSeconds: 5 on the probe, a liveness failure turned into a restarted container in 13 s (three failed probes, then the grace period); a busybox httpd that ignores SIGTERM would otherwise have waited out the Pod's 30 s.

How probes behave during a rollout and a shutdown

During a rolling update a new Pod failing readiness is the system working: it stays out of the endpoints until it is ready, and maxUnavailable keeps the old Pods serving; the run rolled out a template that never creates its readiness file, rollout status timed out with the new Pod at ready=false and both old Pods still ready=true in the slice, and kubectl rollout undo brought the Deployment back to two ready endpoints. A new Pod failing liveness during a rollout is a misconfiguration: the kubelet restarts it before it ever joins the Service, and the rollout stalls in CrashLoopBackOff with a healthy previous version still running, which is the moment to check whether a startup probe is missing. On the way out the order reverses: at termination the Pod is removed from endpoints and sent SIGTERM at about the same time, so a preStop hook that sleeps a few seconds, or an application that fails readiness on SIGTERM and keeps serving for the drain period, is what stops in-flight requests from hitting a Pod that is already gone from the load balancer's point of view.

When a probe change takes the service down: recovery in order of preference

Probe types, and the one to avoid

httpGet is the default choice: any 2xx or 3xx is success, and the handler can be as cheap as returning a static 200 for liveness. tcpSocket only proves the port is open, which a deadlocked server with an accept queue still passes. grpc probes speak the gRPC health protocol to services that have no HTTP endpoint. exec runs a command in the container, forks a process every period, and needs a shell or binary that a distroless image does not have; it is the probe type to justify rather than default to. Whatever the type, the check must be cheap and must not itself allocate the resources it is checking for.

Probes decide whether a Pod gets traffic and whether it is restarted. They do not decide whether it can be scheduled or how much it may consume (requests and limits take over there), nor how many replicas a node drain leaves running, the job of a PodDisruptionBudget. An image with no shell changes the probe choice too: the exec type is unavailable there, a limitation distroless images turn into a feature.

Related posts

Quick reference