>_SecOpsLog_
CoursesKubernetes fundamentalsHealth checks

Health checks

Liveness and readiness, gently.

Beginner10 min · lesson 13 of 24
In plain terms
Health checks are a nurse asking two questions: “are you alive?” (restart if not) and “ready for customers?” (hold the queue if not). Small checks that prevent big outages.

An app can be running and still be useless. The process is alive, the container has not crashed, and yet the thing is frozen: stuck waiting on a database that never answers, or so swamped it cannot take another request. Kubernetes cannot see any of that by itself. Left alone, the only question it can answer is whether your program is still a process. Health checks are how you teach it to ask better questions.

A nurse doing rounds on a hospital ward stops at every bed and asks two things: are you alive, and are you well enough to see visitors? That is the shape of a health check in Kubernetes. The technical name is a probe, a small check the cluster repeats on a timer, and it asks your app those same two questions over and over for as long as it runs.

One piece of the cluster does the walking. On every node (a machine where Pods run) there is a kubelet, a small agent that starts your containers and keeps an eye on them. The kubelet is the nurse on rounds. It runs the probes, reads the answers, and acts on them.

Liveness and readiness

"Are you alive?" is the liveness probe. When it fails, Kubernetes concludes the app has hung and restarts the container to give it a clean start. That is how a frozen app gets rescued even though its process never actually crashed.

"Are you well enough for visitors?" is the readiness probe. When it fails, Kubernetes stops sending that Pod traffic and leaves it running. No restart, no drama. It waits for the Pod to report ready again. A Service (Kubernetes' built-in load balancer across your Pods) quietly routes around the unready one while that happens.

The two pull in opposite directions. Liveness unsticks a hung app by restarting it. Readiness keeps callers away from a Pod that is in no state to answer, whether it is still booting or briefly buried in work. Most of the time you want both.

A probe can ask its question in three ways. It can send an HTTP request (the same kind of web request your browser sends) to a health URL and pass when a good response comes back. It can knock on a port with a plain TCP connection: TCP, short for Transmission Control Protocol, is the ordinary way two programs open a network conversation, and a port is the numbered slot a program listens on. Or it can run a command inside the container and pass when that command exits cleanly. Real apps usually serve small dedicated URLs for this, /healthz for liveness and /ready for readiness, so each probe checks exactly what it is meant to.

Try it

Here is a Deployment you can apply as-is. A Deployment is the object that keeps a set of identical Pods running for you; this one runs two copies of nginx (a small, dependable web server) with both probes wired up. Save it as probes-demo.yaml. YAML is a plain-text format where indentation carries meaning, so copy the spacing exactly as shown.

probes-demo.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
  labels:
    app: web
spec:
  replicas: 2
  selector:
    matchLabels:
      app: web
  template:
    metadata:
      labels:
        app: web
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 5
            periodSeconds: 10
          livenessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 10
            periodSeconds: 10

Two settings do the work here. initialDelaySeconds is how long Kubernetes waits after the container starts before it probes at all, which gives a slow app room to boot. periodSeconds is how often it checks after that. So readiness starts at 5 seconds, liveness at 10, and both repeat every 10. Both point at /, a page nginx already serves, so the demo needs nothing else set up. A real app would point them at dedicated URLs like /healthz and /ready.

Send it to the cluster with kubectl, the command-line tool you use to talk to Kubernetes.

Terminal
kubectl apply -f probes-demo.yaml
Output
deployment.apps/web created

Now see what came up.

Terminal
kubectl get pods -l app=web
Output
NAME                   READY   STATUS    RESTARTS   AGE
web-6d4c8f5b7c-2xk9p   1/1     Running   0          22s
web-6d4c8f5b7c-8ftlm   1/1     Running   0          22s

READY shows 1/1, meaning the one container in each Pod is passing its readiness probe. For the first few seconds it would have read 0/1, while the probe had not passed yet. That is Kubernetes holding traffic back on purpose until the app says it can cope.

To see how the probes are set up, and what the kubelet has been doing about them, describe one of those Pods by name.

Terminal
kubectl describe pod web-6d4c8f5b7c-2xk9p
Output
Name:      web-6d4c8f5b7c-2xk9p
Namespace: default
Status:    Running
Containers:
  web:
    Image:          nginx:1.27
    Port:           80/TCP
    State:          Running
    Ready:          True
    Restart Count:  0
    Liveness:       http-get http://:80/ delay=10s timeout=1s period=10s #success=1 #failure=3
    Readiness:      http-get http://:80/ delay=5s timeout=1s period=10s #success=1 #failure=3
Events:
  Type    Reason     Age   From               Message
  ----    ------     ----  ----               -------
  Normal  Scheduled  41s   default-scheduler  Successfully assigned default/web-6d4c8f5b7c-2xk9p to node-1
  Normal  Pulled     40s   kubelet            Container image "nginx:1.27" already present on machine
  Normal  Created    40s   kubelet            Created container web
  Normal  Started    40s   kubelet            Started container web

The Liveness and Readiness lines spell out each probe's settings. That #failure=3 means Kubernetes puts up with three checks failing back to back before it does anything, so a single blip costs you nothing. Below them, the Events are the kubelet's own account of how this Pod got running.

When liveness turns on a healthy app

That tolerance of three matters, because a wrong liveness probe is the one that bites. Every failure it counts moves the container closer to a restart. The classic beginner slip is copying the tidy /healthz pattern onto an app that does not serve it. Plain nginx has no /healthz page, so it answers every check with a 404 (the HTTP code for "no such page"), and a probe reads 404 as a failure. Aim liveness there and Kubernetes will restart a perfectly healthy app, again and again. Here is that mistake in one manifest.

liveness-too-strict.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-loop
  labels:
    app: web-loop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-loop
  template:
    metadata:
      labels:
        app: web-loop
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
          livenessProbe:
            httpGet:
              path: /healthz
              port: 80
            periodSeconds: 5
            failureThreshold: 2
            timeoutSeconds: 1

Readiness still points at /, which nginx serves, so it keeps passing. Liveness points at /healthz, which nginx has never heard of. And periodSeconds: 5 with failureThreshold: 2 means two failed checks are all it takes to trigger a restart. Apply it, give it a minute or two, then list the Pod.

Terminal
kubectl apply -f liveness-too-strict.yaml
Output
deployment.apps/web-loop created
Terminal
kubectl get pods -l app=web-loop
Output
NAME                        READY   STATUS    RESTARTS      AGE
web-loop-5f8c6d9b74-7t2qn   1/1     Running   4 (12s ago)   3m28s

Read that row carefully. READY is 1/1 and STATUS is Running, so the app is up and passing readiness. Yet RESTARTS reads 4 and keeps climbing. A healthy Pod with a restart count that will not stop rising is the fingerprint of a liveness probe firing at an app that is fine. Describe the Pod and the reason is right there.

Terminal
kubectl describe pod web-loop-5f8c6d9b74-7t2qn
Output (trimmed to the Events)
Events:
  Type     Reason     Age                From     Message
  ----     ------     ----               ----     -------
  Warning  Unhealthy  9s (x11 over 3m)   kubelet  Liveness probe failed: HTTP probe failed with statuscode: 404
  Normal   Killing    9s (x4 over 3m)    kubelet  Container web failed liveness probe, will be restarted

Two lines tell the whole story. Liveness probe failed ... statuscode: 404 says the check is wrong, not the app: nginx returned 404 because there is no /healthz for it to serve. Killing ... will be restarted is the kubelet doing exactly what you asked of it, the same agent that keeps your app alive now knocking it down. The fix is to point liveness at something the app genuinely serves. If the app is slow to boot, add a startupProbe as well: Kubernetes runs that check alone at first and holds liveness back until it passes, so a long start is never mistaken for a hang.

liveness-fixed.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: web-loop
  labels:
    app: web-loop
spec:
  replicas: 1
  selector:
    matchLabels:
      app: web-loop
  template:
    metadata:
      labels:
        app: web-loop
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
          startupProbe:
            httpGet:
              path: /
              port: 80
            failureThreshold: 30
            periodSeconds: 2
          livenessProbe:
            httpGet:
              path: /
              port: 80
            periodSeconds: 10
            failureThreshold: 3
Terminal
kubectl apply -f liveness-fixed.yaml
Output
deployment.apps/web-loop configured
Terminal
kubectl get pods -l app=web-loop
Output
NAME                        READY   STATUS    RESTARTS   AGE
web-loop-6c7b5d4f98-nq8pd   1/1     Running   0          38s

Same app, same image, one probe changed, and the restarts stopped. Liveness is a powerful switch. Point it at something the app never serves and you have built yourself a machine for killing healthy Pods.

Where this pays off

You have already met rolling updates, where Kubernetes swaps old Pods for new ones a few at a time. Readiness is what makes that safe. A new Pod counts as ready only once its probe passes, so a broken version stalls the rollout instead of quietly taking the place of every healthy Pod you had. Day to day, readiness keeps traffic off Pods that are still starting or briefly overloaded, so nobody on the other end sees an error. Liveness works away in the background, restarting apps that have silently hung, catching the failures that plain crash-detection sails straight past.

Diagram
kubelet probes your Pod
two checks, repeating on a timer
liveness fails
Restart the container
the app hung; a fresh start unsticks it
readiness fails
Hold back traffic
not ready yet, so the Service routes around it, no restart
both pass
Leave it alone
alive and ready, serving normally

Keep the two questions apart in your head and most probe decisions answer themselves. Liveness: should this container be restarted? Readiness: should this container get traffic? A startupProbe buys a slow app time before liveness is allowed to swing. And make readiness mean something. A probe that lightly touches a real dependency tells you the truth, while one hard-coded to answer 200 OK will keep insisting everything is fine while your app drowns.

There is a nastier version of the same trap. If your liveness probe has to take a lock that gets contended under load, heavy traffic makes the probe time out, and Kubernetes starts restarting Pods that were only busy. That is an outage you inflicted on yourself. Set periodSeconds and failureThreshold against what production traffic actually looks like, not against a quiet laptop cluster.

One habit for when you are on call. If a Service has no endpoints while its Pods all say Running, do not reach for the scale command. Check the Ready condition and the readiness probe failures in the describe events first. More Pods that also fail readiness gets you more of nothing.

Try this

Your turn. Add a readiness probe to a running Deployment, watch the Pod flip to Ready only once the probe passes, then check that its address shows up in the endpoint list. For a second round, point the path at something nginx does not serve and watch those endpoints empty out.

terminal
$ kubectl create deployment probed --image=nginx:1.27 --replicas=1
deployment.apps/probed created
 
$ kubectl patch deployment probed --type='json' -p='[
  {"op":"add","path":"/spec/template/spec/containers/0/readinessProbe","value":{"httpGet":{"path":"/","port":80},"initialDelaySeconds":2,"periodSeconds":5}}
]'
deployment.apps/probed patched
 
$ kubectl get pods -l app=probed -w
NAME       READY   STATUS    RESTARTS   AGE
probed-…   0/1     Running   0          3s
probed-…   1/1     Running   0          8s
# Ctrl+C
 
$ kubectl get endpoints probed
NAME     ENDPOINTS        AGE
probed   10.244.1.30:80   20s
 
$ kubectl delete deployment probed
deployment.apps "probed" deleted

Takeaway

The skill worth carrying out of this lesson is reading the symptoms. A Pod that is Ready with a climbing RESTARTS count means liveness is pointed somewhere wrong. Running Pods behind a Service with no endpoints mean readiness never passed. And a probe wired to answer yes no matter what will hide both of those from you until the pager goes off.

Reach for readiness first
Readiness is the safe one. All it can do is move traffic away from a Pod, and it can never restart anything, so lean on it. Keep liveness gentle and aim it at a signal the app really serves. Hand anything slow to boot a startupProbe. If you are unsure about a liveness probe, leaving it off does less damage than getting it wrong.
Quick check
01A Pod reads READY 1/1 and STATUS Running, but its RESTARTS number ticks up about once a minute. You curl the app yourself and it answers fine. What is most likely going on?
Incorrect — Eviction moves a Pod off the node and reschedules it somewhere else. It does not quietly pile up restarts on a Pod that stays Running and Ready.
Correct — Readiness passing (1/1) proves the app answers, so the restarts come from a liveness probe pointed at the wrong path or timed too tightly. kubectl describe pod shows the 'Liveness probe failed' events.
Incorrect — Two problems with that. Readiness never restarts anything, and here it is passing anyway (1/1). Restarts only ever come from liveness.
Incorrect — A healthy Pod sits at 0 restarts no matter how much work it is doing. A rising count always means something is killing the container and starting it again.
02The lesson says readiness is what keeps a rolling update safe. What is it actually doing?
Incorrect — Wrong probe. Readiness never restarts anything (that is liveness), and it has no say over versions.
Incorrect — Readiness has no effect on how many Pods are swapped at a time or how fast it goes. It only decides whether a Pod counts as ready.
Correct — Readiness gates progress, so a version that never turns ready halts the swap and leaves the old Pods in place.
Incorrect — Readiness decides whether a Pod receives traffic at all. It has no preference for the newest one.
03In liveness-fixed.yaml the startupProbe carries failureThreshold: 30 and periodSeconds: 2, and liveness only begins checking once the startupProbe has passed. Roughly how long does a slow app get to boot before liveness can restart it?
Incorrect — That is one period between checks, not the whole startup budget.
Correct — The budget is failureThreshold times periodSeconds, so 30 x 2 gives 60 seconds of grace before liveness comes into play.
Incorrect — The sum is failureThreshold multiplied by periodSeconds (30 x 2), not the threshold on its own.
Incorrect — After 30 failed checks the startup probe itself fails and the container is restarted, so the wait has an end.
Labels and selectorsConfigMaps: settings outside the…

Related

Cluster architectureKubernetes administrationSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep