Self-healing

What happens when something crashes.

A good night watchman walks the building while you sleep. A bulb burns out, he swaps it. A door blows open, he shuts it. By morning the place looks untouched and you never knew anything went wrong. Kubernetes keeps that same watch over your running programs. When one copy of your app falls over, meaning its container (the sealed box holding the program and everything it needs to run) stops, Kubernetes starts a fresh copy without asking you. The habit has a name: self-healing. Crashed container? Restarted. Dead copy? Replaced. Whole machine gone? Its work moves somewhere healthy. Nobody gets paged at 3am for any of it. The cluster (the whole pool of machines Kubernetes runs your apps on) handles it quietly.

What breaks, and who fixes it

Self-healing runs at three levels, and all three work off one idea. You write down what you want. The cluster keeps comparing reality against what you wrote. That written-down wish has a name, desired state, the picture of how things are supposed to look. Say you asked for three copies of your app, always. Kubernetes never stops counting how many are running right now against that three, and it closes any gap the moment one opens.

Level one: a single container crashes. Every worker machine in the cluster (one machine is called a node) has a staff member on duty, a small Kubernetes program called the kubelet that lives on the machine itself. The kubelet watches the containers on its node, and if one dies it starts it again right there, in place. A counter called RESTARTS ticks up when that happens. Level two: a whole Pod dies or gets deleted. A Pod is the smallest thing Kubernetes runs, a thin wrapper around one running copy of your app. When you use a Deployment (the object where you write down how many copies you want), it creates a bookkeeper called a ReplicaSet whose only job is to count Pods and keep the number right. It sees two where three should be, and a replacement is usually up within seconds. Level three: an entire machine fails. The Pods that were sitting on it get recreated on healthy machines, and your app keeps serving from somewhere else.

None of that needs a human. It is also why a single Pod is allowed to be disposable. The older way of running software kept one precious server alive for years, and losing it meant a very bad day and a rebuild from backups. Kubernetes flips that around. Your app stays healthy because copies get replaced constantly, not because any one copy manages to survive.

Watch it heal for yourself

Here is a whole Deployment you can apply and poke at. It is written in YAML, a plain-text format that uses indentation instead of brackets to show what belongs to what. Save it as hello.yaml. It asks for three copies of a small web server. The replicas: 3 line is the entire promise, and everything under template describes what one copy looks like.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: web
          image: nginx:1.27
          ports:
            - containerPort: 80

Send it to the cluster with kubectl, the command-line tool you use to talk to Kubernetes, then list the Pods it made.

kubectl apply -f hello.yaml
kubectl get pods -l app=hello

deployment.apps/hello created
 
NAME                     READY   STATUS    RESTARTS   AGE
hello-7d9c8f5b4-2xk9p    1/1     Running   0          18s
hello-7d9c8f5b4-8vq4m    1/1     Running   0          18s
hello-7d9c8f5b4-lr7cd    1/1     Running   0          18s

Three copies, all Running. Now break one on purpose. Pick any Pod name from that list, delete it, and watch what the cluster does next. The -w flag on the second command means watch, so it keeps printing changes live instead of answering once and quitting.

kubectl delete pod hello-7d9c8f5b4-2xk9p
kubectl get pods -l app=hello -w

pod "hello-7d9c8f5b4-2xk9p" deleted
 
NAME                     READY   STATUS    RESTARTS   AGE
hello-7d9c8f5b4-8vq4m    1/1     Running   0          95s
hello-7d9c8f5b4-lr7cd    1/1     Running   0          95s
hello-7d9c8f5b4-qm2ft    1/1     Running   0          3s     <-- brand-new, count restored to 3

You deleted a Pod. A new one showed up under a different name and the count went back to three. You never ran a create command for it. The ReplicaSet saw two where three should be and closed the gap by itself. That gap-closing is the whole trick, and you watched it happen live. The same loop runs for every kind of failure, whether you broke something by hand or a real crash did it for you.

When healing runs out of road

Deleting a Pod is a polite kind of failure, because the replacement comes up healthy. The failure you hit on your first real app is messier: a container that dies the instant it starts, usually from a wrong image tag, a missing file, or a setting pointing at nothing. Here is a Deployment rigged to fail exactly that way. Its container prints one line, waits two seconds, then exits with an error. Then it does the same thing again.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: crasher
spec:
  replicas: 1
  selector:
    matchLabels:
      app: crasher
  template:
    metadata:
      labels:
        app: crasher
    spec:
      containers:
        - name: app
          image: busybox:1.36
          command: ["sh", "-c", "echo starting; sleep 2; exit 1"]

Apply it, give it a minute to fail a few times, and list it.

kubectl apply -f crasher.yaml
kubectl get pods -l app=crasher

deployment.apps/crasher created
 
NAME                       READY   STATUS             RESTARTS      AGE
crasher-6b9f7c4d8-w2m4k    0/1     CrashLoopBackOff   5 (46s ago)   3m1s

READY says 0/1 and the STATUS is CrashLoopBackOff, the single most common thing a beginner watches go wrong. The kubelet started the container, it died, the kubelet started it again, and RESTARTS has already reached 5. To find out why, ask the Pod to describe itself.

kubectl describe pod crasher-6b9f7c4d8-w2m4k

Containers:
  app:
    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       Error
      Exit Code:    1
    Restart Count:  5
...
Events:
  Type     Reason   Age                   From     Message
  ----     ------   ----                  ----     -------
  Normal   Pulled   3m (x5 over 3m)       kubelet  Container image "busybox:1.36" already present on machine
  Warning  BackOff  15s (x12 over 2m45s)  kubelet  Back-off restarting failed container app

The Last State line is the tell. The container Terminated with Exit Code 1, which means the program inside failed on its own. Down in Events you can watch the kubelet at work, Back-off restarting failed container, over and over. Backoff is the catch. Each retry waits longer than the one before it, ten seconds, then twenty, then up to five minutes, so a broken Pod does not hammer the node forever. And here is the honest limit of self-healing. Restarting cannot fix a bug in your code, a file that is not there, or an image tag you typed wrong. Kubernetes will loop on it patiently, but a Pod stuck in CrashLoopBackOff is the cluster telling you it tried and cannot. Run kubectl logs crasher-6b9f7c4d8-w2m4k --previous to read the dead container's last words, fix the real cause, then apply again. (Point a Deployment at an image name that does not exist and you meet the close cousin of this state, ImagePullBackOff.)

Teaching the cluster what healthy means

So far Kubernetes has healed things that clearly died or clearly refuse to start. There is a subtler blind spot. By default it only knows whether your container's process is running, not whether the app inside is doing anything useful. A web server can be up while it is frozen solid, or while it answers every request with an error. Its Pod still shows Running, so self-healing sees nothing wrong and leaves it alone. You fix that by teaching Kubernetes how to check. A probe is a small health check, a question the cluster asks your app over and over, the way a nurse takes a pulse on the hour. A liveness probe asks whether you are still alive in there. If the answer stops coming, Kubernetes restarts that copy even though the process never technically crashed. A readiness probe asks whether you are ready for traffic yet. If not, the cluster stops sending users to that copy until it says yes. You will wire up both in the next lesson. The encouraging part is that a plain Deployment already hands you a self-healing app, with crashes and dead machines covered out of the box.

Self-healing is not magic, and it pays to know which piece does what. Controllers recreate Pods that vanish. Kubelets restart containers that exit. Neither of them reads your code. A process that crashes in a loop turns healing into CrashLoopBackOff, which is noise rather than health. When a restart counter climbs, read it as an alarm and go fix the crash or the failing liveness probe behind it.

Losing a whole node is the slowest heal of the three. The cluster waits out a built-in grace timer (Kubernetes calls it a toleration timeout) before it accepts the machine is really gone, and only then do its Pods get rescheduled elsewhere. Stateless Deployments come back so smoothly that healing looks free. Anything holding data asks more of you: persistent volumes so the data outlives the Pod, and a stable identity so the replacement knows which copy it is meant to be.

During a real incident, deleting a sick Pod is a fair nudge, as long as something owns it and will make another one. Delete a bare Pod, one created on its own with no Deployment or ReplicaSet behind it, and nothing brings it back. You have taken your own service down by hand. Check the ownerReferences field on a Pod first and you will know which of the two you are holding.

Try this

Start a Deployment, delete one of its Pods, and watch the replacement arrive. Then kill the process inside a container and watch the restart counter move.

$ kubectl create deployment heal --image=nginx:1.27 --replicas=2
deployment.apps/heal created
 
$ POD=$(kubectl get pod -l app=heal -o jsonpath='{.items[0].metadata.name}')
$ kubectl delete pod $POD
pod "heal-…" deleted
 
$ kubectl get pods -l app=heal -w
NAME       READY   STATUS              RESTARTS   AGE
heal-…     1/1     Running             0          30s
heal-…     0/1     ContainerCreating   0          1s
heal-…     1/1     Running             0          3s
# Ctrl+C
 
$ kubectl exec deploy/heal -- nginx -s stop
$ kubectl get pods -l app=heal
NAME     READY   STATUS    RESTARTS   AGE
heal-…   1/1     Running   1          55s
heal-…   1/1     Running   0          40s
 
$ kubectl delete deployment heal
deployment.apps "heal" deleted

Takeaway

Controllers replace missing Pods. Kubelets restart exited containers. That covers a crash or a dead machine the moment it happens, with nothing from you. A RESTARTS count is only good news the first time it moves. Keep watching it climb and the cluster has done everything it can, which leaves the fix to you: read the logs, check the image tag, or write the liveness probe that turns a frozen app into one Kubernetes can actually see.

Related