You ship a new image. The container starts, then fails its health check and never reports ready. Your Deployment has a readiness probe. What happens?

The rollout stops partway, your old Pods carry on serving, and you undo it. The new Pod never passes its readiness probe, so Kubernetes refuses to count it as ready and stops swapping. Your healthy Pods stay up, and kubectl rollout undo puts you back on the good version.

Your Deployment has no readiness probe. You ship a version whose image pulls and starts without complaint, but the app inside is broken and returns errors to every request. What does the rolling update do?

It replaces every healthy Pod with the broken version, because Kubernetes counts a Pod as ready the instant it starts. With no probe, each new Pod is marked ready immediately, so the broken version works its way through all of them.

You run kubectl set image with a mistyped tag. kubectl rollout status prints one line and then hangs, and kubectl get pods shows three old Pods at 1/1 Running plus one new Pod at 0/1 ImagePullBackOff. What is the situation, and what do you do?

The three old Pods are carrying every request and the rollout is safely stalled; kubectl rollout undo puts you back on the last good version. The healthy old Pods take all the traffic while the rollout waits, and undo restores the version that worked.

Courses Kubernetes fundamentalsUpdating without downtime

Updating without downtime

Ship a new version, and undo it.

Beginner10 min · lesson 11 of 24

In plain terms

A rolling update is repainting a fence one plank at a time while people lean on it — never a gap — and the old paint is right there if you need to undo the colour.

A hallway has ten light bulbs, all switched on. You want to swap every one for a brighter model, and you do not want the hallway to go dark for even a second. So you change one bulb, check that it lights, then move to the next. The corridor stays lit the whole way through. Kubernetes updates a running app the same way, one piece at a time, with no moment where the app stops working.

That dark moment has a name. Downtime is any stretch where your app stops answering and people get an error page instead of the thing they came for. Five seconds on a quiet Sunday is annoying. Five seconds during a sale is failed payments and abandoned sign-ups. So here is the goal for this lesson: change the version your users are running, and never once go dark.

What a rolling update actually does

A few words first, because they show up in every line below. A container is your app packed into a sealed box along with everything it needs to run, so it behaves the same on any machine. An image is one frozen version of that box; nginx:1.25 means version 1.25 of nginx, a free web server that a large slice of the internet runs on. A Pod is the smallest thing Kubernetes runs, and for now you can treat it as a wrapper holding one running container. A Deployment is the manager sitting above all of that. Ask it for three Pods and it keeps three alive, replacing any that crash or vanish.

Point the Deployment at a new image and it does not tear down all three Pods and start over. That would leave a window with nobody answering, the exact downtime you are trying to dodge. It runs a rolling update instead. One new Pod starts on the new version. Kubernetes waits until that Pod is healthy. Only then does it remove one old Pod. Then it repeats, a Pod at a time, until every Pod is running the new version. At every point along the way there are enough healthy Pods to take the traffic, so nobody hits an error while you ship.

Here is a complete Deployment you can save and run. It keeps three copies of nginx and carries a readiness probe, which we come back to shortly. YAML is a plain-text format for describing what you want, and it uses indentation (the spaces at the start of a line) to show what sits inside what. Save this as hello-deployment.yaml.

hello-deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello
  labels:
    app: hello
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello
  template:
    metadata:
      labels:
        app: hello
    spec:
      containers:
        - name: web
          image: nginx:1.25
          ports:
            - containerPort: 80
          readinessProbe:
            httpGet:
              path: /
              port: 80
            initialDelaySeconds: 3
            periodSeconds: 5

Send it to the cluster with kubectl, the command-line tool for talking to Kubernetes. Your cluster is the group of machines Kubernetes manages on your behalf, pooled together and treated as one big computer.

apply the Deployment

$ kubectl apply -f hello-deployment.yaml
deployment.apps/hello created

Wait a few seconds, then list your Pods. The READY column counts how many containers inside each Pod have passed their checks and been cleared to receive real user traffic. A line that reads 1/1 means that Pod is open for business.

check the Pods are running

$ kubectl get pods
NAME                     READY   STATUS    RESTARTS   AGE
hello-7c9f8b6d4b-2xk9p   1/1     Running   0          20s
hello-7c9f8b6d4b-8fjq2   1/1     Running   0          20s
hello-7c9f8b6d4b-lm4rt   1/1     Running   0          20s

Ship a new version

Now move the app to a newer nginx. You could edit the YAML file and apply it again, or change it in place with a single command. The container inside the Pod is called web (look at the name field in the manifest above), so web is the thing you point at the new image.

trigger the rolling update

$ kubectl set image deployment/hello web=nginx:1.26
deployment.apps/hello image updated

The swap starts in the background straight away. Watch it with rollout status, which prints a line as each Pod is replaced and reports success only once every new Pod is ready and every old one is gone.

watch it finish safely

$ kubectl rollout status deployment/hello
Waiting for deployment "hello" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 2 out of 3 new replicas have been updated...
Waiting for deployment "hello" rollout to finish: 1 old replicas are pending termination...
deployment "hello" successfully rolled out

One step of a rolling update

When a release goes wrong

That was the happy path. Real versions break, though, and the whole job of a rolling update is keeping that break away from your users. So break one on purpose. A very common slip when you are in a hurry is a typo in the image tag, so point the app at nginx:1.99, a version that was never built.

ship a tag that doesn't exist

$ kubectl set image deployment/hello web=nginx:1.99
deployment.apps/hello image updated

kubectl takes that without a murmur. It checks that your request is well formed, not that the image exists anywhere in the world. The trouble only surfaces when Kubernetes tries to run it. Watch the rollout and you will see it stop making progress.

the rollout stalls

$ kubectl rollout status deployment/hello
Waiting for deployment "hello" rollout to finish: 1 out of 3 new replicas have been updated...

One line, then nothing. No success, no error, no movement. That silence is your early warning. Open a second terminal and list the Pods to find out why.

see which Pod is stuck

$ kubectl get pods
NAME                     READY   STATUS             RESTARTS   AGE
hello-5f6d8c9b74-q4m2t   1/1     Running            0          4m
hello-5f6d8c9b74-t8kzr   1/1     Running            0          4m
hello-5f6d8c9b74-w3xpl   1/1     Running            0          4m
hello-b4c7f9d2a6-p2wqz   0/1     ImagePullBackOff   0          40s

Read that output slowly, because it is the safety net doing its job. The new Pod is stuck at 0/1 with the status ImagePullBackOff, meaning Kubernetes tried to download the image, failed, and is now waiting a while before it tries again. Your three original Pods are still 1/1 Running and still answering every request. The rollout refused to kill a healthy Pod to make room for one that never became ready, so your users saw nothing at all. To find out what went wrong, describe the stuck Pod and read its Events.

read the failure

$ kubectl describe pod hello-b4c7f9d2a6-p2wqz
Name:      hello-b4c7f9d2a6-p2wqz
Status:    Pending
Containers:
  web:
    Image:   nginx:1.99
    State:   Waiting
      Reason:  ImagePullBackOff
Events:
  Type     Reason   Age                 From     Message
  ----     ------   ----                ----     -------
  Normal   Pulling  40s (x3 over 78s)   kubelet  Pulling image "nginx:1.99"
  Warning  Failed   38s (x3 over 76s)   kubelet  Failed to pull image "nginx:1.99": manifest for nginx:1.99 not found
  Warning  Failed   38s (x3 over 76s)   kubelet  Error: ErrImagePull
  Normal   BackOff  12s (x5 over 76s)   kubelet  Back-off pulling image "nginx:1.99"
  Warning  Failed   12s (x5 over 76s)   kubelet  Error: ImagePullBackOff

The Events list at the bottom is Kubernetes writing down everything it tried, and counts like x3 fold the repeated attempts into one line. The line that matters is manifest for nginx:1.99 not found, which is plain English for there is no such version. Nearly every time that means a mistyped tag or an image you forgot to push. Now put things back.

roll back to the last good version

$ kubectl rollout undo deployment/hello
deployment.apps/hello rolled back
$ kubectl rollout status deployment/hello
deployment "hello" successfully rolled out

The broken Pod is thrown away and the good version returns the same careful way, one Pod at a time. The Deployment had kept the previous revision (its saved record of the settings that were working), so there was no rebuild and no scramble at midnight. One command and you are where you started.

A missing image is the easy case. Kubernetes cannot run what it cannot download, so it never got as far as swapping the broken Pod in. The readiness probe stretches that same protection over a nastier case: an image that downloads fine, starts fine, and is quietly broken inside. A readiness probe works like a new hire's first day. Nobody routes live customer calls to them until they have shown they can handle one. In the manifest above, the probe asks each Pod for its home page every five seconds, and a version that answers with errors never counts as ready, so the rollout stalls the same way you watched it stall a moment ago.

How fast that swap moves is a setting, not a fixed rule. A rolling update has two dials. maxSurge is how many extra Pods Kubernetes may run above your replica count while the update is in flight. maxUnavailable is how many of your Pods it is allowed to have missing at the same time. Back in the hallway: maxSurge is how many spare bulbs you may carry in your hands, and maxUnavailable is how many sockets can sit empty before the corridor goes dim. The defaults are deliberately cautious and fine for a three-replica app like this one. Turn maxUnavailable up to make releases faster and you hand back the gap you were trying to close, which is how a bad release becomes an outage.

kubectl rollout undo is the seatbelt. Underneath, every version of your Deployment gets its own ReplicaSet, a small controller whose only job is keeping a fixed number of copies of one exact Pod template running. Rolling back builds nothing new. It scales the previous version's ReplicaSet back up and the broken one back down, which is why undo lands in seconds. Reach for it before you start deleting Pods by hand at 2am. And leave yourself a note while you are calm: a change-cause annotation on the Deployment records why a revision exists, so the history reads like a story instead of a column of numbers.

Say error rates climb five minutes after a deploy and every graph turns red. The order you do things in matters. Check rollout status to see whether the new version even finished, look at the revision history to see what moved, then undo. Working out why the new image is broken can wait until your users are back on a version that answers. Availability first, pride later.

Try this

Run the whole loop yourself on a throwaway Deployment. Ship a new image, label each change so the history means something later, watch the status go green, then undo and confirm the earlier revision came back. Delete the Deployment at the end so nothing is left running in your cluster.

terminal

$ kubectl create deployment roll --image=nginx:1.26 --replicas=3
deployment.apps/roll created
 
$ kubectl annotate deployment/roll kubernetes.io/change-cause='start 1.26' --overwrite
deployment.apps/roll annotated
 
$ kubectl set image deployment/roll nginx=nginx:1.27
deployment.apps/roll image updated
 
$ kubectl annotate deployment/roll kubernetes.io/change-cause='bump 1.27' --overwrite
$ kubectl rollout status deployment/roll
deployment "roll" successfully rolled out
 
$ kubectl rollout undo deployment/roll
deployment.apps/roll rolled back
 
$ kubectl rollout status deployment/roll
deployment "roll" successfully rolled out
 
$ kubectl rollout history deployment/roll
REVISION  CHANGE-CAUSE
2         bump 1.27
3         start 1.26
 
$ kubectl delete deployment roll
deployment.apps "roll" deleted

Takeaway

Three commands carry this lesson: kubectl set image to ship, kubectl rollout status to watch, kubectl rollout undo to retreat. Type them enough times on a scratch Deployment that your fingers know them before an incident does. And the readiness probe in that first manifest is what gives the middle command any meaning, because without one, status will cheerfully tell you a broken release rolled out fine.

Skip the readiness probe and a broken version can replace every healthy Pod

With no readiness probe, Kubernetes counts a Pod as ready the second its container starts, even when the app inside is failing every request. Nothing pauses the rollout, so the broken version walks straight through all of your healthy Pods and takes the app down with it. Add a readiness probe and the rollout waits for the app to genuinely answer, then stops dead on a bad version. From there, kubectl rollout undo puts you back on the last good release.

Self-healingLabels and selectors

Cluster architectureKubernetes administration Security domains & threat modelKubernetes security & hardening The cluster attack surfaceKubernetes attack & defense Kubernetes interview questionsInterview prep