Rolling updates & rollbacks

Shipping a new version with no downtime, and undoing it.

Push a new image to a Deployment and watch the traffic graph. A couple of Pods die, a couple of new ones come up, and the line never dips. Nobody gets paged. That's a rolling update, and a Deployment does it by default every time you change its pod template. A Pod, if the word is new to you, is the smallest thing Kubernetes runs: one or more containers that start and stop together. The rolling part is the trick that lets you swap out every one of them without the app ever going dark.

Think of a busy diner changing shifts. You don't lock the doors, send the whole crew home, and hope the next crew clocks in on time. You swap people out a couple at a time so there's always someone at the grill. Kubernetes does the same with Pods. And the reason this lesson matters is that the two things that actually hurt in production, a release that's quietly broken and a release you need to undo right now, both live right here.

What the Deployment is really moving

Here's the part that trips people up. A Deployment doesn't touch Pods directly. Think of a general contractor: you hand them the plan, they don't pick up a hammer, they put a crew on the job. The Deployment is the contractor, a ReplicaSet is the crew, and the Pods are the workers. A ReplicaSet has exactly one job: keep N identical copies of a Pod running, no more and no fewer. When you change the Deployment's pod template (usually a new image), the deployment controller, a control loop running in the cluster's control plane, stands up a brand-new ReplicaSet for the new template and starts shifting replicas from the old crew to the new one, a batch at a time.

How big a batch is set by two knobs. maxSurge is how many extra Pods above your desired count the rollout may create. maxUnavailable is how many of your desired Pods may be missing at any moment. Both default to 25%. So on a 4-replica Deployment, Kubernetes might add one new Pod and drop one old one at a time, keeping at least three serving the whole way through.

kubectl set image deploy/web app=registry.local/app:1.5.0

deployment.apps/web image updated

kubectl get rs -l app=web

NAME             DESIRED   CURRENT   READY   AGE
web-6d4f8b9c7c   0         0         0       9m
web-7f9c5d8b64   4         4         4       40s

Notice the old ReplicaSet is still there, scaled to zero. Kubernetes keeps it on purpose. That empty-but-remembered ReplicaSet is the whole reason a rollback takes seconds instead of a rebuild, which we'll get to.

The readiness probe is the brake

A rolling update is only safe because of one gate: the readiness probe. A readiness probe is a check Kubernetes runs against each new Pod that asks, in effect, "can you take traffic yet?" Until the Pod answers yes, it isn't counted as ready, the Service (the stable internal address that spreads traffic across your Pods) drops it from rotation, and the rollout will not move on to the next batch. It's the new hire who doesn't get put on the floor with real customers until they say they're good to go.

Skip this and the whole safety net is gone. With no readiness probe, a Pod counts as ready the instant its container process starts. So a release that boots, throws an error, and serves 500s to everyone still looks perfectly ready to the controller. The rollout marches to completion and replaces every healthy Pod with a broken one. The mechanics did exactly what you told them. Nothing ever told them the new version was sick.

spec:
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1
      maxUnavailable: 0
  template:
    spec:
      containers:
        - name: app
          image: registry.local/app:1.5.0
          readinessProbe:
            httpGet:
              path: /healthz
              port: 8080
            initialDelaySeconds: 5
            periodSeconds: 5

With maxUnavailable set to 0, Kubernetes refuses to remove a healthy old Pod until a new one reports ready. So if the new version can't pass /healthz, the rollout doesn't tear anything down. It just stops and waits. It waits until progressDeadlineSeconds runs out (600 seconds by default), then the controller marks the rollout failed and rollout status exits non-zero, which is exactly what you want a deploy pipeline to catch. Here is what that looks like on the release after the one above: 1.5.0 landed fine and web-7f9c5d8b64 is serving all four Pods, and the next image you push, app:1.6.0, is broken.

kubectl rollout status deploy/web

Waiting for deployment "web" rollout to finish: 1 out of 4 new replicas have been updated...
error: deployment "web" exceeded its progress deadline

When status says that, don't guess. Look at the Pods. A stuck rollout is almost always one of three things: the new Pods can't schedule (Pending), the container keeps dying (CrashLoopBackOff), or it runs but never passes readiness (Running, 0/1). kubectl get pods tells you which, and kubectl describe pod names the exact reason down in its events.

kubectl get pods -l app=web

NAME                   READY   STATUS             RESTARTS   AGE
web-7f9c5d8b64-2xk9p   1/1     Running            0          12m
web-7f9c5d8b64-8mzt4   1/1     Running            0          12m
web-7f9c5d8b64-qv7rn   1/1     Running            0          12m
web-7f9c5d8b64-h4d8s   1/1     Running            0          12m
web-5c7d94b8f6-lp4qd   0/1     CrashLoopBackOff   4          2m

All four old Pods still serving, one new Pod flailing. The old version is carrying every request while the rollout sits politely stuck. Nothing is down. That's the readiness probe and maxUnavailable:0 doing their job together, and it's the calmest kind of failed deploy you can have.

Undoing a bad release

The old paint cans are still on the shelf. Because that previous ReplicaSet is sitting there scaled to zero, undoing a release just scales it back up and scales the new one down, the same rolling mechanics running in reverse. Nothing gets rebuilt, and the old image is almost always still cached on the nodes, so the previous version is back in seconds. Every template change is recorded as a numbered revision, so you can list them and pick a known-good one.

kubectl rollout history deploy/web

deployment.apps/web 
REVISION  CHANGE-CAUSE
1         <none>
2         <none>
3         <none>
4         <none>

CHANGE-CAUSE is empty because the old --record flag is deprecated (it saved the literal command you ran, which was rarely what you wanted to read back later). If you want a readable history, set it yourself on each change with kubectl annotate deploy/web kubernetes.io/change-cause="...". To reverse the bad release, roll back to a specific revision and confirm it converged.

kubectl rollout undo deploy/web --to-revision=3 && kubectl rollout status deploy/web

deployment.apps/web rolled back
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out

surge pods need spare CPU and memory. A full cluster cannot honor maxSurge and then looks hung.

record or annotate every set image. Future you will thank present you.

Abort a bad rollout early. Waiting for all replicas to go unhealthy is how outages get long.

Try this

Trigger a rolling update, watch pods replace gradually, then roll back. Capture rollout history so you can name the revision you restored.

$ kubectl set image deploy/web app=registry.local/app:1.5.0
 
$ kubectl get rs -l app=web
 
$ kubectl rollout status deploy/web
 
$ kubectl get pods -l app=web
 
$ kubectl rollout history deploy/web
 
$ kubectl rollout undo deploy/web --to-revision=2 && kubectl rollout status deploy/web

Takeaway

Ship by rolling; undo by revision. Readiness gates the old pods leaving. If readiness is wrong, rollouts stall or dump bad traffic.

Related