A rollout to nginx:1.26 hangs. kubectl get deploy web shows 4 desired but only 3 updated, and the new Pods are Running yet never reach Ready. What's the most likely cause?

The readiness probe is failing, so the controller won't count the new Pods as available. Running-but-never-Ready is the textbook readiness signature. Until a new Pod passes its probe the rollout can't advance, which is the gate doing its job, or a probe pointed at the wrong path or port.

A rollout stalls and the Deployment's progressDeadlineSeconds elapses. What does exceeding that deadline actually do?

It reports failure by setting Progressing=False with reason ProgressDeadlineExceeded. The deadline only flips a condition. The controller carries on reconciling, so the half-finished new ReplicaSet keeps whatever pods it started while the old version serves the rest, until a human runs kubectl rollout undo.

A Deployment has replicas: 4, maxSurge: 25%, and maxUnavailable: 25% (surge rounds up, unavailable rounds down). During a rolling update, what is the most pods that can exist at once and the fewest that stay available?

Up to 5 total, and at least 3 available. Surge is 25% of 4 = 1 rounded up, giving 4 + 1 = 5 max, and unavailable is 25% of 4 = 1 rounded down, giving 4 - 1 = 3 available.

Courses Kubernetes administrationDeployments

Deployments

Versioned rollouts and rollback over ReplicaSets.

Intermediate12 min · lesson 9 of 65

In plain terms

A Deployment is the store manager who rolls out the new uniform one cashier at a time — no one is ever left undressed — and if the new uniform turns out bad, instantly puts everyone back in the old one.

Ship a new version of an app to a hundred users by hand and you get two bad choices. Take everything down, swap the code, bring it back up, and everyone stares at an error page for a minute. Or swap servers one at a time and hope you don't lose the thread halfway through. A Deployment does the second option for you, carefully, and it keeps the previous version on file so it can put it back in seconds. Think of a shift manager handing out new uniforms: one cashier changes at a time, the till is never left unattended, and if the new uniform rips on the first shift, everyone's back in the old one before the next customer walks in. Under the hood a Deployment doesn't touch Pods directly (a Pod is the smallest thing Kubernetes runs, usually a single container). It manages a ReplicaSet, an object whose one job is keeping a fixed number of identical Pods alive, and it spins up a brand new ReplicaSet every time the app changes. That indirection is the whole reason rollouts and rollbacks work.

One line changes, and a controller does the rest

A thermostat doesn't wait for you to flip a switch every time the room cools. You set a target, it reads the actual temperature, and it nudges the two together on its own. The deployment controller runs that same loop over every Deployment in the cluster. You never tell Kubernetes to run a rollout. You edit the desired state, the controller notices the gap between what you asked for and what's actually running, and it closes that gap. When you change the Pod template (almost always a new image tag), the controller fingerprints the new template into a short hash and stamps it onto a label named pod-template-hash. That hash is how it keeps old Pods and new Pods from ever fighting over the same set. It creates a fresh ReplicaSet carrying the new hash, scales it up a bit, scales the old ReplicaSet down a bit, and repeats until the new one owns every replica and the old one sits at zero. The old ReplicaSet is not deleted. It's parked at zero, and that parked object is exactly what makes rollback instant.

terminal

$ kubectl create deployment web --image=nginx:1.25 --replicas=3
deployment.apps/web created
$ kubectl set image deployment/web nginx=nginx:1.26
deployment.apps/web image updated
$ kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 1 out of 3 new replicas have been updated...
Waiting for deployment "web" rollout to finish: 2 out of 3 new replicas have been updated...
deployment "web" successfully rolled out

While that command runs, both ReplicaSets exist side by side. That mid-flight moment is the clearest look you'll ever get at what the controller is doing. Add -o wide and it even shows you which image each one carries and the pod-template-hash baked into its selector.

terminal

$ kubectl get rs -l app=web -o wide
NAME             DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
web-6b47c9f8d4   2         2         1       12s   nginx        nginx:1.26   app=web,pod-template-hash=6b47c9f8d4
web-7d4f8c6b59   2         2         2       6m    nginx        nginx:1.25   app=web,pod-template-hash=7d4f8c6b59

Look at the READY column trailing CURRENT on the new ReplicaSet: two Pods exist, only one is serving. That gap is the safety valve. A new Pod counts as progress only once it passes its readiness probe, a small health check the kubelet (the agent running on every node) keeps running against the container. Leave the probe out and Kubernetes assumes a Pod is ready the instant its process starts. So if your app needs ten seconds to warm up, or it boots on a bad config but keeps the process alive, an unprobed rollout marches right through and swaps every healthy Pod for a broken one. Write a readiness probe that reflects real health and a bad version stalls the rollout instead of quietly finishing it.

The two dials: surge and unavailable

How fast the swap happens comes down to two numbers. maxSurge is how many extra Pods you'll allow above the desired count while rolling, the spare uniforms you can hand out before collecting the old ones. maxUnavailable is how many you'll let drop below the count, how many cashiers can be mid-change at once. Both default to 25%, rounded so surge rounds up and unavailable rounds down. Set maxUnavailable to 0 and maxSurge to 1 and you never drop below the ready count you asked for, one extra Pod added at a time. That buys you Pod-level availability, not zero dropped requests. An old Pod starts shutting down the moment it is told to, while its address is still working its way out of kube-proxy (the per-node component that programs Service routing) and out of your ingress, so in-flight requests still die unless the container also has a preStop hook that waits a few seconds and shuts down cleanly when Kubernetes sends it the stop signal, SIGTERM. The other strategy, Recreate, throws all of that out. It kills every old Pod, then starts the new ones, which means real downtime. You reach for it on purpose, when two versions genuinely cannot run at once, like a database migration that isn't backward compatible.

deployment.yaml

apiVersion: apps/v1
kind: Deployment
metadata:
  name: web
spec:
  replicas: 4
  progressDeadlineSeconds: 120       # report failure if no progress in 2 min
  strategy:
    type: RollingUpdate
    rollingUpdate:
      maxSurge: 1                     # at most 1 extra Pod above 4
      maxUnavailable: 0              # never dip below 4 ready
  selector:
    matchLabels: { app: web }
  template:
    metadata:
      labels: { app: web }
    spec:
      containers:
        - name: nginx
          image: nginx:1.26
          readinessProbe:
            httpGet: { path: /, port: 80 }
            initialDelaySeconds: 3
            periodSeconds: 5

Once it's applied, the fastest way to tell a healthy rollout from a wedged one is describe. The Conditions block is the honest status, and the Events at the bottom are the controller narrating its own scaling decisions.

terminal

$ kubectl describe deployment web
Replicas:               4 desired | 4 updated | 4 total | 4 available | 0 unavailable
Conditions:
  Type           Status  Reason
  ----           ------  ------
  Available      True    MinimumReplicasAvailable
  Progressing    True    NewReplicaSetAvailable
Events:
  Type    Reason             Age   From                   Message
  ----    ------             ----  ----                   -------
  Normal  ScalingReplicaSet  30s   deployment-controller  Scaled up replica set web-6b47c9f8d4 to 1
  Normal  ScalingReplicaSet  20s   deployment-controller  Scaled down replica set web-7d4f8c6b59 to 3

Rollout won't finish: read the signal

kubectl rollout status hangs

new Pods aren't becoming ready

Progressing=False

reason: ProgressDeadlineExceeded

the deadline only reports; the controller keeps trying. Old version still serves the rest. Nothing auto-reverts, so roll back yourself.

CrashLoopBackOff

container starts, then dies

bad image or config. kubectl logs the new Pod, fix, re-apply.

Running, not Ready

readiness probe failing

wrong probe path or port, or the app really is unhealthy. Check kubectl describe pod.

Pending

scheduler can't place the surge Pod

no node has the CPU/memory it requests. Free capacity or lower requests.

Every stuck rollout falls into one of these. Read the Pod phase and the Progressing condition first, before touching any knobs.

Putting the old version back

Every change to the Pod template becomes a numbered revision, and because the old ReplicaSet is still parked at zero, a rollback is just the controller scaling it back up. undo returns to the previous revision; --to-revision jumps to a specific one. Kubernetes keeps only the last ten of these parked ReplicaSets around by default (revisionHistoryLimit), so you can roll back exactly as far as that window reaches and no further. One habit pays for itself here: record why each rollout happened, because the history is close to useless without a reason next to it. The old --record flag is deprecated, so set the change-cause yourself with an annotation before you ship. Revision 1 below shows <none>, because nothing recorded why it happened.

terminal

$ kubectl annotate deployment/web kubernetes.io/change-cause="nginx 1.26 security patch"
deployment.apps/web annotated
$ kubectl rollout history deployment/web
deployment.apps/web
REVISION  CHANGE-CAUSE
1         <none>
2         nginx 1.26 security patch
$ kubectl rollout undo deployment/web --to-revision=1
deployment.apps/web rolled back

progressDeadlineSeconds does not roll back for you

When a rollout stops making progress, the Deployment waits progressDeadlineSeconds (default 600) and then flips the Progressing condition to False with reason ProgressDeadlineExceeded. That is the entire behavior: it reports, and nothing else happens. The controller does not stop reconciling, so the new broken ReplicaSet keeps whatever Pods it managed to start while the old version serves the rest, and if the blockage later clears (the image finally pulls, quota frees up, a node gets capacity) the rollout finishes on its own and Progressing flips back to True. Until then you are running two versions, one of them broken, and only a human typing kubectl rollout undo puts the old one back. Watch rollout status in your pipeline and roll back explicitly on failure. Don't assume the deadline tidies up after itself, because it doesn't.

maxUnavailable and maxSurge decide how painful a rollout feels under load. Defaults are fine until they are not — tune with real capacity in mind.

A stuck rollout is usually readiness, image pull, or a budget that cannot progress. rollout status and describe beat random pod deletes.

Change cause annotations make history readable at 3 a.m. Put the ticket id in them.

Try this

Create a Deployment, set a new image, and watch rollout status. Then undo and confirm the previous ReplicaSet takes traffic again.

terminal

$ kubectl create deployment web --image=nginx:1.25 --replicas=3
deployment.apps/web created
$ kubectl set image deployment/web nginx=nginx:1.26
deployment.apps/web image updated
$ kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 2 out of 3 new replicas have been updated...
deployment "web" successfully rolled out
 
$ kubectl rollout undo deployment/web
deployment.apps/web rolled back
$ kubectl rollout status deployment/web
Waiting for deployment "web" rollout to finish: 1 old replicas are pending termination...
deployment "web" successfully rolled out
$ kubectl get rs -l app=web -o wide
NAME             DESIRED   CURRENT   READY   AGE   CONTAINERS   IMAGES       SELECTOR
web-6b47c9f8d4   0         0         0       4m    nginx        nginx:1.26   app=web,pod-template-hash=6b47c9f8d4
web-7d4f8c6b59   3         3         3       9m    nginx        nginx:1.25   app=web,pod-template-hash=7d4f8c6b59

Takeaway

Deployments version ReplicaSets. RollingUpdate is the default path to a new image; rollback is just pointing desire at an older revision.

ReplicaSetsNamespaces

What Kubernetes isKubernetes fundamentals Security domains & threat modelKubernetes security & hardening The cluster attack surfaceKubernetes attack & defense Kubernetes interview questionsInterview prep

Deployments

One line changes, and a controller does the rest

The two dials: surge and unavailable

Putting the old version back

Try this

Takeaway

Related