Upgrading a cluster

The kubeadm upgrade sequence, in order.

One rule decides how every cluster upgrade goes: a node's agent may run behind the control plane, never ahead of it. That agent is the kubelet, the program sitting on each machine that actually starts and stops your containers. The control plane is the brain of the cluster. The single piece everything talks to is the API server, the front door for every command you type and every check-in a kubelet makes. A kubelet is allowed to lag a few versions behind the API server. It is never allowed to be newer. That one constraint is the whole reason you upgrade the brain first and the workers second. Climbing one minor version per step is a separate rule, and it comes from kubeadm: it only supports a single-minor upgrade path, and the control-plane pieces have to stay far closer together than the kubelets do.

Think of repainting a long bridge while cars keep crossing. You close one lane, paint it, reopen it, then move to the next. The bridge never fully shuts. A cluster upgrade works the same way. kubeadm, the tool that set the cluster up in the first place, turns the job into a fixed sequence: upgrade the control plane, then pull each worker machine out of rotation for a minute, upgrade it, put it back, and repeat. Doing one machine at a time leaves somewhere for the evicted pods to land. (A pod is Kubernetes' wrapper around one or more containers, the smallest thing the cluster schedules.) One task comes before you touch anything else: take an etcd snapshot. The upgrade rewrites the control plane, and etcd, the key-value database that stores every object in the cluster, is the only part you cannot rebuild from your manifests and images. The backup command gets its own lesson. Just never start an upgrade without a fresh snapshot.

You start on the first control-plane machine. Installing the new kubeadm takes two steps people forget. The package repository is pinned to one minor version, so a 1.30 machine cannot even see the 1.31 packages until you point its repo file at v1.31. Skip that and apt answers that version 1.31.2-1.1 for kubeadm was not found. The kubeadm, kubelet and kubectl packages are also held on purpose, so apt refuses to move them until you unhold. Switch the repo, unhold, install the exact version, hold again. Then ask kubeadm what a safe move looks like. kubeadm upgrade plan reads the running versions and prints the exact version you can jump to, plus any preconditions it wants met first. It changes nothing. Treat it as the dry run.

# the repo is pinned per minor version: point it at 1.31 first
$ sudo sed -i 's|/v1.30/|/v1.31/|' /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-mark unhold kubeadm
$ sudo apt-get install -y kubeadm=1.31.2-1.1
$ sudo apt-mark hold kubeadm
$ sudo kubeadm upgrade plan

[upgrade/versions] Cluster version: v1.30.4
[upgrade/versions] kubeadm version: v1.31.2
 
Components that must be upgraded manually after you upgrade the control plane:
COMPONENT   NODE     CURRENT   TARGET
kubelet     cp-1     v1.30.4   v1.31.2
kubelet     node-1   v1.30.4   v1.31.2
 
Upgrade to the latest stable version:
COMPONENT                 NODE   CURRENT    TARGET
kube-apiserver            cp-1   v1.30.4    v1.31.2
kube-controller-manager   cp-1   v1.30.4    v1.31.2
kube-scheduler            cp-1   v1.30.4    v1.31.2
etcd                      cp-1   3.5.12-0   3.5.15-0
 
You can now apply the upgrade by executing:
  kubeadm upgrade apply v1.31.2

The plan checks out, so you apply it. kubeadm upgrade apply swaps the control-plane pieces in one shot: the API server, the scheduler (which picks the node each pod runs on), the controller-manager (which keeps the running cluster matching what you declared, like a thermostat nudging the room back toward the temperature you set), and etcd. Here is the part people miss. These pieces run as static pods, meaning their definitions live as plain files in /etc/kubernetes/manifests on the machine. kubeadm rewrites those files to point at the new image versions, and the kubelet, which watches that folder, restarts each pod to match. That is why the API server drops offline for a few seconds in the middle of an apply.

$ sudo kubeadm upgrade apply v1.31.2

[upgrade/staticpods] Writing new Static Pod manifests to "/etc/kubernetes/tmp/kubeadm-upgraded-manifests"
[upgrade/staticpods] Component "kube-apiserver" upgraded successfully!
[upgrade/staticpods] Component "kube-controller-manager" upgraded successfully!
[upgrade/staticpods] Component "kube-scheduler" upgraded successfully!
[upgrade/etcd] Upgrading etcd: 3.5.12-0 -> 3.5.15-0
[upgrade] SUCCESS! Your cluster was upgraded to "v1.31.2". Enjoy!
 
[upgrade/kubelet] Now proceed with upgrading the kubelet on this node if you have not already.

Then the nodes, one at a time

The control-plane machine still needs its own kubelet and kubectl packages upgraded and the kubelet restarted, and it gets drained and uncordoned around that work like any other node. Then you work through the workers with the same rhythm you repeat for each: drain, upgrade, uncordon. Draining a node means evicting the pods off it and marking it unschedulable, so nothing new lands there while you work. The eviction respects any PodDisruptionBudget you have set, a rule that says keep at least N copies of this app alive at all times, so a drain can never knock a service below its floor.

$ kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data

node/node-1 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-7bx9q, kube-system/cilium-4mn2p
evicting pod payments/api-6d9f7c8b5-2xk4t
evicting pod default/web-5c7d9b6f4-lq8mn
pod/web-5c7d9b6f4-lq8mn evicted
pod/api-6d9f7c8b5-2xk4t evicted
node/node-1 drained

The --ignore-daemonsets flag is there for a reason. DaemonSet pods (one copy pinned to every node, like a log shipper or the network agent) belong to the node and cannot be moved, so drain refuses to start without your say-so. If a drain just sits there and never finishes, the usual culprit is a PodDisruptionBudget it cannot satisfy: the app is already at its minimum, so evicting one more copy would break the rule. Run kubectl get pdb -A to see the budgets. Scaling the app up first, or loosening the budget, lets the drain finish. A stuck drain is protecting your users, not failing you.

Now log into that worker and run kubeadm upgrade node. On a worker this never touches the control plane. It reads the kubelet configuration the cluster already holds for you (kubeadm keeps it in a ConfigMap, a small bundle of settings, named kubelet-config in the kube-system namespace) and refreshes this node's local copy to match. After that you install the matching kubelet and kubectl packages and restart the kubelet. Every machine carries its own repo file and its own holds, so the repo switch and the unhold happen again here.

# same repo switch and unhold, this time on the worker
$ sudo sed -i 's|/v1.30/|/v1.31/|' /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-mark unhold kubeadm
$ sudo apt-get install -y kubeadm=1.31.2-1.1
$ sudo apt-mark hold kubeadm
$ sudo kubeadm upgrade node
$ sudo apt-mark unhold kubelet kubectl
$ sudo apt-get install -y kubelet=1.31.2-1.1 kubectl=1.31.2-1.1
$ sudo apt-mark hold kubelet kubectl
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet

[upgrade] Reading configuration from the cluster...
[upgrade] Skipping phase. Not a control plane node.
[kubelet-start] Writing kubelet configuration to file "/var/lib/kubelet/config.yaml"
[upgrade] The configuration for this node was successfully updated!
[upgrade] Now go ahead and upgrade the kubelet package using your package manager.

Give the kubelet a few seconds to re-register with the API server, confirm the node reports Ready on the new version, then uncordon it so the scheduler can use it again. Checking Ready at every single node is the entire discipline. Uncordon a half-broken node and the scheduler will happily send fresh pods to a machine that cannot run them.

$ kubectl uncordon node-1
$ kubectl get nodes

node/node-1 uncordoned
NAME     STATUS   ROLES           AGE    VERSION
cp-1     Ready    control-plane   287d   v1.31.2
node-1   Ready    <none>          287d   v1.31.2
node-2   Ready    <none>          287d   v1.30.4

On a managed platform (Amazon EKS, Google GKE, Azure AKS) the provider runs this whole dance for you. You pick a version, and it upgrades the control plane and rolls the node pools while honoring the same skew rules you just walked through. Knowing the manual sequence is still what lets you reason about what that button is actually doing when a rollout stalls at 3 a.m., and it is exactly what the CKA (Certified Kubernetes Administrator) exam puts in front of you.

Managed services change the buttons but not the order. Still upgrade the control plane before the node pools.

Backup etcd before control-plane upgrades you own. Snapshots are cheap insurance.

Deprecated APIs removed in the target version will break apply mid-flight. Run a deprecation scan first.

Try this

In a disposable kubeadm lab, run kubeadm upgrade plan, upgrade the control plane, then upgrade one worker. Refuse to skip the plan output.

# 1. on the first control-plane node
$ sudo sed -i 's|/v1.30/|/v1.31/|' /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.31.2-1.1 && sudo apt-mark hold kubeadm
$ sudo kubeadm upgrade plan
$ sudo kubeadm upgrade apply v1.31.2
 
# 2. same node: drain it, upgrade its kubelet, put it back
$ kubectl drain cp-1 --ignore-daemonsets --delete-emptydir-data
$ sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.31.2-1.1 kubectl=1.31.2-1.1 && sudo apt-mark hold kubelet kubectl
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet
$ kubectl uncordon cp-1
 
# 3. the worker: drain from your kubectl machine, then log into node-1
$ kubectl drain node-1 --ignore-daemonsets --delete-emptydir-data
$ sudo sed -i 's|/v1.30/|/v1.31/|' /etc/apt/sources.list.d/kubernetes.list
$ sudo apt-get update
$ sudo apt-mark unhold kubeadm && sudo apt-get install -y kubeadm=1.31.2-1.1 && sudo apt-mark hold kubeadm
$ sudo kubeadm upgrade node
$ sudo apt-mark unhold kubelet kubectl && sudo apt-get install -y kubelet=1.31.2-1.1 kubectl=1.31.2-1.1 && sudo apt-mark hold kubelet kubectl
$ sudo systemctl daemon-reload && sudo systemctl restart kubelet
$ kubectl uncordon node-1
 
# 4. check your work: every node Ready and on v1.31.2
$ kubectl get nodes

Takeaway

Upgrade is ordered: control plane, then workers, staying inside skew. plan before apply. Drain workers as you go.

Related