>_SecOpsLog_
CoursesKubernetes administrationOS upgrades & maintenance

OS upgrades & maintenance

Patching a node host without dropping workloads.

Intermediate10 min · lesson 57 of 65
In plain terms
Patching a node is servicing one checkout lane at a time, so the shop never fully closes: empty the lane, fix it, reopen it, then move on to the next one.

A kernel vulnerability drops on a Tuesday, and every machine in your cluster is now running a patchable-but-still-exposed kernel. You can't just reboot the boxes. Each one is a node, a real server or virtual machine carrying live workloads, and rebooting a node kills everything running on it. So how do you patch the operating system underneath a running Kubernetes cluster without a single user seeing an outage? One node at a time, and never while that node is busy.

It works the way you'd service a bank of elevators. You don't shut the whole bank to fix one car. You flip one car to "out of service" so no new riders step in, let its current passengers reach their floors, take that car offline, work on it, then put it back. The other cars carry the traffic the whole time. Kubernetes patches a node exactly this way. A node runs a kubelet (the small agent on each machine that starts and watches its containers) and some number of Pods (a Pod is the smallest thing Kubernetes runs, usually one container plus its sidecars). Before you touch the host OS, you move those Pods off the node and keep new ones from landing.

Cordon and drain: emptying the node

Two commands do the moving. cordon is the "out of service" sign. It marks the node unschedulable so the scheduler stops placing new Pods there, while leaving the Pods already running untouched. Under the hood it flips one field, spec.unschedulable, to true. drain is the part that clears the car. It evicts the Pods currently on the node so they reschedule onto other nodes. It doesn't yank them; it goes through the Eviction API, the polite path that honors graceful shutdown and respects PodDisruptionBudgets (more on those in a moment). Two flags you'll almost always pass: --ignore-daemonsets, because DaemonSet Pods (one-per-node agents like log shippers and the network plugin, the CNI) are supposed to live on every node and would just block the drain, and --delete-emptydir-data, because a Pod using a scratch emptyDir volume refuses to evict otherwise.

terminal — check the node before you touch it
kubectl get node node-2 -o wide
output
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP   KERNEL-VERSION
node-2   Ready    <none>   88d   v1.31.4   10.0.1.22     5.15.0-105-generic

Note the kernel version, 5.15.0-105. That's the thing we're patching, and comparing it afterward is how you prove the patch actually took. Now drain it.

terminal — empty node-2
kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data
output
node/node-2 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-7bx2k, kube-system/cilium-9fq4d
evicting pod default/checkout-7d9c6b8f5-2xk4p
evicting pod default/web-5c8fbd9d7-lm7qz
pod/checkout-7d9c6b8f5-2xk4p evicted
pod/web-5c8fbd9d7-lm7qz evicted
node/node-2 drained

See that drain cordoned the node itself as its first line, so a separate cordon command is optional. When it prints node/node-2 drained, your workloads are off the machine and the scheduler won't send more. It's safe to reboot.

Patch, reboot, and bring it back Ready

The actual patching happens on the host, not through kubectl. You open a remote shell on the machine (SSH) and let the OS package manager pull kernel, runtime, and library updates. A kernel update almost always needs a reboot to take effect, because you can't swap the running kernel out from under a live system. (Live kernel patching exists, but assume you're rebooting.)

on node-2 (SSH) — apply OS updates, then reboot
sudo apt-get update && sudo apt-get upgrade -y
sudo reboot
output
The following NEW packages will be installed:
  linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic
The following packages will be upgraded:
  containerd.io linux-headers-generic linux-image-generic openssl
4 upgraded, 2 newly installed, 0 to remove
...
Setting up linux-image-5.15.0-119-generic ...
*** System restart required ***
Connection to node-2 closed by remote host.

While node-2 reboots, its kubelet stops reporting to the control plane, so the node's status flips to NotReady, then back to Ready once the machine is up and the kubelet re-registers and the container network comes back. Watch for that. Do not uncordon a node that hasn't actually returned healthy.

terminal — after the reboot
kubectl get node node-2 -o wide
output
NAME     STATUS   ROLES    AGE   VERSION   INTERNAL-IP   KERNEL-VERSION
node-2   Ready    <none>   88d   v1.31.4   10.0.1.22     5.15.0-119-generic

Kernel moved from 5.15.0-105 to 5.15.0-119 and STATUS is Ready. That pair of facts is your proof: the patch landed and the node is healthy. Now let it accept work again.

terminal — return it to service
kubectl uncordon node-2
output
node/node-2 uncordoned

The scheduler can place Pods here again. Move to node-3 and repeat the same steps. One node at a time is not a suggestion; that pacing is what keeps enough capacity online to hold the load while each machine is out of service.

When drain refuses, and how to stop doing this by hand

The most common way this goes sideways: drain hangs, printing "Cannot evict pod ... would violate the disruption budget" on a loop. That's a PodDisruptionBudget (PDB) doing its job. A PDB is a promise you wrote down, something like "keep at least two replicas of checkout running at all times." If evicting one more Pod would break that promise, the Eviction API refuses and drain waits for a replacement to come up elsewhere first. When a drain stalls, your first move is to look at the budget.

terminal — why is the drain stuck?
kubectl get pdb -A
output
NAMESPACE   NAME       MIN AVAILABLE   MAX UNAVAILABLE   ALLOWED DISRUPTIONS   AGE
default     checkout   2               N/A               0                     30d

ALLOWED DISRUPTIONS is 0. The checkout service is already sitting at its minimum, so nothing more can be evicted until a fresh replica is Ready on another node. The real fix is capacity, not force. Give the service room to move: scale the deployment up so the budget has slack, or figure out why a replacement isn't scheduling (no space on other nodes, or a readiness probe that never passes). Once a replacement can come up, the eviction proceeds on its own.

--disable-eviction switches the safety off
It's tempting to unstick a stalled drain with a flag. The one that actually pushes past the budget is kubectl drain --disable-eviction. It skips the Eviction API and deletes Pods with a plain delete, which ignores PodDisruptionBudgets completely. (--force is a different flag, and a common mix-up. It only lets drain remove bare Pods that no controller manages, and on its own it won't get you past a PDB at all.) Bypassing the budget is the exact outage the whole drain dance exists to prevent. You can take a service below its minimum and drop live traffic in the process. Read a refusing drain as a signal that you don't have the spare replicas or node capacity to lose this machine right now. Add capacity or replicas, or wait for the current roll to catch up. Don't force the eviction.
After the reboot, what do you do next?
After patch + reboot, check the node
kubectl get node node-2
Ready
uncordon and move on
kubectl uncordon node-2, then drain the next node
NotReady
debug before you uncordon
journalctl -u kubelet on the host; confirm the container runtime and CNI came back up
drain refused
the PDB has no slack
kubectl get pdb; add replicas or free node capacity, don't bypass the budget
Uncordon only after the node is genuinely Ready. A stuck drain is the safety net working, not a bug to override.

Patching 40 nodes by hand is how nodes quietly fall behind and stay vulnerable. So you automate the same dance rather than inventing a new one. On managed platforms, the hosted Kubernetes services from the big clouds (Amazon's EKS, Google's GKE, Azure's AKS), you turn on node auto-upgrade: you set a maintenance window and the provider cordons, drains, and replaces nodes on a schedule. On self-managed clusters an operator like the system-upgrade-controller runs it. You describe the rollout as a Plan object and the controller walks the fleet node by node with the drain safety built in.

upgrade-plan.yaml — system-upgrade-controller
apiVersion: upgrade.cattle.io/v1
kind: Plan
metadata:
  name: os-patch
  namespace: system-upgrade
spec:
  concurrency: 1                 # one node at a time
  nodeSelector:
    matchExpressions:
      - {key: node-role.kubernetes.io/control-plane, operator: DoesNotExist}
  serviceAccountName: system-upgrade
  cordon: true
  drain:
    force: false                 # don't delete unmanaged bare pods
    ignoreDaemonSets: true
    disableEviction: false       # use the eviction API, so PDBs are honored
  upgrade:
    image: your-registry/os-patcher:1.4
terminal — confirm the plan is registered
kubectl -n system-upgrade get plan os-patch
output
NAME       IMAGE                          CHANNEL   VERSION
os-patch   your-registry/os-patcher:1.4

concurrency: 1 is the whole point. It holds the roll to a single node at a time, the same pacing you'd keep by hand. The drain leaves disableEviction off, so evictions go through the normal path and your PodDisruptionBudgets are honored instead of steamrolled. OS patching is just repeated safe node drains: cordon, drain, patch, reboot, verify Ready, uncordon, next. Cluster version upgrades, coming up next, follow the same rhythm.

Reboots without drain are how you discover missing PodDisruptionBudgets in production.

Kernel and CNI modules can break after OS updates. Keep a rollback image for the node.

Document the maintenance window and the node name in the change ticket. Guesswork causes double drains.

Try this

Pick a worker, cordon and drain, apply a harmless package update or reboot in lab, wait for Ready, then uncordon. Time the steps.

terminal
$ kubectl get node node-2 -o wide
 
$ kubectl drain node-2 --ignore-daemonsets --delete-emptydir-data
 
node/node-2 cordoned
Warning: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-7bx2k, kube-system/cilium-9fq4d
evicting pod default/checkout-7d9c6b8f5-2xk4p
evicting pod default/web-5c8fbd9d7-lm7qz
pod/checkout-7d9c6b8f5-2xk4p evicted
pod/web-5c8fbd9d7-lm7qz evicted
node/node-2 drained
 
The following NEW packages will be installed:
  linux-image-5.15.0-119-generic linux-modules-5.15.0-119-generic
The following packages will be upgraded:
  containerd.io linux-headers-generic linux-image-generic openssl
4 upgraded, 2 newly installed, 0 to remove
...
Setting up linux-image-5.15.0-119-generic ...
*** System restart required ***
Connection to node-2 closed by remote host.
 
$ kubectl get node node-2 -o wide
 
$ kubectl uncordon node-2

Takeaway

Patch the host only after workloads are off it. kubelet and runtime versions must stay in skew with the control plane.

Quick check
01You run kubectl drain node-2 and it stalls, repeating "Cannot evict pod ... would violate the disruption budget." What's the right move?
Incorrect — No. --force doesn't do what people expect here. It only lets drain delete bare Pods that no controller manages; it doesn't touch the Eviction API or the PDB, so the drain still stalls on the budget. The flag that actually pushes past a PDB is --disable-eviction, and using it just deletes Pods below your minimum and risks the live traffic you're protecting.
Correct — ALLOWED DISRUPTIONS is 0 because the service is at its minimum. Give it slack (more replicas, or somewhere for a replacement to schedule) and the eviction proceeds safely on its own.
Incorrect — No. An unpatched kernel stays exposed on every Pod that shares it. The node still needs the patch; you just have to make room to drain it safely first.
Incorrect — No. That throws away the availability guarantee entirely and lets the drain evict below a safe level. The budget is telling you the truth: you don't yet have the capacity to lose this node.
02After patching and rebooting node-2, which pair of facts together proves the OS patch landed and the node is safe to return to service?
Incorrect — the drain ran before the patch, and the DaemonSet warning is expected; neither says anything about the reboot's result.
Incorrect — uncordon is the last step you take, and only after you have already confirmed the node is healthy.
Correct — the changed kernel proves the patch took, and Ready proves the kubelet, runtime, and network all came back.
Incorrect — a reboot does not clear the cordon; you uncordon manually, and that field says nothing about the kernel.
03You drained, patched, and rebooted node-2. Several minutes later kubectl get node node-2 still shows STATUS NotReady. What is the right next step?
Incorrect — uncordoning a NotReady node lets the scheduler send Pods to a machine that cannot run them yet.
Correct — NotReady means the kubelet, runtime, or network has not recovered; investigate and fix it before returning the node to service.
Incorrect — the node is already drained; re-draining does nothing about a kubelet or CNI that failed to start after the reboot.
Incorrect — that is destructive and premature; the node most likely just needs its kubelet or network sorted out after the reboot.
Node managementReleases & version skew

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep