>_SecOpsLog_
CoursesKubernetes administrationWorker-node failure

Worker-node failure

NotReady nodes and the kubelet.

Advanced10 min · lesson 64 of 65
In plain terms
A NotReady node is a worker who stopped clocking in. Go to their station: is the time clock broken (the kubelet), are they out of desk space (disk full), did their badge expire? It’s nearly always one of those three.

You run kubectl get nodes and one line reads NotReady. Every Pod on that node is now in question. A Pod is the smallest thing Kubernetes runs, one or more containers that share a fate and live and die together. If that machine was carrying real traffic, something is already paging someone. A worker node dropping to NotReady is one of the incidents you'll actually handle, over and over, on a real cluster. The good news is small but real: the cause is almost always one of about five things, and there's a fixed drill that names which one every time.

What NotReady actually means

Every node runs a small agent called the kubelet, and the kubelet is what starts your Pods and keeps an eye on the machine. It works like a night watchman who calls the front desk every few seconds to say he's there and everything's fine. Kubernetes writes those check-ins down in two places. The kubelet updates the node's status, and it renews a tiny record called a Lease, in a namespace named kube-node-lease, by default every 10 seconds. Notice what the control plane is really watching. Not the node. The phone. A component called the node controller, which lives inside the kube-controller-manager, reads those leases. If a node stops renewing for roughly 40 seconds (a setting called the node-monitor-grace-period), the controller changes that node's Ready condition from True to Unknown, and kubectl starts printing NotReady. Read the exact word. Not broken. Unknown. The control plane honestly can't tell, because the watchman went quiet.

That silence is only one of the two ways a node goes NotReady, and telling them apart is the whole game. Sometimes the watchman does call in, but only to report bad news: still here, but the alarm system is dead and I can't do my rounds. That's the kubelet, alive and talking to the API server, setting its own Ready condition to False because it knows it can't run Pods properly. Both cases print NotReady in kubectl get nodes. The difference shows up when you describe the node. Unknown means the kubelet went dark and the control plane is guessing. False means the kubelet is up and honestly telling you it's stuck. Either way the control plane reacts the same. It stamps the node with a taint, which is just a do-not-run-here mark, and gives it the NoExecute effect, which adds and evict what's already running. A silent node gets tainted node.kubernetes.io/unreachable; a node reporting False gets node.kubernetes.io/not-ready. By default every Pod carries an automatic five-minute toleration for those taints (a toleration is a permission slip that says I'm allowed to stay a while even with that mark), so your Pods sit tight for 300 seconds before they're evicted and rescheduled onto healthy nodes. That delay is on purpose. A node that blips for twenty seconds shouldn't set off a stampede. It also means a genuinely dead node costs you five minutes of reduced capacity before the work moves, which is exactly why you spread replicas across nodes and availability zones.

which node is unhappy?
kubectl get nodes
output
NAME     STATUS     ROLES           AGE    VERSION
cp-1     Ready      control-plane   58d    v1.31.4
node-1   Ready      <none>          58d    v1.31.4
node-2   Ready      <none>          58d    v1.31.4
node-3   NotReady   <none>          58d    v1.31.4

kubectl get nodes tells you which node, not why. For the why, describe the node and read two things: the Conditions block, and the message the control plane recorded when it decided the node wasn't healthy.

read the conditions and the recorded message
kubectl describe node node-3
output (trimmed to the parts that matter)
Taints:  node.kubernetes.io/not-ready:NoExecute
         node.kubernetes.io/not-ready:NoSchedule
Conditions:
  Type             Status  Reason                       Message
  ----             ------  ------                       -------
  MemoryPressure   False   KubeletHasSufficientMemory   kubelet has sufficient memory available
  DiskPressure     False   KubeletHasNoDiskPressure     kubelet has no disk pressure
  Ready            False   KubeletNotReady              container runtime is down
Events:
  Type     Reason        Age   From             Message
  ----     ------        ----  ----             -------
  Normal   NodeNotReady  3m    node-controller  Node node-3 status is now: NodeNotReady

There's the split in black and white. Ready is False, not Unknown, so the kubelet is alive and reporting for itself. Its reason is KubeletNotReady and its message is container runtime is down. The kubelet is telling you, plainly, that it can't run Pods because the thing underneath it that actually runs containers has stopped. Memory and disk both read False, so this isn't a resource problem. The fix isn't in the cluster. It's on the node.

Get on the node and read the kubelet

This is the step people skip, and it's the one that closes the case. Get onto the box (SSH in, which is just the standard way to log into a remote machine, or use your cloud console, whatever you've got) and ask three things in order. Is the kubelet process actually alive? What is it logging? And is the engine underneath it healthy? That engine is the container runtime, usually a program called containerd, the lower-level piece that pulls images and runs the containers. The kubelet talks to it over a defined channel called the CRI, the Container Runtime Interface. If the runtime is dead, the kubelet can be running perfectly and still get nothing done. It's a chef with a working phone and no stove.

on node-3: process, logs, runtime
sudo systemctl status kubelet --no-pager
sudo journalctl -u kubelet --no-pager -n 15
sudo systemctl is-active containerd
output
● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled)
     Active: active (running) since Wed 2026-07-15 09:12:03 UTC; 19h ago
 
Jul 16 04:20:11 node-3 kubelet[1183]: E0716 04:20:11.882 kubelet.go:2419] "Container runtime not ready" runtimeReady="RuntimeReady=false reason:ContainerRuntimeNotReady message:container runtime is not running"
Jul 16 04:20:12 node-3 kubelet[1183]: E0716 04:20:12.401 remote_runtime.go:294] "Status from runtime service failed" err="rpc error: code = Unavailable desc = connection error: dial unix /run/containerd/containerd.sock: connect: connection refused"
failed

Read it from the top. The kubelet is active (running), so the process itself is fine. But it's logging connection refused against the containerd socket, again and again, and is-active containerd comes back failed. The runtime crashed out from under a healthy kubelet. That's why the node reported Ready: False instead of going Unknown. The kubelet never stopped talking to the control plane, it just kept saying it couldn't work. Cause found. Dead runtime, live node.

Recovery: restart-and-verify, or drain-and-replace

Once the logs name the cause, recovery is short. For a crashed runtime or a crashed kubelet, restart the runtime first, then the kubelet on top of it, then watch the node come back. The first two commands run on the node. The last one runs from your workstation. Don't skip the verify. A node that flips back to Ready is the only proof it worked, and it usually returns within a heartbeat cycle or two.

on node-3, then verify from your workstation
sudo systemctl restart containerd
sudo systemctl restart kubelet
kubectl get nodes node-3
output
NAME     STATUS   ROLES    AGE   VERSION
node-3   Ready    <none>   58d   v1.31.4

Some causes don't restart away. If the disk is full, you free space (more on that in a second). If the node is out of memory, it's overcommitted, and the real fix is right-sizing the Pods' resource requests so the scheduler stops packing too much onto it. And if the host is genuinely gone (hardware fault, a hypervisor that swallowed the VM, a kubelet certificate that expired and won't renew while you're under pressure), stop nursing it. Cordon the node so nothing new lands on it, drain it so the Pods move elsewhere, then delete and replace it. In a healthy cluster with replicas spread around, losing one worker is a non-event. That's the entire reason you run more than one.

give up on a dead node cleanly
kubectl cordon node-3
kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data
output
node/node-3 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-8x2fq, kube-system/cilium-l7d4m
evicting pod default/web-7c9f8b6d4-2xkpz
evicting pod default/api-5b8c9d7f6-nq4wl
pod/web-7c9f8b6d4-2xkpz evicted
pod/api-5b8c9d7f6-nq4wl evicted
node/node-3 drained
A full disk is the sneakiest NotReady cause
containerd hangs on to every image it has ever pulled, plus your container logs, under /var/lib/containerd and /var/log/pods. On a busy node those quietly grow. The kubelet starts deleting unused images once disk usage crosses its image garbage-collection line (85% full by default), but if logs and data keep filling the disk past the kubelet's eviction threshold (the default trips when less than 10% of the disk is free), the kubelet raises DiskPressure, stops accepting new Pods, and begins evicting running ones. Let the disk fill all the way and the kubelet can wedge completely and drag the node to NotReady. On the node, df -h shows it in one line. Then crictl rmi --prune deletes unused images, you truncate the old logs, and the node recovers fast. The durable fix is watching node disk before it bites, because this one is fully preventable, and it comes back on the same node if all you ever do is clean up once.
NotReady: name the cause, then act
Node NotReady in kubectl get nodes
describe node reads the verdict: Ready Unknown (kubelet went silent) or False (kubelet up, reporting a problem)
process dead
kubelet crashed or misconfigured
systemctl status shows it failed; fix the config, restart kubelet
runtime down
containerd not answering on the CRI socket
restart containerd, then kubelet, verify Ready
disk full
DiskPressure, evictions begin
df -h; crictl rmi --prune, clear old logs
memory
MemoryPressure, node overcommitted
right-size Pod requests so it stops overpacking
cert / network
expired kubelet cert, or no route to the API
renew the cert; check the path to the API server on :6443
host gone
hardware or VM truly dead
cordon, drain, delete, replace the node
NotReady just means Ready isn't True. describe tells you which kind: Unknown (the kubelet went quiet) or False (the kubelet's up but can't do its job). The node's own logs name the exact cause, and recovery is either restart-and-verify or drain-and-replace.

Disk pressure and memory pressure appear in node conditions. Read them before you blame the CNI.

Clock skew breaks TLS and tokens. Chrony matters on nodes.

After recovery, drain is still the polite way to reshuffle if the node was partially dead with stuck pods.

Try this

Cordon nothing yet — describe a NotReady node if you have one, or simulate by stopping kubelet in lab. Check kubelet logs and runtime health before rebooting out of habit.

terminal
$ kubectl get nodes
 
$ kubectl describe node node-3
 
$ sudo systemctl status kubelet --no-pager
$ sudo journalctl -u kubelet --no-pager -n 15
$ sudo systemctl is-active containerd
 
$ sudo systemctl restart containerd
$ sudo systemctl restart kubelet
$ kubectl get nodes node-3
 
$ kubectl cordon node-3
$ kubectl drain node-3 --ignore-daemonsets --delete-emptydir-data
 
node/node-3 cordoned
WARNING: ignoring DaemonSet-managed Pods: kube-system/kube-proxy-8x2fq, kube-system/cilium-l7d4m
evicting pod default/web-7c9f8b6d4-2xkpz
evicting pod default/api-5b8c9d7f6-nq4wl
pod/web-7c9f8b6d4-2xkpz evicted
pod/api-5b8c9d7f6-nq4wl evicted
node/node-3 drained

Takeaway

NotReady is a node agent or runtime problem until proven otherwise. Get on the host; the API cannot heal a dead kubelet.

Quick check
01A node is NotReady. describe node shows Ready: False with reason KubeletNotReady and the message "container runtime is down." On the box, the kubelet is active (running) but logs "connection refused" to /run/containerd/containerd.sock every second. What's the fastest correct fix?
Correct — The kubelet is fine; the runtime under it died, which is why it reported Ready: False instead of going silent. Restart the runtime first, then the kubelet, then verify with kubectl get nodes.
Incorrect — No. Deleting the node object doesn't touch the dead runtime, and a healthy kubelet would just re-register the node seconds later. You've fixed nothing.
Incorrect — Overkill. That's the move for a host that's truly gone. A crashed runtime is a ten-second restart, not a node replacement.
Incorrect — That hides the symptom. The Pods still can't run while the runtime is down, so you've only made the cluster slower to notice. It also wouldn't apply here, since the node reported False rather than going Unknown, and that timer only governs the Unknown path.
02Two NotReady nodes look identical in kubectl get nodes, but describe node shows one with Ready: Unknown and the other with Ready: False. What does the Unknown one specifically tell you?
Correct — Unknown means the check-ins stopped and the node controller inferred trouble after the grace period; it can't actually see what's wrong.
Incorrect — that's the Ready: False case; Unknown means the kubelet went quiet, not that it's reporting a problem.
Incorrect — a crashed runtime under a live kubelet produces Ready: False with a message, not Unknown.
Incorrect — DiskPressure is a condition a live kubelet reports, which shows as False, not the silent Unknown state.
03A node goes NotReady. describe node shows DiskPressure: True; on the box, df -h shows the disk nearly full from containerd images under /var/lib/containerd and pod logs. What actually fixes it?
Incorrect — a restart frees no space, so the disk stays full and DiskPressure returns right away.
Incorrect — that just silences the alarm; the disk is still full and pods still can't get the space they need.
Incorrect — that's the move for a truly dead host; a full disk is reclaimable without discarding the node.
Correct — freeing disk clears DiskPressure and lets the node recover, and monitoring stops the same node filling up again.
Control-plane failureNetwork troubleshooting

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep