Kubelet & kube-proxy

The two agents that make a worker node useful.

kubectl get nodes says one node is NotReady. Every pod on it is minutes away from being kicked off and rescheduled somewhere else. Nobody deployed anything. Nothing you can see actually crashed. So what broke? Almost always it's one of the two small agents that run on every worker node, and the fix depends entirely on which one.

A worker node is just a Linux machine with a container runtime installed (the software that actually runs containers). Left alone, that machine can't join a cluster, run a Pod, or answer a single request. A Pod, by the way, is the smallest unit Kubernetes schedules: one or more containers that share a single IP address, like roommates sharing one street address. Two small programs turn that bare machine into a real cluster member. One is the kubelet, the agent Kubernetes runs on every node; it starts your pods and keeps them alive. The other is kube-proxy, which wires up the networking that lets other pods reach them. Most node troubleshooting is really just knowing what each of these does, and what it flatly refuses to do.

The kubelet: the only thing that actually runs a pod

Think of the kubelet as the one worker standing on the factory floor. Somewhere upstairs, a planner decides which job goes to which floor and pins the assignment to a board. That board is the API server, the cluster's front door and its single source of truth about what's supposed to be running where. The kubelet doesn't argue with the board. It reads the jobs assigned to its own floor and goes and starts the machines.

In real terms: the kubelet watches the API server for pods that have been assigned (bound) to its node. For each one, it talks to the container runtime, containerd or CRI-O, through a standard plug called the CRI, short for Container Runtime Interface. That plug is how the kubelet says pull this image, start these containers, without caring which runtime sits underneath. Then it settles in and watches. It runs the liveness and readiness probes you wrote (small health checks: is the container alive, and is it ready for traffic), restarts containers that fall over, and reports the health of every pod, plus the node itself, back to the API server.

That last part, the reporting, is a heartbeat. A thermostat that stops sending readings looks broken even if the room is fine, and the cluster treats a silent node the same way. About every 10 seconds the kubelet renews a tiny Lease object (basically a timestamp that says 'still here') in a namespace called kube-node-lease. If the control plane stops seeing those renewals for roughly 40 seconds, a value called the node-monitor-grace-period, the node controller marks the node NotReady and, a few minutes later, starts evicting its pods. Read that carefully. NotReady doesn't mean the pods died. It means the kubelet went quiet. The containers might still be happily serving traffic while the cluster decides they're gone.

The kubelet has one more trick worth knowing. Point it at a plain folder on disk and it runs any YAML it finds there directly, as what's called a static pod, with no scheduler and no API server involved in the decision. That folder is not a default. You name it in the kubelet's own config, as staticPodPath, and out of the box it is unset; clusters built with kubeadm set it to /etc/kubernetes/manifests, which is why you see that path quoted everywhere. This is the bootstrapping trick that lets a self-hosted cluster start itself: on a kubeadm control-plane node the API server, etcd (the cluster's key-value database, where all cluster state lives), and the scheduler all come up as static pods, because the kubelet can run them before there's any cluster around to ask for permission. On a managed cluster you never see it, because the provider runs those pieces for you out of sight.

kubectl get nodes -o wide

NAME           STATUS     ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE             CONTAINER-RUNTIME
k8s-cp-1       Ready      control-plane   58d   v1.31.4   10.0.1.10     Ubuntu 24.04.1 LTS   containerd://1.7.22
k8s-worker-1   Ready      <none>          58d   v1.31.4   10.0.1.21     Ubuntu 24.04.1 LTS   containerd://1.7.22
k8s-worker-2   NotReady   <none>          58d   v1.31.4   10.0.1.22     Ubuntu 24.04.1 LTS   containerd://1.7.22

When a node flips to NotReady, resist the urge to keep poking at kubectl. The kubelet's own log on the machine will usually tell you what's wrong in plain words. Secure Shell (SSH) into the node and read it directly.

sudo journalctl -u kubelet --no-pager -n 4

Jul 16 09:14:02 k8s-worker-2 kubelet[1183]: E0716 09:14:02.881  1183 kubelet.go:2988] "Container runtime network not ready" networkReady="NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: cni plugin not initialized"
Jul 16 09:14:03 k8s-worker-2 kubelet[1183]: I0716 09:14:03.104  1183 setters.go:602] "Node became not ready" node="k8s-worker-2" condition={"Type":"Ready","Status":"False","Reason":"KubeletNotReady","Message":"container runtime network not ready: cni plugin not initialized"}

And there it is, right in the log. The container runtime came up fine, but the CNI plugin (Container Network Interface, the piece that hands each pod its IP address) never finished starting. Without it, no pod can get an address, so the kubelet does the honest thing and refuses to report Ready. Fix the network plugin and the node rejoins on its own. Other failures read just as plainly: a full disk, a crashed kubelet, a dead runtime. Each one leaves a clear reason in this same log, which is exactly why you look here before anywhere else.

kube-proxy, and why nothing flows through it

This next part trips up almost everyone. Despite the name, kube-proxy is not a switchboard operator patching your call through to a free line. It's more like the clerk who keeps the building's directory up to date so callers dial the right extension themselves. kube-proxy writes the rules. It never touches the actual packets.

Pods are disposable. Kubernetes creates, kills, and replaces them constantly, and every replacement comes up with a fresh IP address. So handing a client a pod's raw IP is pointless; it can go stale at any moment. A Service fixes that by publishing one stable virtual address, the ClusterIP, in front of a shifting set of pods. kube-proxy's job is to keep that address pointed at whichever pods are actually ready. It watches Services and their EndpointSlices (the running list of which pod IPs are ready to receive traffic right now) and writes that mapping straight into the node's Linux kernel. It can do this a few ways: classic iptables rules, IPVS (a load balancer built into the kernel), or the newer nftables backend, which reached beta in v1.31 and became generally available in v1.33. Once the rules are written, kube-proxy steps back.

That split is the whole point. When a client connects to the ClusterIP, the kernel's own plumbing takes over: netfilter (which filters and rewrites packets) and connection tracking (which remembers where each open connection is headed) swap the destination for a real pod IP and forward it on. Nothing is listening on the ClusterIP. There's no process there to connect to at all. You could kill kube-proxy this second and existing traffic would keep flowing on the rules it already wrote, not one dropped packet. You can watch both halves it manages: the endpoints it reads, and the rule it writes.

kubectl get endpointslices -l kubernetes.io/service-name=web

NAME        ADDRESSTYPE   PORTS   ENDPOINTS                   AGE
web-8kx2p   IPv4          80      10.244.1.37,10.244.2.19     12d

sudo iptables -t nat -L KUBE-SERVICES -n | grep default/web

KUBE-SVC-XG3D4YQ2K7L5N6PA  tcp  --  0.0.0.0/0  10.104.12.7  /* default/web cluster IP */ tcp dpt:80

So the ClusterIP 10.104.12.7 is nothing more than a target in a kernel NAT rule, and that rule fans out to the two ready pod IPs pulled from the EndpointSlice. NAT, network address translation, just means the kernel is rewriting one address into another on the way through. That single rule is your whole troubleshooting map. Pods that won't start, or won't stay healthy, are the kubelet's department. A ClusterIP that won't answer is a networking question, but you check the endpoints first and only then go looking at kube-proxy.

crictl talks to the CRI runtime. docker ps is the wrong reflex on modern nodes after the dockershim removal.

Kubelet flags and config files on disk outrank what you wish were true in a spreadsheet. Drift after OS patching is common.

When kube-proxy dies, the rules it already wrote stay in the kernel, so nothing breaks in that first minute. What stops is the updating. Every pod replaced from then on comes up with an address no rule knows about, and the old rules keep aiming at pod IPs that are already gone, so Service traffic quietly blackholes while every pod still reads Running. That split between pod health and Service health is a classic on-call trap.

Try this

Pick a Ready worker and SSH into it. Confirm the kubelet systemd unit is running, then list the containers it started with crictl. Back on your workstation, look at the kube-proxy pods, so you have now seen both agents that keep that box in the cluster. The label below is the kubeadm one; if nothing comes back, run kubectl -n kube-system get ds to find what your installer named it.

$ kubectl get nodes -o wide
 
# then SSH to the Ready worker you picked
$ systemctl status kubelet --no-pager
$ sudo crictl ps
 
# back on your workstation
$ kubectl -n kube-system get pods -l k8s-app=kube-proxy -o wide

Takeaway

Kubelet makes pods real; kube-proxy makes Services reachable on the node. NotReady usually means go to the machine, not just stare at kubectl.

Related