The kube-scheduler has been crash-looping for a quarter of an hour. You run kubectl create deployment web --image=nginx --replicas=3. What do you find when you go looking for the Pods?

All three Pods exist, but they sit in Pending, because nothing has picked a node for them. The API server accepts the object and the ReplicaSet controller creates the Pods, but binding a Pod to a node is the scheduler's job and nobody else's. With it down, the Pods wait in Pending with no Scheduled event until it recovers.

You are logged into a worker node built with kubeadm and you want to list the containers actually running on that host, without going through kubectl. Which tool do you reach for, and why?

`crictl ps`, because it talks to the container runtime (containerd or CRI-O) over the CRI (Container Runtime Interface, the standard socket a kubelet uses to drive a runtime). The lesson uses `sudo crictl ps` for exactly this, now that the Docker shim is gone.

A control-plane incident has taken out both the API server and the scheduler, yet every application dashboard is green and no user has complained. Why is calling this 'fine' a mistake?

Nothing heals or reschedules any more, so the next node failure takes its Pods down with no replacement and the pain arrives all at once. Each kubelet keeps serving from the Pod specs it already holds, but change and self-healing have stopped, so a later node failure goes unrecovered.

Courses Kubernetes administrationCluster architecture

Cluster architecture

Control plane vs worker nodes, and the components on each.

Beginner12 min · lesson 1 of 65

In plain terms

Think of a restaurant. The control plane is the manager and the order book — it decides and remembers, but never cooks. The worker nodes are the kitchen line — they cook, but make no decisions. Neither does the other’s job, on purpose.

You type one command, kubectl apply -f app.yaml, and roughly a dozen separate programs wake up to make it real. Not one of them is kubectl. That gap between the command you ran and the machinery that carries it out is the whole design of a Kubernetes cluster. Learn to name the parts and you stop guessing when something breaks.

A cluster is two kinds of machine. Control-plane nodes decide what should run and remember it. Worker nodes run your containers, packaged into Pods (a Pod is the smallest thing Kubernetes will place on a machine: usually one container, sometimes two or three that need to share a single network address). You never hand a worker its orders directly. You write down what you want, and the control plane behaves like a thermostat. You set it to 21 degrees, it nudges the heating, checks the room, nudges again, and it never stops checking. Swap 'temperature' for 'three copies of my web server' and that correcting loop is the entire model. The rest is detail.

The control plane: four programs, one job each

Four programs make up the control plane, and each one has a single job. The kube-apiserver is the only door in. Picture a building with one staffed front desk: every visitor signs in there or does not get in. Every command you run, every controller, and every worker's agent talks to that API server (application programming interface, the set of endpoints everything else calls) and to nothing else. Nothing gets written unless the request clears three gates: authentication (who are you), authorization (are you allowed to do this), and admission (is this object even legal here). etcd is the cluster's notebook, a key-value database (data stored as names and their values, like a giant dictionary) that holds both what you asked for and what is true right now. Lose etcd and you have lost the cluster, which is why it is the one thing you back up religiously. The kube-scheduler is the host at a busy restaurant: it watches for Pods that have no table yet and seats each one on a node. The kube-controller-manager runs the correcting loops, one per object type, that keep dragging reality toward what you wrote down.

terminal

$ kubectl get nodes -o wide

output

NAME     STATUS   ROLES           AGE   VERSION   INTERNAL-IP   OS-IMAGE
cp-1     Ready    control-plane   42d   v1.32.1   10.0.1.10     Ubuntu 24.04 LTS
node-1   Ready    <none>          42d   v1.32.1   10.0.1.21     Ubuntu 24.04 LTS
node-2   Ready    <none>          42d   v1.32.1   10.0.1.22     Ubuntu 24.04 LTS

terminal

$ kubectl -n kube-system get pods --field-selector spec.nodeName=cp-1

output

NAME                           READY   STATUS    RESTARTS      AGE
etcd-cp-1                      1/1     Running   0             42d
kube-apiserver-cp-1            1/1     Running   0             42d
kube-controller-manager-cp-1   1/1     Running   2 (9d ago)    42d
kube-scheduler-cp-1            1/1     Running   2 (9d ago)    42d

On a managed service (Amazon EKS, Google GKE, Azure AKS, the hosted Kubernetes offerings from the three big clouds) the provider runs and patches those four for you and you never log into the machines. On a cluster you built yourself with kubeadm (the tool that bootstraps a cluster from scratch), they run as static Pods, which is why they turn up in the kube-system namespace above. Same four components either way. What changes is who gets paged when one of them falls over.

What really happens when you run kubectl apply

Follow one Deployment through the system. kubectl turns your YAML (the indented text format you write Kubernetes config in) into an object and posts it to the API server, which checks it and writes it to etcd. The Deployment controller spots a Deployment it has not acted on yet and creates a ReplicaSet (the object whose only job is keeping a fixed number of identical Pods alive). The ReplicaSet controller sees that it wants three Pods and has zero, so it creates three. Those Pods have no node yet, so the scheduler picks one for each and writes that choice back through the API server. Only now does the kubelet on the chosen node notice a Pod bound to it. It tells the container runtime to pull the image and start the container, then reports the result back up. Every arrow in that chain passes through the API server and lands in etcd. That is why those two are the parts you guard hardest.

From kubectl apply to a running Pod

Nothing pushes work down onto a node. Each program watches the API server and reacts to what it sees, so a Pod moves forward only as fast as the next program in the chain notices it.

terminal

$ kubectl create deployment web --image=nginx:1.27 --replicas=3

output

deployment.apps/web created

terminal

$ kubectl get events --sort-by=.lastTimestamp | grep web-6b7f8c9d4-2xk9p

output

0s   Normal   Scheduled   pod/web-6b7f8c9d4-2xk9p   Successfully assigned default/web-6b7f8c9d4-2xk9p to node-1
0s   Normal   Pulling     pod/web-6b7f8c9d4-2xk9p   Pulling image "nginx:1.27"
2s   Normal   Pulled      pod/web-6b7f8c9d4-2xk9p   Successfully pulled image "nginx:1.27" in 1.83s
2s   Normal   Created     pod/web-6b7f8c9d4-2xk9p   Created container: nginx
2s   Normal   Started     pod/web-6b7f8c9d4-2xk9p   Started container nginx

Read that list top to bottom and you are watching the handoff happen. Scheduled is the scheduler's line. Pulling and Started are the kubelet driving the runtime. When a Pod is stuck, events are the first place you look, because the last successful line tells you which stage of the chain it died at.

The worker node, and why NotReady means you get on the box

Every worker runs three things. The kubelet is the foreman on that machine: the agent that takes Pod specs from the API server, drives the runtime, runs the health checks, and reports whether the node and its Pods are alive. The container runtime (containerd or CRI-O, the programs that actually pull images and start containers) does the physical work. Docker's built-in shim was removed back in v1.24, so on a current cluster you inspect containers with crictl, not docker. kube-proxy programs the node's networking so traffic aimed at a Service (the stable address that fronts a group of Pods) lands on the right Pods. Here is the part that matters at 3am. The kubelet is an ordinary systemd service on the host (systemd is Linux's service manager), not a Pod the control plane can restart for you. So when a node reports NotReady, the API server cannot fix it, because the thing that reports readiness is the thing that is down. You get onto the box.

terminal

$ systemctl status kubelet --no-pager

output

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled)
     Active: active (running) since Tue 2026-06-02 09:12:04 UTC; 6 weeks ago
   Main PID: 1183 (kubelet)
     Memory: 71.4M

terminal

$ sudo crictl ps        # containers via the container runtime, not docker

output

CONTAINER      IMAGE          STATE     NAME    ATTEMPT   POD
a1b2c3d4e5f6   nginx@sha256   Running   nginx   0         web-6b7f8c9d4-2xk9p

Your apps can look fine while the control plane is dead, right up until they don't

If the API server or the scheduler dies, Pods that are already running keep running. Each kubelet holds its Pod specs locally and restarts containers on its own, so users notice nothing. What stops is change and self-healing. Lose a whole node while the control plane is unreachable and nothing recreates its Pods elsewhere, new deployments hang forever, and kubectl returns connection errors. A green application dashboard during a control-plane outage is not proof you are fine. It means the damage is stored up until the next node failure, and then it all arrives at once. Alert on control-plane health directly: can you reach the API server, and does etcd still have quorum (a majority of its members alive and talking to each other). Do not rely on app uptime alone.

Managed clusters hide the control-plane machines from you, but the same four jobs are still being done somewhere: API server, etcd, scheduler, controller manager. With kubeadm you can see them as static Pods in kube-system. On EKS or GKE you depend on them exactly as much, and your escalation path is a support ticket rather than a login to the machine.

Seven names cover almost every cluster you will meet. Four on the control plane (API server, etcd, scheduler, controller manager) and three on every worker (kubelet, container runtime, kube-proxy). When someone tells you 'the cluster is broken', your first question is which of those seven they mean, because the fix for a dead scheduler and the fix for a dead kubelet have nothing in common.

Walk one Deployment from apply to Running yourself, with events sorted by time. Scheduled is the scheduler. Pulling and Started are the kubelet. Do that once on a lab cluster and the sequence stops being a picture you memorised and becomes a checklist you can run under pressure.

Pressure pushes people straight to restarting things. Resist that here. kubectl describe pod and the event stream tell you which of those seven programs last did its job, and that answer is faster than a speculative restart and a lot cheaper. A Pod sitting in Pending with no Scheduled event points at the scheduler. A Pod that got scheduled but never started pulling points at the kubelet or the runtime on that one node. Same chain, read backwards from wherever it stopped.

Try this

On a lab cluster, list the nodes, then look at what is actually running on the control-plane node. The goal is naming the programs that wake up after kubectl apply, not memorising a diagram.

terminal

$ kubectl get nodes -o wide
 
$ kubectl -n kube-system get pods --field-selector spec.nodeName=cp-1
 
NAME                           READY   STATUS    RESTARTS      AGE
etcd-cp-1                      1/1     Running   0             42d
kube-apiserver-cp-1            1/1     Running   0             42d
kube-controller-manager-cp-1   1/1     Running   2 (9d ago)    42d
kube-scheduler-cp-1            1/1     Running   2 (9d ago)    42d
 
$ kubectl create deployment web --image=nginx:1.27 --replicas=3
 
$ kubectl get events --sort-by=.lastTimestamp | grep web-6b7f8c9d4-2xk9p
 
$ systemctl status kubelet --no-pager

Takeaway

The control plane decides and remembers. Workers run. Every write travels through the API server and lands in etcd, so when something stalls, the useful question is which link in that chain last succeeded.

kubectl & the object model

What Kubernetes isKubernetes fundamentals Security domains & threat modelKubernetes security & hardening The cluster attack surfaceKubernetes attack & defense Kubernetes interview questionsInterview prep

Cluster architecture

The control plane: four programs, one job each

What really happens when you run kubectl apply

The worker node, and why NotReady means you get on the box

Try this

Takeaway

Related