>_SecOpsLog_
CoursesKubernetes fundamentalsThe cluster: control plane & nodes

The cluster: control plane & nodes

The two halves of every cluster.

Beginner10 min · lesson 3 of 24
In plain terms
A cluster is a restaurant. The control plane is the manager and the order book — it decides and remembers. The worker nodes are the kitchen — they do the cooking. Kubernetes is how the two coordinate.

A busy restaurant runs on a split you can see from the door. Out front, someone takes your order, writes it down, and keeps track of what has been cooked and what is still waiting. Back in the kitchen, the cooks make the food. Neither side does the other's job. They stay in sync by passing tickets back and forth. A Kubernetes cluster is built the same way. The word cluster means a group of computers, real or virtual, that Kubernetes ties together so you can treat them as one big machine.

Two halves, two jobs

Every cluster splits into two roles. The control plane is the front of the restaurant. It decides, and it remembers. The worker nodes are the kitchen. They run your actual programs. A node is one machine in the cluster, and a worker node is a machine whose whole purpose is running your apps. A practice cluster on your laptop might have one machine wearing both hats at once. A real production cluster can spread the same two roles across dozens or hundreds of machines. The split is about jobs, not about how many computers you own.

Your apps run as containers. A container is a program packed up with everything it needs, so it behaves the same on your laptop and on a server in a data centre. Kubernetes never hands you a bare container, though. It wraps one or more containers in a Pod, the smallest unit Kubernetes will schedule and start. Most Pods hold a single container, so for now you can read one Pod as one running program: a small box with your container inside it.

Here is where almost every beginner trips. You do not log into the worker machines to start your apps. You talk to the control plane and nothing else, and it arranges for the workers to do the work. Want three copies of your app running at all times? You say that to the control plane, and it works out where and how. The tool you use for that conversation is kubectl (say it "kube control"), the command-line program that carries your requests to the cluster. It gets a lesson of its own next.

terminal
kubectl get nodes
output
NAME       STATUS   ROLES           AGE   VERSION
cp-1       Ready    control-plane   9d    v1.31.0
worker-1   Ready    <none>          9d    v1.31.0
worker-2   Ready    <none>          9d    v1.31.0

Read that output top to bottom. The machine cp-1 carries the control-plane role, so that one is the brain. The two workers show <none> under ROLES, which means they are ordinary worker machines with no special duty beyond running your apps. Ready means the machine is healthy and available for work. AGE is how long it has been part of the cluster, and VERSION is the Kubernetes version it runs. You talk to cp-1. It tells worker-1 and worker-2 what to run.

What the control plane is made of

You do not need to memorize the internals yet, but a short tour makes the rest of the course easier. The control plane is a handful of small programs working together. The API server is the single front door, and every request goes through it, including yours. (API stands for Application Programming Interface, a formal name for the doorway one program uses to talk to another.) Behind that door sits a store called etcd, the cluster's memory: a written record of everything that is supposed to exist. The scheduler is the host with the seating chart, deciding which worker node each new Pod lands on. The controllers behave like a thermostat. A thermostat compares the room against the temperature you set and turns on the heat when the two do not match. Controllers do exactly that for the cluster, over and over, comparing what is actually running against what you asked for and closing any gap. A Pod dies at 3am, a controller sees the gap, a replacement starts. Nobody gets paged.

terminal
kubectl get pods -n kube-system
output
NAME                            READY   STATUS    RESTARTS   AGE
coredns-7db6d8ff4d-abcde        1/1     Running   0          9d
etcd-cp-1                       1/1     Running   0          9d
kube-apiserver-cp-1             1/1     Running   0          9d
kube-controller-manager-cp-1    1/1     Running   0          9d
kube-proxy-2xk9p                1/1     Running   0          9d
kube-scheduler-cp-1             1/1     Running   0          9d

The -n kube-system part points kubectl at kube-system, the built-in area where Kubernetes keeps its own internal Pods. Every control-plane piece runs as a Pod in there: kube-apiserver, etcd, kube-scheduler and kube-controller-manager, plus coredns (the cluster's own DNS, or Domain Name System, the phone book that turns names into addresses) and kube-proxy for networking. One heads-up. On a managed cluster from a cloud provider, the control-plane entries are missing from this list, because the provider runs the control plane for you and keeps it out of sight.

What a worker node does all day

Each worker runs a small agent called the kubelet. The kubelet is the cook reading the order ticket. It takes instructions from the control plane, starts the containers it is told to start, watches them, and reports their health back up the line. Beside it, a networking piece wires the Pods together so they can reach each other and the outside world. That is the whole job. Workers wait for orders and carry them out.

That steady reporting doubles as the cluster's early-warning system. When a worker reboots, fills its disk, or falls off the network, its kubelet goes quiet, and the control plane notices the silence. Run the same command on a bad day and the machine tells on itself:

terminal
kubectl get nodes
output
NAME       STATUS     ROLES           AGE   VERSION
cp-1       Ready      control-plane   9d    v1.31.0
worker-1   Ready      <none>          9d    v1.31.0
worker-2   NotReady   <none>          9d    v1.31.0

worker-2 has gone NotReady. To find out why, ask that node to describe itself. The Conditions and Events near the bottom of the output spell it out in plain words:

terminal
kubectl describe node worker-2
output
Name:         worker-2
Conditions:
  Type     Status    Reason              Message
  ----     ------    ------              -------
  Ready    Unknown   NodeStatusUnknown   Kubelet stopped posting node status.
Events:
  Type     Reason         Age   From             Message
  ----     ------         ----  ----             -------
  Normal   NodeNotReady   3m    node-controller  Node worker-2 status is now: NodeNotReady

Status Unknown, with the message "Kubelet stopped posting node status", means the machine went quiet and the control plane can no longer trust what is happening on it. Look at who raised the event: node-controller. That is one of those thermostat controllers acting on its own, with no human involved. After a short grace period it moves worker-2's Pods onto the healthy nodes, so your app keeps serving traffic while you go and work out whether that box rebooted or filled its disk.

terminal
kubectl cluster-info
output
Kubernetes control plane is running at https://192.168.49.2:8443
CoreDNS is running at https://192.168.49.2:8443/api/v1/namespaces/kube-system/services/kube-dns:dns/proxy
 
To further debug and diagnose cluster problems, use 'kubectl cluster-info dump'.

That address is the API server, the front door you send everything to. Notice there is one URL for the whole cluster, not one per machine. You never point kubectl at a worker. Send your request to that single address and the control plane takes over: it writes down what you asked for, the scheduler picks a node, and the kubelet on that node starts your containers. You describe the goal. The cluster works out the steps.

How a cluster is organized
Control plane: the brain (decides and remembers)
API server
the front door you talk to
etcd store
the cluster's memory
Scheduler
picks which worker runs each Pod
Controllers
keep reality matching your request
Worker nodes: the muscle (run your apps)
kubelet
starts and watches your containers
Networking
connects the Pods

Every cluster has two jobs: decide, and do. The control plane (API server, etcd, scheduler, controller manager) decides. The worker nodes run the kubelet, a container runtime (the software that actually starts and stops containers) and usually a network agent, and they do. When something breaks, the first useful question is which half owns the symptom. Answer that and you stop firing off random kubectl commands and hoping.

etcd is the durable memory of the cluster, but only the API server is allowed to talk to it. Everything else goes through the API: kubectl, the controllers, the kubelets on every node. That is why a sick API server feels like the whole world is on fire even when your apps are still serving. It is also why protecting and backing up that one path is worth far more of your attention than any pretty dashboard.

Say your production cluster shows every node Ready but Pods sitting in Pending. That usually points at the scheduler or at plain capacity, not at "Kubernetes is down". Read the node conditions and the events first. Rebooting control-plane virtual machines in a panic is how a small problem turns into a real outage.

Try this

Map your own cluster. List the nodes with their roles and conditions, then list the control-plane Pods in kube-system, and you can see exactly who lives where.

terminal
$ kubectl get nodes -o wide
NAME                 STATUS   ROLES           AGE   VERSION   INTERNAL-IP
kind-control-plane   Ready    control-plane   12d   v1.30.0   172.18.0.2
kind-worker          Ready    <none>          12d   v1.30.0   172.18.0.3
 
$ kubectl get pods -n kube-system -o wide | head -20
NAME                                         READY   STATUS    NODE
coredns-...                                  1/1     Running   kind-control-plane
etcd-kind-control-plane                      1/1     Running   kind-control-plane
kube-apiserver-kind-control-plane            1/1     Running   kind-control-plane
kube-controller-manager-kind-control-plane   1/1     Running   kind-control-plane
kube-scheduler-kind-control-plane            1/1     Running   kind-control-plane
kube-proxy-...                               1/1     Running   kind-worker
 
$ kubectl describe node kind-worker | Select-String -Pattern 'Conditions:|Ready|MemoryPressure|DiskPressure' -Context 0,6
Ready                 True

Takeaway

Control plane decides, nodes execute. When a symptom shows up, name the half that owns it before you touch anything. A Pod stuck in Pending is a scheduling question. A node stuck at NotReady is a kubelet question. A kubectl command that hangs is an API server question. Aim at that one layer instead of rebooting machines and hoping the symptom goes away.

Fixing it by hand on the worker will not stick
Something breaks and the instinct is to open a shell on worker-1 and restart the container yourself. Do that and one of two things happens. The control plane notices reality no longer matches what you asked for and quietly undoes your change, or your fix disappears the moment that Pod moves to another machine. Tell the control plane what you want with kubectl instead, and let it push the change out to the workers.
Quick check
01You have packaged a new app and you want the cluster to run it. Who do you send that request to?
Incorrect — This is the classic beginner move. You almost never touch a worker directly, and anything you change by hand gets overwritten.
Correct — You describe what you want to the control plane, and it places your app on a worker and keeps it running.
Incorrect — The kubelet takes orders from the control plane only, never from you.
Incorrect — etcd only remembers state. You never write to it yourself; the API server does that on your behalf.
02Inside the control plane, which piece decides which worker node a brand-new Pod will run on?
Correct — It is the host with the seating chart, choosing which worker node each new Pod lands on.
Incorrect — No. etcd is the cluster's memory, a record of what is supposed to exist. It does not place Pods.
Incorrect — No. The kubelet sits on each worker and starts the containers it is told to start. It does not choose the node.
Incorrect — No. The API server (Application Programming Interface server) is the front door every request passes through, but placing Pods on nodes is the scheduler's job.
03kubectl get nodes shows cp-1 and worker-1 as Ready, but worker-2 as NotReady. kubectl describe node worker-2 reports Ready = Unknown with the message "Kubelet stopped posting node status." What happened, and what does Kubernetes do about it?
Incorrect — No. cp-1 and worker-1 are still Ready and still serving. Only worker-2 went quiet.
Correct — The control plane stops trusting a node whose kubelet has stopped reporting, and a controller reschedules its Pods so the app keeps serving while you fix the machine.
Incorrect — No. You do not restart Pods on a worker by hand. A controller reschedules them onto healthy nodes for you.
Incorrect — No. The Pods are recreated on healthy nodes, because the control plane keeps working to make reality match what you asked for.
Containers in a nutshellMeet kubectl

Related

Cluster architectureKubernetes administrationSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep