>_SecOpsLog_
CoursesKubernetes administrationControllers & reconciliation

Controllers & reconciliation

The loops that turn desired state into running pods.

Intermediate12 min · lesson 5 of 65
In plain terms
A controller is a thermostat: it constantly compares the temperature you set (what you want) to the room (what is) and nudges the heater until they match. Kubernetes is a whole house full of little thermostats.

You never actually tell Kubernetes to start a Pod. A Pod is the smallest thing the cluster runs, one or more containers that share a single network address, and there is no 'run this one right now' button anywhere in the system. You write down what you want to exist, and something else makes it true and keeps it true. That something is a controller, and the loop it runs is why a cluster quietly heals itself at 3am while you're asleep.

The loop that never ends

Think of a thermostat. You set it to 21 degrees. It reads the room, sees 19, switches on the heat, reads again, and keeps nudging until the room matches the number you asked for. It never stops checking. A controller is that same idea pointed at a cluster. You declare 'I want three copies of this web server running.' The controller reads the cluster, counts two, and creates one more. A node dies and takes a copy with it. The controller counts two again, and makes a third.

Two words carry the whole model. The spec is your declared wish, the desired state. The status is what is actually true right now, the observed state. Every controller has exactly one job: drag status toward spec, over and over, and never stop.

Here is the part that trips people up. A controller does not react to events one at a time, as in 'a Pod was deleted, therefore create a Pod.' It reacts to the current gap between spec and status, however that gap got there. Think of two night guards. One walks the whole building every round and fixes whatever looks wrong. The other sits by the alarm panel and only moves when a specific bell rings. Miss the bell and the second guard never knows it happened. The first one catches it on the next lap. Kubernetes controllers are the first guard, and engineers call this style level-triggered rather than edge-triggered. It is why controllers are so hard to wedge. Miss a notification, get a duplicate, crash halfway through an action, and none of it matters. Next time around the loop the controller re-reads reality and does whatever is needed. A dropped event never leaves the cluster stuck in a wrong state.

Controllers also don't hammer the API server (the front door to the cluster's database) with constant polling. Each one opens a watch, a long-lived subscription, and the API server pushes changes as they happen. Dozens of these controllers live inside a single process called the kube-controller-manager: one for ReplicaSets, one for Jobs, one for Nodes, one for endpoints, and on it goes. Each watches its own kind of object and reconciles it.

The reconciliation loop
1WatchThe controller subscribes to…2ObserveRead live reality: how many…3DiffCompare desired state (spec)…4ActCreate, update, or delete…5RepeatRe-queue and start over,…

Watch it heal itself

Enough theory. Make a Deployment (a controller that keeps a set of identical Pods running) and watch the loop work.

terminal
kubectl create deployment web --image=nginx:1.27 --replicas=3
kubectl get pods -l app=web
output
deployment.apps/web created
NAME                  READY   STATUS    RESTARTS   AGE
web-6b7f8c9d4-4qk9t   1/1     Running   0          9s
web-6b7f8c9d4-hs2vp   1/1     Running   0          9s
web-6b7f8c9d4-zt7mb   1/1     Running   0          9s

Now delete one Pod by hand, the way a crash or a dead node would, and list again straight away.

terminal
kubectl delete pod web-6b7f8c9d4-4qk9t
kubectl get pods -l app=web
output
pod "web-6b7f8c9d4-4qk9t" deleted
NAME                  READY   STATUS              RESTARTS   AGE
web-6b7f8c9d4-hs2vp   1/1     Running             0          51s
web-6b7f8c9d4-zt7mb   1/1     Running             0          51s
web-6b7f8c9d4-n8xql   0/1     ContainerCreating   0          2s

You didn't ask for that new Pod. You never do. The ReplicaSet controller saw its count drop from three to two and closed the gap. You can catch it in the act in the ReplicaSet's own event log, which names the exact controller that acted.

terminal
kubectl describe rs -l app=web | grep -A4 Events
output
Events:
  Type    Reason            Age   From                   Message
  ----    ------            ----  ----                   -------
  Normal  SuccessfulCreate  2s    replicaset-controller  Created pod: web-6b7f8c9d4-n8xql

How the controller knows which pods are its

When you create a Deployment you don't get Pods directly. The Deployment makes a ReplicaSet, and the ReplicaSet makes the Pods. Three layers. Each Pod carries a little note of parentage called an ownerReference that points back up to the ReplicaSet that made it. That note is how the controller knows which Pods to count as its own, and it is how a cascade delete finds everything to clean up when you remove the Deployment.

terminal
kubectl get pod web-6b7f8c9d4-n8xql \
  -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
kubectl get rs -l app=web -o custom-columns=\
NAME:.metadata.name,DESIRED:.spec.replicas,CURRENT:.status.replicas,READY:.status.readyReplicas
output
ReplicaSet/web-6b7f8c9d4
NAME            DESIRED   CURRENT   READY
web-6b7f8c9d4   3         3         3

DESIRED comes from spec, what you asked for. CURRENT and READY come from status, what the controller has actually achieved. When those numbers match, the loop is quiet. When they stay apart, a controller is either mid-fix or stuck, and that gap is the first thing to check when something feels off.

When the loops stop

Reconciliation has to run somewhere, and that somewhere can fail. If the kube-controller-manager stops, desired state quietly stops being enforced. Already-running Pods keep running, because the kubelet (the agent on each node that actually starts and watches containers) babysits them on its own, with no controller involved. So the cluster still looks healthy. But delete a Pod now and nothing replaces it. Kick off a rollout and it just hangs there. A node dies in the night and no one notices. The lights are on and nobody's reconciling.

So when self-healing seems broken, don't stare at the Pod. Look one level up. Is the controller-manager even running? Does the ReplicaSet show DESIRED 3 but CURRENT 2 and never catch up? Are there FailedCreate events complaining about a resource quota or a missing service account? The controller almost always tells you why it can't converge, right there in the events.

terminal
kubectl -n kube-system get pods -l component=kube-controller-manager
kubectl -n kube-system get lease kube-controller-manager
output
NAME                           READY   STATUS    RESTARTS   AGE
kube-controller-manager-cp-1   1/1     Running   2          6d
NAME                      HOLDER                          AGE
kube-controller-manager   cp-1_9f3c1a2b-7d4e-4c8a-b1e2   6d
One controller-manager is awake; the others are only watching
In a highly available control plane you run three kube-controller-manager pods, but only one does any work. They compete for a single Lease object in the kube-system namespace, and only the holder reconciles. The other two sit idle as hot standbys. If the leader can't renew its lease (a network blip, a slow etcd, which is the cluster's key-value database, or clock skew between nodes), leadership should jump to a standby within a few seconds. When it doesn't, because all three are wedged or leader election got disabled, every control loop stops at once while all three pods still report Running. Your dashboards stay green and nothing heals. Watch the HOLDER field on that lease. If it never changes during an incident, or points at a node that is already dead, you've found your stall.

Owner references and garbage collection explain why deleting a Deployment removes its ReplicaSets and pods. Break the owner link carefully or you orphan workloads.

If two controllers fight over the same object, you get thrash. Prefer one clear owner for each concern.

Status conditions are how controllers report progress. Read them before you restart random pods.

Try this

Create a Deployment, delete one pod, and watch the ReplicaSet recreate it. That correcting loop is reconciliation — desired state pulling reality back into line.

terminal
$ kubectl create deployment web --image=nginx:1.27 --replicas=3
$ kubectl get pods -l app=web
 
$ kubectl delete pod web-6b7f8c9d4-4qk9t
$ kubectl get pods -l app=web
 
$ kubectl describe rs -l app=web | grep -A4 Events
 
$ kubectl get pod web-6b7f8c9d4-n8xql \
  -o jsonpath='{.metadata.ownerReferences[0].kind}/{.metadata.ownerReferences[0].name}{"\n"}'
$ kubectl get rs -l app=web -o custom-columns=\
$ NAME:.metadata.name,DESIRED:.spec.replicas,CURRENT:.status.replicas,READY:.status.readyReplicas
 
$ kubectl -n kube-system get pods -l component=kube-controller-manager
$ kubectl -n kube-system get lease kube-controller-manager
 
NAME                           READY   STATUS    RESTARTS   AGE
kube-controller-manager-cp-1   1/1     Running   2          6d
NAME                      HOLDER                          AGE
kube-controller-manager   cp-1_9f3c1a2b-7d4e-4c8a-b1e2   6d

Takeaway

Controllers do not push once and forget. They watch, compare, and act forever. Your YAML is a thermostat setting, not a one-shot script.

Quick check
01Your Deployment is rolling out a broken image and every new Pod crashes. A teammate keeps running kubectl delete pod on the crashing Pods to 'clear them out,' but fresh broken Pods keep appearing. What is actually happening?
Incorrect — No. The deletes succeed. The old Pods really are removed, then brand-new ones appear with different names.
Correct — Deleting a Pod only changes observed state. The desired state, the broken image in the template, never changed, so the controller keeps making Pods that match it.
Incorrect — No. etcd stores the current live objects. There is no stale cache re-emitting deleted Pods.
Incorrect — No. These are entirely new Pods with new names, created by the ReplicaSet controller, not kubelet restarts of one Pod.
02A controller can miss a change notification, receive a duplicate, or crash halfway through an action and still leave the cluster in the right state. What design makes that true?
Incorrect — the lesson's whole point is that delivery is not relied on; missed or duplicate events are tolerated by design.
Incorrect — that is the guard who only moves when the bell rings and misses it; Kubernetes controllers are deliberately not edge-triggered.
Correct — like the guard who walks the whole building each round, it fixes whatever is wrong now, so a dropped event never wedges it.
Incorrect — it reads current state, not a full history replay; the robustness comes from level-triggered reconciliation, not replay.
03In a highly available control plane, self-healing has stopped cluster-wide: deleted Pods aren't replaced and rollouts hang, yet all three kube-controller-manager Pods show Running and the dashboards are green. Where do you look first?
Incorrect — the kubelet babysits already-running containers but does not run the reconciliation loops; the controller-manager does.
Correct — the three Pods compete for one Lease and only the holder works; a leader that can't renew, or leader election disabled, halts all loops with Pods still Running.
Incorrect — a full etcd is one failure mode, but here the symptom is leader election, and etcd would raise its own NOSPACE alarm.
Incorrect — the symptom is that nothing is being created to schedule in the first place, which points at the controllers, not the scheduler.
etcd — the cluster storeKubelet & kube-proxy

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep