>_SecOpsLog_
CoursesKubernetes administrationStatic pods

Static pods

Pods the kubelet runs directly, without the API.

Advanced10 min · lesson 29 of 65
In plain terms
A static pod runs straight from a note pinned to the node’s fridge — no manager involved. It’s how the control plane starts itself up before the manager even exists.

Delete the API server and it comes back before you can blink. Try it on a kubeadm cluster: kubectl delete pod kube-apiserver-cp1 -n kube-system prints deleted, and a second later the pod is Running again with an age of a few seconds. No Deployment did that, no ReplicaSet, no scheduler, not even a controller. None of the usual machinery that creates pods and restarts them when they die was involved at all. The API server is a static pod, and static pods answer to exactly one thing: a file on a node's disk.

Think of a sticky note pinned to the fridge that reads keep the coffee pot full. Nobody manages it. There's no shift lead handing out tasks. Whoever is in the kitchen just reads the note and does what it says. A static pod is that note. The kitchen worker is the kubelet, the agent Kubernetes runs on every node whose whole job is to keep that node's containers matching what it has been told to run. Normally the kubelet takes its orders from the API server, the front door to the cluster's shared state (which lives in etcd, a key-value database). A static pod skips all of that. The kubelet reads a Pod manifest straight off the local disk and runs it. A Pod, for the record, is Kubernetes' smallest unit: one or more containers that share a network address and get placed together.

The kubelet runs it, and nobody else can touch it

A kubelet only watches a directory if you point it at one. The path comes from staticPodPath in the kubelet's own config file (older setups pass --pod-manifest-path on the command line instead), and it has no default: a kubelet with neither set runs no static pods at all. kubeadm writes /etc/kubernetes/manifests into that config on every node it sets up, which is why the directory is there on a kubeadm cluster. Other distributions pick their own path or leave it unset, so check before you assume. Once a path is set, drop a Pod YAML in there and the kubelet starts it within a second or two. Edit the file and it restarts the pod. Delete the file and the pod goes away. The kubelet spots changes by hashing the Pod object it parses out of the file, not the raw bytes, so a real change to the spec triggers a restart while adding a comment, reindenting, or merely touching the timestamp does nothing.

inspect the static pod directory on a control-plane node
sudo ls /etc/kubernetes/manifests/
sudo grep staticPodPath /var/lib/kubelet/config.yaml
output
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
staticPodPath: /etc/kubernetes/manifests

Here's the wrinkle that trips people up. The kubelet also copies each static pod up into the API server, so you can still find it with your everyday tools. That copy is called a mirror pod, and it's read-only. You can kubectl get it and kubectl describe it, but you cannot manage it. Delete the mirror and the kubelet recreates it from the file immediately, because the file, not the API, is the source of truth. The mirror even wears its origin on its sleeve. Its name is the pod's name with the node name stuck on the end.

Make one yourself

You don't need to touch the control plane to watch this work. SSH to a node and run that grep first: if staticPodPath comes back empty, the kubelet there is watching nothing and a file you drop in will just sit on disk. When the path is set, write a manifest into that directory. Within seconds the kubelet runs it, and a mirror shows up in kubectl with that give-away node suffix.

/etc/kubernetes/manifests/web.yaml (written on node worker-1)
apiVersion: v1
kind: Pod
metadata:
  name: web
  namespace: default
spec:
  containers:
  - name: nginx
    image: nginx:1.27
    ports:
    - containerPort: 80
check it from the control plane
kubectl get pod -o wide
output
NAME           READY   STATUS    RESTARTS   AGE   IP           NODE       NOMINATED NODE   READINESS GATES
web-worker-1   1/1     Running   0          18s   10.244.1.7   worker-1   <none>           <none>

See the name? web-worker-1, not plain web. That node suffix is the tell-tale sign of a static pod's mirror. Now try to kill it the ordinary way.

delete the mirror, then look again
kubectl delete pod web-worker-1
kubectl get pod web-worker-1
output
pod "web-worker-1" deleted
NAME           READY   STATUS    RESTARTS   AGE
web-worker-1   1/1     Running   0          3s

It's back with a three-second age. The delete removed the mirror object, the kubelet saw the file still sitting on disk, and it rebuilt the pod. To actually remove a static pod you delete its manifest file or move it out of the watched directory: sudo rm /etc/kubernetes/manifests/web.yaml, and within a couple of seconds the pod and its mirror both disappear.

Why the control plane is built this way

Here's the payoff. How do you start a cluster when the thing that places pods (the scheduler) and the thing that stores their definitions (the API server, backed by etcd) are themselves pods that need to be started? You can't schedule the scheduler through the scheduler. kubeadm's answer is static pods. At boot the kubelet reads etcd.yaml, kube-apiserver.yaml, kube-scheduler.yaml and kube-controller-manager.yaml from the manifest directory and runs them directly, with no API involved. Once the API server is up, the rest of the cluster talks to it normally.

This is also why, when the control plane is on fire, kubectl is not your first tool. If the API server static pod is misconfigured and there is no healthy replica left to answer, kubectl is precisely the thing that stops working. You go to the node and ask the container runtime directly with crictl, the low-level client for the runtime that actually runs the containers, and you read the kubelet's own logs with journalctl -u kubelet.

debug on the node when kubectl is dead
sudo crictl ps -a --name kube-apiserver
output (a bad edit put the apiserver into a crash loop)
CONTAINER      IMAGE          CREATED          STATE    NAME             ATTEMPT   POD ID
b3f1a9c2d7e4   6f7e3a1b2c9d   18 seconds ago   Exited   kube-apiserver   4         9a8b7c6d5e4f

A rising ATTEMPT count and an Exited state say the container keeps dying. Pull the reason with sudo crictl logs b3f1a9c2d7e4, which will usually spell out the exact bad flag or unparseable value, then fix the file and let the kubelet restart it.

Where a static pod actually lives
Control-plane node: the source of truth
/etc/kubernetes/manifests/*.yaml
you edit these files directly; nothing else defines the pod
kubelet
watches the directory, hashes the pod it parses from each file, runs it with no scheduler and no API
running containers (etcd, apiserver, ...)
started straight through the container runtime, up even if the API is down
API server: a read-only window
mirror pod: name has the node suffix
kubectl get / describe can see it
kubectl delete <mirror>
removes the mirror only; the kubelet rebuilds it from the file in seconds
The file on the node is authoritative. The API holds only a read-only mirror, so you change static pods by editing files on the node and debug them there, not through kubectl.
Editing a control-plane static pod has no undo
When you open kube-apiserver.yaml and add a flag, there is no dry run, no admission check, and no rollback. The moment you save, the kubelet parses the file, hashes the pod it got out of it, and restarts the container. If you fat-fingered a flag or set a value the apiserver rejects, the API server on that node crash-loops. On a single control-plane cluster kubectl goes dark for everyone, and nothing records the failure, because the component that would write the event is the one you just broke. On an HA control plane the load balancer routes around the dead replica, so kubectl keeps answering and the only sign is one mirror pod stuck in CrashLoopBackOff, which is easy to miss for hours. Always copy the manifest somewhere outside the manifests directory before editing (sudo cp kube-apiserver.yaml /root/kube-apiserver.bak), change one thing at a time, and keep a second terminal on the node running crictl ps and journalctl -u kubelet so you can see it recover or catch the error and fix the file forward.

This is how self-hosted control planes boot before the API is healthy. That circular dependency is intentional.

Moving a manifest out of the directory is a stop, and moving it back is a start. That is the usual way to restart a control-plane component by hand. What bites people is where the copy lands. The kubelet parses every file in the watched directory that does not begin with a dot, whatever the extension, so a kube-apiserver.yaml.bak left beside the original is read as a second manifest. Keep the same pod name in it and one of the two copies is ignored with a duplicate-name warning in the kubelet log, which may not be the copy you expect. Change the name and both try to run and fight over the same host port. Save backups somewhere else, as the box above does with /root.

Never delete only the mirror pod and expect the workload to die. Delete the manifest, or move it out of the watched directory, and the container and its mirror both go within seconds.

Try this

On a kubeadm control-plane node, list the manifest directory and match every filename to a mirror pod in kube-system. The config.source annotation is the giveaway: only pods the kubelet built from a file on that node carry the value file. Read etcd.yaml if you like, but do not edit it unless you mean it.

terminal
sudo ls /etc/kubernetes/manifests/
 
kubectl get pods -n kube-system \
  -o custom-columns='NAME:.metadata.name,SOURCE:.metadata.annotations.kubernetes\.io/config\.source'
output
etcd.yaml  kube-apiserver.yaml  kube-controller-manager.yaml  kube-scheduler.yaml
 
NAME                          SOURCE
coredns-76f75df574-7q6bp      <none>
coredns-76f75df574-lqp2t      <none>
etcd-cp1                      file
kube-apiserver-cp1            file
kube-controller-manager-cp1   file
kube-proxy-9v4kx              <none>
kube-scheduler-cp1            file

Four files, four pods marked file, each wearing the node name cp1. Everything else in kube-system came from the API, created by a Deployment or a DaemonSet, and has no source annotation at all.

Takeaway

Static pods are kubelet-local from manifests on disk. The API shows a mirror pod, but the kubelet is the source of truth.

Quick check
01You edit /etc/kubernetes/manifests/kube-apiserver.yaml on the control-plane node, add --max-requests-inflightt=800 (note the typo), and save. What happens next?
Incorrect — Static pods are never applied through the API. The kubelet reads the file off disk on its own; kubectl apply has no role here.
Correct — The kubelet reconciles from the file, the apiserver rejects the unknown flag and exits. On a single control-plane cluster kubectl dies with it; on an HA cluster the load balancer hides the loss and only that mirror pod looks wrong. Either way you diagnose and fix it on the node.
Incorrect — There is no revision history for a static pod. The file is the only source of truth, so there is nothing to roll back to and no controller watching it.
Incorrect — The scheduler never touches static pods. They are pinned to whichever node holds the manifest, so nothing moves and nothing picks up the work.
02Why does kubeadm run etcd, the API server, the scheduler, and the controller manager as static pods rather than as Deployments?
Incorrect — No: it is not about speed; it is about bootstrapping when no API server or scheduler exists yet.
Correct — you cannot schedule the scheduler through the scheduler, so the kubelet reads the files directly with no API involved.
Incorrect — No: static pods have no replica field; they are one-per-node from a file, not scaled by a controller.
Incorrect — No: the scheduler never touches static pods, and at boot it is not even running yet.
03You want to stop the web static pod on worker-1. kubectl delete pod web-worker-1 reports 'deleted', but seconds later it's Running again with a 3-second age. Why, and how do you actually remove it?
Incorrect — No: static pods have no ReplicaSet, so there is no controller object to scale.
Incorrect — No: it fully deleted, and the kubelet then rebuilt it from the file, so force-deleting the mirror again changes nothing.
Correct — the on-disk file is the source of truth, so remove it or move it out of the directory (e.g. sudo rm .../web.yaml) and both the pod and its mirror disappear.
Incorrect — No: the scheduler never places static pods; they are pinned to whichever node holds the manifest file.
DaemonSetsMultiple schedulers & profiles

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep