kubectl returns 'connection refused'. You SSH to the control-plane node: the kubelet is active (running), but crictl ps -a shows no kube-apiserver container at all, not even a crash-looping one. etcd, the scheduler, and the controller-manager are all Running. What's the most likely cause?

A syntax error in kube-apiserver.yaml that the kubelet can't parse, so it never creates the pod. A manifest the kubelet can't parse is silently skipped, so the container never appears at all. That matches a missing container exactly. Confirm it with journalctl -u kubelet.

A cluster runs a 3-member etcd, and two members are lost at once. etcd is the only record of every object, and the API server keeps no state of its own. What happens?

etcd loses quorum and deliberately stops accepting writes, dragging the whole control plane down. A majority must agree before etcd records a change, so with only one of three it halts writes to avoid splitting into two disagreeing copies of the truth.

kubectl died overnight with 'connection refused' though nobody deployed anything. On the node, kubeadm certs check-expiration shows every leaf cert as while the CA is still valid for years. What's the correct recovery?

Run kubeadm certs renew all, bounce the static pods, then refresh your kubeconfig from admin.conf. Renewing the leaves fixes the expiry, static pods only pick up new certs when you make the kubelet recreate them, and ~/.kube/config still holds the old expired client cert until you copy the new /etc/kubernetes/admin.conf over it.

Courses Kubernetes administrationControl-plane failure

Control-plane failure

When the API server or a controller is down.

Advanced12 min · lesson 63 of 65

In plain terms

When the front desk (the API server) itself is dark, you can’t phone ahead — you walk to the building and check the breaker panel by hand (crictl on the node). It’s usually an expired badge or a bad note left on the fridge.

There is no pause and no spinner. kubectl get pods comes straight back with The connection to the server 10.0.0.10:6443 was refused. You run it again and it fails just as fast, because a refused connection is the kernel telling you nothing is listening on that port at all. kubectl get nodes, kubectl describe, all dead. This is the outage that unsettles people, because the tool you'd normally grab to investigate an outage is the exact thing that just broke.

Think of the API server as the front desk of a large building. Application Programming Interface is a mouthful, so just picture a reception desk that every request passes through: yours, the scheduler's, every node's agent. When the desk is staffed you phone ahead and someone looks things up for you. When it goes dark, the phone just rings out. You stop redialing. You walk over and check the breaker panel yourself. For Kubernetes, walking over means logging into the control-plane node over SSH (a secure remote login to the machine) and inspecting the layer just below the API. It's a different discipline from day-to-day kubectl, and it's the one that gets your cluster back.

The one thing that stays up

Here's why recovery is even possible: the control-plane components don't run the way your apps do. The control plane is the cluster's brain. It's a handful of programs that make the decisions: the API server out front, the scheduler that picks which node each pod runs on, the controller-manager that keeps nudging the cluster toward the state you asked for, and etcd, the database that remembers all of it. Your apps are ordinary pods (a pod is Kubernetes' unit of one or more containers that run together): you hand one to the API server, which writes it down, and the scheduler picks the node it lands on. Those four brain programs don't get placed that way. They run as static pods instead. A static pod is one the kubelet launches straight from a file on disk, without asking the API server first. The kubelet does report a read-only copy of each one, called a mirror pod, back to the API server whenever it can reach it, which is why control-plane pods show up in kubectl get pods -n kube-system and why editing them there changes nothing.

The kubelet is the agent that runs on every node, the process that actually starts and stops containers. On a control-plane node it watches one folder, /etc/kubernetes/manifests/. Drop a pod definition in there and the kubelet runs it. Edit the file and it restarts the pod. Remove the file and it stops the pod. All of that reads from local disk, so the kubelet keeps the control plane alive even when the API server it normally reports to is down. That's your way in. kubectl is blind because it needs the API server, but the kubelet and the container runtime beneath it are still fully awake on the node. crictl is how you talk to them: a command-line inspector for the container runtime (the low-level software that actually runs containers, usually containerd or CRI-O), sitting at the same layer kubectl normally hides from you.

on the node: agent alive? static pods?

# kubectl is dead: SSH to the control-plane node and work below the API.
sudo systemctl status kubelet    # is the node agent itself alive?
sudo crictl ps -a                # what state are the static pods in?

expected output

● kubelet.service - kubelet: The Kubernetes Node Agent
     Loaded: loaded (/usr/lib/systemd/system/kubelet.service; enabled; preset: enabled)
    Drop-In: /usr/lib/systemd/system/kubelet.service.d
             └─10-kubeadm.conf
     Active: active (running) since Thu 2026-07-16 03:11:42 UTC; 2h 4min ago
   Main PID: 1148 (kubelet)
 
CONTAINER      IMAGE          CREATED         STATE     NAME                      ATTEMPT   POD ID          POD
7d9f0a1b2c3d   c3994bc69610   9 seconds ago   Exited    kube-apiserver            12        f00dcafe1234    kube-apiserver-cp-1
b4c5d6e7f8a9   73deb9a3f702   2 hours ago     Running   etcd                      0         a11ce7feed01    etcd-cp-1
c5d6e7f8a9b0   45b3524d38f8   2 hours ago     Running   kube-controller-manager   0         b22df8affe12    kube-controller-manager-cp-1
d6e7f8a9b0c1   e3f5c9d0a1b2   2 hours ago     Running   kube-scheduler            0         c33ea9bccf23    kube-scheduler-cp-1

Read that output the way a first responder reads vitals. The kubelet is active (running), so the node agent is fine. etcd, the controller-manager, and the scheduler are all Running. The API server is Exited on its twelfth attempt, which means it's crash-looping: the kubelet keeps restarting it and it keeps dying. That narrows the whole problem down to one container. Now you find out why it won't stay up, and there are three usual answers.

Certs, then the manifest, then etcd

First and most common, an expired certificate. Kubernetes components prove who they are to each other with TLS (Transport Layer Security) certificates, like staff badges that open the doors inside the building. kubeadm hands them out with a one-year lifetime by default. When they lapse you get one of two very different pictures. If the badge the API server uses to reach etcd has expired, it never gets as far as serving: it fails on the way up, the kubelet restarts it, it fails again, and because nothing is listening on 6443 you get connection refused (that's the crash-loop in the output above). If instead it's the serving certificate, the one the API server shows to you, the process runs happily and the port answers, but every request is turned away with x509: certificate has expired or is not yet valid. Either way the cluster looks stone dead, even though no code changed and nobody deployed anything. The giveaway is the timing: nobody touched a thing, and it fell over anyway, usually right at the 365-day mark and usually overnight. Clusters that get upgraded on a schedule never see this, because kubeadm upgrade renews the certs as a side effect. The cluster that dies is the one nobody has touched in a year.

check certificate expiry

sudo kubeadm certs check-expiration

expected output: leaf certs expired, the CA is still good

CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jul 15, 2026 09:12 UTC   <invalid>       ca                      no
apiserver                  Jul 15, 2026 09:12 UTC   <invalid>       ca                      no
apiserver-etcd-client      Jul 15, 2026 09:12 UTC   <invalid>       etcd-ca                 no
apiserver-kubelet-client   Jul 15, 2026 09:12 UTC   <invalid>       ca                      no
controller-manager.conf    Jul 15, 2026 09:12 UTC   <invalid>       ca                      no
scheduler.conf             Jul 15, 2026 09:12 UTC   <invalid>       ca                      no
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jul 13, 2035 09:12 UTC   9y              no

<invalid> in the residual column is your answer. Notice the certificate authority (the ca that signs all the others) is still valid for years, so you don't rebuild trust from scratch. You renew the leaf certificates, bounce the pods that use them, and then fix your own kubeconfig. Static pods don't pick up new certs on their own, so you nudge the kubelet by moving the manifests out of the folder and back in, which forces it to recreate the pods. The last step is the one that catches almost everyone: kubeadm certs renew all writes a fresh /etc/kubernetes/admin.conf, but your ~/.kube/config is a year-old copy of that file with the expired client certificate still baked into it. Copy the new one over it, or kubectl keeps failing after a recovery that actually worked and you go hunting for a second fault that isn't there.

renew certs, bounce the static pods, refresh your kubeconfig

sudo kubeadm certs renew all
 
# static pods won't reload certs by themselves; make the kubelet recreate them:
sudo mv /etc/kubernetes/manifests/*.yaml /tmp/ && sleep 20
sudo mv /tmp/*.yaml /etc/kubernetes/manifests/
 
# your kubeconfig still holds the OLD client cert; replace it with the new admin.conf:
sudo cp /etc/kubernetes/admin.conf $HOME/.kube/config
sudo chown $(id -u):$(id -g) $HOME/.kube/config

expected output

certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller-manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
 
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager, kube-scheduler and etcd, so that they can use the new certificates.

Second, a broken static-pod manifest. If someone hand-edited /etc/kubernetes/manifests/kube-apiserver.yaml, maybe to add a flag or an audit-log path, one typo takes the API server with it. When the file is still valid YAML (the indentation-based text format these config files use) but the flag is wrong, the container crash-loops (that's the Exited you saw earlier), and sudo crictl logs <apiserver-container-id> prints the exact flag it choked on. Fix the file, save it, and the kubelet rebuilds the pod within seconds. No restart command needed. The file on disk is the source of truth.

Third, etcd. etcd is the cluster's key-value database, the one and only record of every object in the cluster: every pod, secret, and config lives there. The API server keeps no state of its own. It reads and writes everything to etcd. So if etcd is unreachable or its data is corrupt, the API server starts, fails to reach its datastore, and falls over immediately. Check etcd's health from the node, with one catch: etcdctl is not installed on the host in a kubeadm cluster. It ships inside the etcd container image, so you reach it through crictl exec (a context deadline exceeded in the output means etcd isn't answering). And know your quorum, which is just the majority of members that have to agree before etcd will record a change. A three-member cluster survives losing one member. Lose two and etcd deliberately stops accepting writes, so it never splits into two disagreeing copies of the truth, and that alone drags the whole control plane down with it. When the data itself is gone, not just unreachable, this is the moment the etcd snapshot you've been taking becomes the restore path. If you haven't been taking one, this is where you learn why everyone says to.

check etcd health, from inside the etcd container

# etcdctl isn't on the host in a kubeadm cluster; it lives in the etcd image.
# If this comes back empty, etcd itself is down: sudo crictl ps -a --name etcd
ETCD_ID=$(sudo crictl ps --name etcd -q)
 
sudo crictl exec -it "$ETCD_ID" etcdctl \
  --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  endpoint health

expected output

https://127.0.0.1:2379 is healthy: successfully committed proposal: took = 7.8ms

A typo in a static-pod manifest can make the container vanish, not crash

There are two ways a bad edit to /etc/kubernetes/manifests/kube-apiserver.yaml fails, and they look nothing alike. A valid file with a wrong flag crash-loops, so crictl ps -a shows an Exited kube-apiserver with a climbing attempt count. But a file the kubelet can't even parse (broken indentation, a stray character) gets silently skipped, so crictl ps -a shows no kube-apiserver container at all. People go hunting for a crashing container, find nothing, and start suspecting the network or etcd. When a control-plane container is simply missing after an edit, run journalctl -u kubelet, look for a manifest parse error, and re-check your YAML indentation.

Control-plane triage, from the node

kubectl: connection refused

SSH to the control-plane node and work below the API

is it up?

kubelet active + crictl ps -a

find which static pod is Exited or missing

cause 1

certs expired?

kubeadm certs check-expiration → renew all, bounce the pods

cause 2

manifest broken?

crictl logs apiserver → fix the file in /etc/kubernetes/manifests

cause 3

etcd unhealthy?

etcdctl endpoint health → restore from snapshot if data is lost

When kubectl dies, stop using it. Go to the node, check the kubelet and static pods with crictl, then work certs, then the manifest, then etcd, in that order of likelihood.

A dead kubectl is a symptom, not a diagnosis, and re-running it adds no data. Read which failure you got: connection refused comes back instantly and means nothing is listening, while a few seconds of silence ending in i/o timeout points at the network, a firewall, or the wrong endpoint.

Certificate expiry presents as auth failure across components. Check dates early.

etcd quorum loss needs member status, not random API restarts.

Try this

Do this on a lab cluster, never one anyone depends on. You are going to break the API server on purpose and then diagnose it with kubectl unavailable, which is the only honest way to find out whether the offline path is in your fingers yet. Renaming a flag keeps the file valid YAML, so you get the crash-loop version of the failure rather than the vanishing-container version.

on a lab node: break it, then find it without kubectl

# LAB ONLY. Keep a backup outside the manifests folder.
sudo cp /etc/kubernetes/manifests/kube-apiserver.yaml ~/kube-apiserver.yaml.bak
 
# Valid YAML, bogus flag name: the API server will crash-loop.
sudo sed -i 's/--etcd-servers=/--etcd-serverz=/' /etc/kubernetes/manifests/kube-apiserver.yaml
 
# kubectl is dead now. Work below the API:
sudo crictl ps -a --name kube-apiserver
sudo crictl logs "$(sudo crictl ps -a --name kube-apiserver -q | head -1)"
 
# Put the file back; the kubelet rebuilds the pod on its own within seconds.
sudo cp ~/kube-apiserver.yaml.bak /etc/kubernetes/manifests/kube-apiserver.yaml

expected output: the container tells you exactly what you broke

CONTAINER      IMAGE          CREATED         STATE     NAME             ATTEMPT   POD ID         POD
9a8b7c6d5e4f   c3994bc69610   6 seconds ago   Exited    kube-apiserver   3         f00dcafe1234   kube-apiserver-cp-1
 
Error: unknown flag: --etcd-serverz
Usage:
  kube-apiserver [flags]

Takeaway

Control-plane failure means work on the box. Kubelet and crictl first to see which static pod is down, then certificates, then the manifest, then etcd, in that order of likelihood.

Application failureWorker-node failure

What Kubernetes isKubernetes fundamentals Security domains & threat modelKubernetes security & hardening The cluster attack surfaceKubernetes attack & defense Kubernetes interview questionsInterview prep

Control-plane failure

The one thing that stays up

Certs, then the manifest, then etcd

Try this

Takeaway

Related