You run kubeadm certs renew all and it reports success on every certificate, but kubectl still fails with "x509: certificate has expired or is not yet valid". What's the most likely reason?

The control-plane components are still using the certificates they loaded at startup, so you must restart the static Pods. Renewal rewrites the files but doesn't restart anything. Cycle the manifests (or reboot) so the kubelet recreates the static Pods with the fresh certs. The renew output tells you to do exactly this on its last line.

A kubeadm cluster is built and nobody touches it for just over a year, then the whole control plane stops answering kubectl. Why does this tend to strike around the first birthday?

kubeadm signs the leaf certificates with a one-year lifetime by default, while the CA is good for ten years, so the leaves expire first. The gap between one-year leaf certs and the ten-year CA is exactly why untended clusters die at the one-year mark.

On one node, kubectl logs and kubectl exec fail with x509 errors while the API server and every other node stay healthy. The cluster has serverTLSBootstrap enabled. What's the most likely cause?

The kubelet's serving-certificate CSR is sitting Pending and needs approval. With serverTLSBootstrap on, nothing auto-approves the serving CSR, so it stays Pending and logs/exec against that node fail until you approve it.

Courses Kubernetes administrationTLS & certificates

TLS & certificates

The PKI every component authenticates with.

Advanced12 min · lesson 44 of 65

In plain terms

TLS certificates are tamper-proof ID badges every component carries, all signed by the same trusted office. Let a badge expire and that component gets turned away at every single door.

One morning kubectl just stops. (kubectl is the command-line tool you use to talk to the cluster.) Every command comes back with the same line: Unable to connect to the server: x509: certificate has expired or is not yet valid. Nobody deployed anything. Nobody touched the network. Your Pods, the little bundles of containers that actually run your app, are still up, and your users are still being served, but the cluster's brain has gone quiet. Nine times out of ten the cause is one certificate that hit its expiry date, and the clock that killed it started ticking the day someone built the cluster. That x509 is just the name of the certificate format, so the error is really saying something simple. A badge went out of date.

A secure office building runs on badges. Every door has a guard, and the guards don't know anyone by face. They trust the badge, because it was printed and signed by one badging office in the basement, and they've been told to trust whatever that office signs. Kubernetes plays the same game. Its components don't trust the network they sit on, so they never take each other's word for anything. Every component carries a certificate, which is just a badge signed by one trusted office, and every connection between them is checked with TLS (Transport Layer Security, the same handshake that puts the little padlock in your browser). The badging office is the cluster CA, the Certificate Authority.

The cluster runs its own badging office

A Kubernetes cluster ships with its own PKI (Public Key Infrastructure), a fancy name for the machinery that issues badges and checks them. At the top sits the cluster CA: one certificate and one private key. The API server (the cluster's single front door), the kubelet (the agent that runs on every node and does what the control plane tells it), etcd (the key-value database that holds all cluster state), the scheduler, the controllers, and the admin kubeconfig on your laptop (the credentials file kubectl reads) all carry certificates the CA signed. When the kubelet phones the API server, both sides show a badge and both sides check it. That two-way check is mutual TLS, and it's why a random process on the network can't start barking orders at your control plane. On a kubeadm-built cluster (kubeadm is the tool most people use to stand a cluster up) the badges live in one directory, and the whole web of trust hangs off two files, ca.crt and ca.key.

the badges on a control-plane node

$ ls /etc/kubernetes/pki/
apiserver.crt                 ca.crt                  front-proxy-ca.crt
apiserver.key                 ca.key                  front-proxy-ca.key
apiserver-etcd-client.crt     etcd                    front-proxy-client.crt
apiserver-etcd-client.key     sa.key                  front-proxy-client.key
apiserver-kubelet-client.crt  sa.pub
apiserver-kubelet-client.key
 
$ sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -enddate
subject=CN = kube-apiserver
notAfter=Jun 15 09:12:04 2026 GMT

Guard ca.key like the master key it is. Anyone who holds that file can print a badge for any identity in the cluster, including cluster-admin, and every guard will wave it through every door. It stays on the control-plane node, readable only by root. It never goes into a backup that leaves the box, and it never lands in a git repo. etcd runs its own separate CA inside that etcd/ folder, so if the main CA leaks, the attacker still doesn't automatically hold the keys to the database.

Know the expiry before it knows you

kubeadm signs the leaf certificates (the API server's, the kubelet client, the etcd client, and the rest) with a one-year lifetime by default. The CA itself gets ten years. That gap is exactly why forgotten clusters die on their first birthday. The office keeps signing badges for a decade, but the badges it already handed out go stale in twelve months. Before you plan a renewal, check the real dates. kubeadm has a subcommand that reads every cert on the node and prints how much time each one has left.

check every certificate at once

$ sudo kubeadm certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 15, 2026 09:12 UTC   9d              ca                      no
apiserver                  Jun 15, 2026 09:12 UTC   9d              ca                      no
apiserver-etcd-client      Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
apiserver-kubelet-client   Jun 15, 2026 09:12 UTC   9d              ca                      no
controller-manager.conf    Jun 15, 2026 09:12 UTC   9d              ca                      no
etcd-healthcheck-client    Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
etcd-peer                  Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
etcd-server                Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
front-proxy-client         Jun 15, 2026 09:12 UTC   9d              front-proxy-ca          no
scheduler.conf             Jun 15, 2026 09:12 UTC   9d              ca                      no
super-admin.conf           Jun 15, 2026 09:12 UTC   9d              ca                      no
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 13, 2035 09:12 UTC   9y              no
etcd-ca                 Jun 13, 2035 09:12 UTC   9y              no
front-proxy-ca          Jun 13, 2035 09:12 UTC   9y              no

Nine days left in that output means you're about to have a very bad Monday. Notice the kubelet's own client certificate isn't in the list. Modern kubelets rotate that client badge on their own: as it nears expiry, the kubelet asks the API server for a fresh one and swaps it in with nobody watching. kubeadm keeps that cert under /var/lib/kubelet/pki, which is why check-expiration doesn't track it. That auto-rotation is the one part of the setup that mostly looks after itself. The control-plane certs above do not, and that's where you come in.

Renewing, and the restart everyone forgets

Renewal is one command. kubeadm re-signs every leaf certificate with the CA and writes the new files to disk. Here's the trap. Writing a new file on disk does nothing for a program that's already running. Think of a guard who memorized the badge design at the start of the shift. You can post a new design on the wall, but that guard keeps checking against the one in their head until they clock out and clock back in. The control-plane components read their certificates once, when they start. etcd, the scheduler, and the controller-manager won't look at the files again until you restart them. So the renew succeeds, kubectl still fails, and the fix looks broken until this one detail clicks.

renew, then prove the running process is still stale

$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
 
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager,
kube-scheduler and etcd, so that they can use the new certificates.
 
$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate
notAfter=Jun 15 14:31:20 2027 GMT                       # the file on disk is fresh
$ echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -enddate
notAfter=Jun 15 09:12:04 2026 GMT                       # but the wire still shows the old cert

recycle the static pods, then re-verify

# static Pods only restart when their manifest file changes, so move the manifests out and back:
$ sudo mv /etc/kubernetes/manifests/*.yaml /tmp/  &&  sleep 20  &&  sudo mv /tmp/*.yaml /etc/kubernetes/manifests/
 
$ echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -enddate
notAfter=Jun 15 14:31:20 2027 GMT                       # served cert finally matches the file

On the managed platforms (the hosted Kubernetes services from Amazon, Google, and Microsoft: EKS, GKE, AKS) you never see any of this. The provider owns the control plane and rotates its certificates behind the curtain, which is one of the honest reasons teams pay for managed Kubernetes. On a cluster you run yourself, put certificate expiry on a calendar, or script the renew-and-restart on a schedule, and treat kubeadm certs check-expiration as boring routine maintenance instead of something you reach for after the outage has already started.

The badges kubectl can see: the CSR API

Most of the PKI is just files on a node. One piece you drive with kubectl. When a brand-new node joins, its kubelet has no client badge yet, so it can't authenticate to ask for anything. It presents a short-lived bootstrap token instead and submits a CertificateSigningRequest (a CSR, literally a request to the badging office for a badge). kubeadm sets things up so those kubelet client requests get approved automatically, and you rarely catch one. The kubelet's serving certificate is a different story. That's the badge a node shows so the API server trusts it for things like kubectl logs, kubectl exec, and metrics scraping. If your cluster turns on serving-certificate bootstrapping (the serverTLSBootstrap setting, which isn't on by default), the kubelet asks for that serving badge too, under its own node identity, and nothing approves it for you. It sits there Pending. On clusters that verify those serving certs, logs and exec against that node fail with x509 errors that look like a networking problem and aren't.

approve a pending kubelet serving CSR

$ kubectl get csr
NAME        AGE    SIGNERNAME                      REQUESTOR            REQUESTEDDURATION   CONDITION
csr-8n2kd   109s   kubernetes.io/kubelet-serving   system:node:node-2   <none>              Pending
 
$ kubectl certificate approve csr-8n2kd
certificatesigningrequest.certificates.k8s.io/csr-8n2kd approved
 
$ kubectl get csr csr-8n2kd
NAME        AGE   SIGNERNAME                      REQUESTOR            REQUESTEDDURATION   CONDITION
csr-8n2kd   2m    kubernetes.io/kubelet-serving   system:node:node-2   <none>              Approved,Issued

Renewing certs does not restart anything

kubeadm certs renew all rewrites the files on disk. It does not restart a single process. etcd, the controller-manager, the scheduler, and the kube-apiserver keep running on the certificates they loaded at startup, so until you restart them the cluster is still working off the old, expiring badges and kubectl keeps failing. The renew even prints this in its last line, and people skip past it every time. Cycle the manifests in /etc/kubernetes/manifests out and back (or reboot the node) so the kubelet recreates the static Pods, then confirm with openssl s_client that the served cert actually changed.

Reading an x509 failure

A component or kubectl fails with an x509 TLS error

read the exact error text first, then branch on it

expired

Control-plane cert lapsed

"certificate has expired": run kubeadm certs renew all, then restart the static Pods

unknown authority

Wrong CA in the kubeconfig

"signed by unknown authority" means your client trusts a CA that didn't sign this server, so refresh admin.conf

valid but 403

TLS is fine, authorization denies

the handshake worked, so this is RBAC (Role-Based Access Control); check your roles and bindings, not certs

logs/exec fail

kubelet serving cert

the node badge is unapproved or stale; check kubectl get csr for Pending rows

The exact error text tells you which certificate to look at. Expiry means renew and restart. Unknown authority means a CA mismatch on the client side. A clean handshake that still returns 403 isn't a TLS problem at all.

Front-proxy certs matter for aggregation. Missing them breaks metrics and extension APIs in surprising ways.

Rotate before expiry, not after. Calendar the ninety-day marks if your install uses short lifetimes.

Never copy pki directories into tickets or chat. Treat them like production passwords.

Try this

On a kubeadm node, list /etc/kubernetes/pki and check certificate expiry with openssl. Correlate apiserver and etcd cert names.

terminal

$ ls /etc/kubernetes/pki/
apiserver.crt                 ca.crt                  front-proxy-ca.crt
apiserver.key                 ca.key                  front-proxy-ca.key
apiserver-etcd-client.crt     etcd                    front-proxy-client.crt
apiserver-etcd-client.key     sa.key                  front-proxy-client.key
apiserver-kubelet-client.crt  sa.pub
apiserver-kubelet-client.key
 
$ sudo openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -subject -enddate
subject=CN = kube-apiserver
notAfter=Jun 15 09:12:04 2026 GMT
 
$ sudo kubeadm certs check-expiration
CERTIFICATE                EXPIRES                  RESIDUAL TIME   CERTIFICATE AUTHORITY   EXTERNALLY MANAGED
admin.conf                 Jun 15, 2026 09:12 UTC   9d              ca                      no
apiserver                  Jun 15, 2026 09:12 UTC   9d              ca                      no
apiserver-etcd-client      Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
apiserver-kubelet-client   Jun 15, 2026 09:12 UTC   9d              ca                      no
controller-manager.conf    Jun 15, 2026 09:12 UTC   9d              ca                      no
etcd-healthcheck-client    Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
etcd-peer                  Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
etcd-server                Jun 15, 2026 09:12 UTC   9d              etcd-ca                 no
front-proxy-client         Jun 15, 2026 09:12 UTC   9d              front-proxy-ca          no
scheduler.conf             Jun 15, 2026 09:12 UTC   9d              ca                      no
super-admin.conf           Jun 15, 2026 09:12 UTC   9d              ca                      no
 
CERTIFICATE AUTHORITY   EXPIRES                  RESIDUAL TIME   EXTERNALLY MANAGED
ca                      Jun 13, 2035 09:12 UTC   9y              no
etcd-ca                 Jun 13, 2035 09:12 UTC   9y              no
front-proxy-ca          Jun 13, 2035 09:12 UTC   9y              no
 
$ sudo kubeadm certs renew all
[renew] Reading configuration from the cluster...
[renew] FYI: You can look at this config file with 'kubectl -n kube-system get cm kubeadm-config -o yaml'
 
certificate embedded in the kubeconfig file for the admin to use and for kubeadm itself renewed
certificate for serving the Kubernetes API renewed
certificate the apiserver uses to access etcd renewed
certificate for the API server to connect to kubelet renewed
certificate embedded in the kubeconfig file for the controller manager to use renewed
certificate for liveness probes to healthcheck etcd renewed
certificate for etcd nodes to communicate with each other renewed
certificate for serving etcd renewed
certificate for the front proxy client renewed
certificate embedded in the kubeconfig file for the scheduler manager to use renewed
 
Done renewing certificates. You must restart the kube-apiserver, kube-controller-manager,
kube-scheduler and etcd, so that they can use the new certificates.
 
$ openssl x509 -in /etc/kubernetes/pki/apiserver.crt -noout -enddate
notAfter=Jun 15 14:31:20 2027 GMT                       # the file on disk is fresh
$ echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -enddate
notAfter=Jun 15 09:12:04 2026 GMT                       # but the wire still shows the old cert
 
# static Pods only restart when their manifest file changes, so move the manifests out and back:
$ sudo mv /etc/kubernetes/manifests/*.yaml /tmp/  &&  sleep 20  &&  sudo mv /tmp/*.yaml /etc/kubernetes/manifests/
 
$ echo | openssl s_client -connect 127.0.0.1:6443 2>/dev/null | openssl x509 -noout -enddate
notAfter=Jun 15 14:31:20 2027 GMT                       # served cert finally matches the file
 
$ kubectl get csr
NAME        AGE    SIGNERNAME                      REQUESTOR            REQUESTEDDURATION   CONDITION
csr-8n2kd   109s   kubernetes.io/kubelet-serving   system:node:node-2   <none>              Pending
 
$ kubectl certificate approve csr-8n2kd
certificatesigningrequest.certificates.k8s.io/csr-8n2kd approved
 
$ kubectl get csr csr-8n2kd
NAME        AGE   SIGNERNAME                      REQUESTOR            REQUESTEDDURATION   CONDITION
csr-8n2kd   2m    kubernetes.io/kubelet-serving   system:node:node-2   <none>              Approved,Issued

Takeaway

Cluster PKI binds components together. Expired certs look like total outages. Know where files live and how kubeadm renews them.

AuthenticationKubeConfig

What Kubernetes isKubernetes fundamentals Security domains & threat modelKubernetes security & hardening The cluster attack surfaceKubernetes attack & defense Kubernetes interview questionsInterview prep