>_SecOpsLog_
CoursesKubernetes administrationMonitoring cluster components

Monitoring cluster components

metrics-server, kubectl top, and what to watch.

Intermediate10 min · lesson 52 of 65
In plain terms
metrics-server is the live speedometer — how fast are we going right now. Prometheus is the trip recorder that keeps the whole history and sounds the alarm when something’s off.

Run kubectl top pods on a brand-new cluster and there's a decent chance you get back error: Metrics API not available instead of a table of numbers. Nothing's broken. Kubernetes just doesn't measure live resource usage on its own. The control plane knows what you asked for, the CPU and memory a Pod requested (a Pod being the smallest thing you deploy, one or more containers sharing a network address). It has no idea what that Pod is actually burning right now. To see that, you install a small add-on called metrics-server. It's the speedometer bolted to the dashboard. It tells you how fast you're going this instant and nothing about the trip so far.

Where the live numbers come from

Every worker node already runs a kubelet, the agent that launches your containers and reports their health up to the control plane. Baked into that kubelet is a stats collector called cAdvisor that watches the Linux cgroups (control groups, the kernel feature that boxes in and counts what each container uses) and reads their real CPU and memory counters straight from the kernel. metrics-server does one narrow job. Every 15 seconds it calls each node's kubelet, pulls the newest per-Pod and per-node numbers, and keeps them in memory. Only the latest sample. It writes nothing to disk and runs no database, so it remembers nothing about a minute ago. That's deliberate, and it's why metrics-server stays tiny even on a cluster with thousands of Pods.

Reading those numbers is the clever bit. metrics-server registers itself with the API server as an extension, an APIService named v1beta1.metrics.k8s.io. After that, kubectl top and the autoscaler just ask the normal API server for metrics.k8s.io, and the API server quietly forwards the request on to metrics-server and hands you the answer. Same front door as every other resource, so the same RBAC (Role-Based Access Control, Kubernetes' permission system) decides who's allowed to read it.

terminal
kubectl top nodes
kubectl top pods -n payments --sort-by=memory
output
NAME     CPU(cores)   CPU%   MEMORY(bytes)   MEMORY%
node-1   243m         12%    1187Mi          31%
node-2   611m         30%    2934Mi          76%
 
NAME                        CPU(cores)   MEMORY(bytes)
payments-api-7d4b9c-x2fjq   118m         512Mi
payments-api-7d4b9c-h9lpm   74m          486Mi

Check it's actually wired in

When top comes back empty, don't start by reading metrics-server's logs. Start by asking whether the extension is registered and healthy. The API server tracks every aggregated API in an APIService object with an AVAILABLE column. If that says True, the plumbing is fine and the problem is your query. If it says False, the API server can't reach metrics-server, and the reason is printed right there in the object.

terminal
kubectl get apiservice v1beta1.metrics.k8s.io
output
NAME                     SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io   kube-system/metrics-server   True        9d

The autoscaler drinks from the same tap

A HorizontalPodAutoscaler (HPA, the controller that adds or removes Pod copies as load changes) doesn't measure anything itself. It reads metrics.k8s.io, the very API metrics-server serves. So two problems share one root. If kubectl top is blank, your HPA's TARGETS column reads <unknown> and it won't scale on CPU or memory, because it's a thermostat wired to a dead sensor. It wants to react, it just can't see the temperature. Fix metrics-server and both come back to life at once.

terminal
kubectl get hpa payments-api
output
NAME           REFERENCE                 TARGETS        MINPODS   MAXPODS   REPLICAS
payments-api   Deployment/payments-api   cpu: 42%/70%   2         10        3
How a number reaches kubectl top
1kubelet + cAdvisorreads cgroup CPU/mem per…2metrics-serverscrapes every node, holds…3metrics.k8s.io APIserved through the API server4kubectl top + HPAlive view and autoscaling
There's no database anywhere on this path. metrics-server keeps only the most recent sample, which is exactly why it can't tell you what happened an hour ago.
metrics-server is Running but top is still empty
This one bites almost everyone on kubeadm and self-managed clusters. metrics-server talks to each kubelet over HTTPS on port 10250 and checks the kubelet's serving certificate. By default that cert is self-signed and isn't issued by a certificate authority metrics-server trusts, so every scrape dies with x509: certificate signed by unknown authority while the pod itself sits there looking perfectly Running. In a lab, add --kubelet-insecure-tls to the metrics-server container args. In production, turn on signed kubelet serving certificates instead of switching the check off.

Past 'right now': keeping history

metrics-server answers 'what's hot this second' and forgets it immediately. It can't tell you what the cluster was doing at 3am when the pager went off, because it stored nothing. For history, dashboards, and alerts you add a real monitoring stack. The usual one is Prometheus (it scrapes metrics on a schedule and stores them as time-series data), Grafana (the dashboards), and Alertmanager (it routes alerts to your pager or Slack). Most people install the three together with the kube-prometheus-stack Helm chart. Helm is Kubernetes' package installer, so a single command lays down the whole stack. The chart also drops in kube-state-metrics, a component that turns cluster objects into numbers you can graph. What's worth watching once you have that history? Node and Pod usage against their requests and limits, so you can spot a node about to run out of memory and start evicting Pods. Restart counts and CrashLoopBackOff, which is a container that keeps failing and getting restarted. Pods stuck Pending, meaning the scheduler can't find a node to place them on. And the control plane itself, mainly API server request latency and the health of etcd, the key-value database holding all cluster state.

You tell Prometheus what to scrape with a ServiceMonitor, a small custom resource the Prometheus Operator watches for. Point it at your app's Service by label, name the port that exposes /metrics, and the operator regenerates Prometheus' scrape config for you. You never hand-edit a config file, and a new app starts getting scraped the moment its ServiceMonitor lands.

servicemonitor.yaml
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  name: payments-api
  namespace: monitoring
  labels:
    release: prometheus
spec:
  selector:
    matchLabels:
      app: payments-api
  endpoints:
    - port: metrics
      interval: 30s
terminal
kubectl apply -f servicemonitor.yaml
kubectl get servicemonitor -n monitoring
output
servicemonitor.monitoring.coreos.com/payments-api created
 
NAME                              AGE
kube-prometheus-stack-apiserver   9d
payments-api                      6s

top is not long-term capacity history. Pair it with a real metrics stack for trends.

Control-plane monitoring on managed services comes from the vendor. Still watch worker resource pressure yourself.

Missing metrics-server makes HPA sit idle. Verify the pipeline before you tune algorithms.

Try this

If metrics-server is present, run kubectl top nodes and top pods. Compare a hot pod to its requests and decide whether you need more replicas or bigger requests.

terminal
$ kubectl top nodes
$ kubectl top pods -n payments --sort-by=memory
 
$ kubectl get apiservice v1beta1.metrics.k8s.io
 
NAME                     SERVICE                      AVAILABLE   AGE
v1beta1.metrics.k8s.io   kube-system/metrics-server   True        9d
 
$ kubectl get hpa payments-api
 
$ kubectl apply -f servicemonitor.yaml
$ kubectl get servicemonitor -n monitoring
 
servicemonitor.monitoring.coreos.com/payments-api created
 
NAME                              AGE
kube-prometheus-stack-apiserver   9d
payments-api                      6s

Takeaway

metrics-server feeds kubectl top and HPA. Component health still needs separate probes for API, etcd, and core DNS.

Quick check
01kubectl top pods returns 'Metrics API not available', yet kubectl -n kube-system get pods shows the metrics-server pod Running and 1/1 Ready. What's the most likely cause?
Correct — Classic on kubeadm and self-managed clusters: the pod runs fine, but each kubelet scrape dies with an x509 error. Fix the kubelet serving certs, or add --kubelet-insecure-tls in a lab.
Incorrect — No. Prometheus is a separate stack. kubectl top reads the metrics.k8s.io API from metrics-server and doesn't touch Prometheus at all.
Incorrect — No. --sort-by only orders results that already exist; it can't conjure the missing Metrics API.
Incorrect — No. The HPA reads the same aggregated API; it doesn't drain or lock it, and both would work together if metrics-server were healthy.
02In a post-incident review you need a pod's memory usage from 3am last night. Why can't metrics-server give it to you, and what can?
Correct — metrics-server is a live speedometer that forgets each sample immediately, while Prometheus is the recorder that keeps the time series.
Incorrect — metrics-server has no retention setting because it stores no history at all.
Incorrect — the sample was never stored, so no query can retrieve what metrics-server already discarded.
Incorrect — it holds just the most recent sample at any moment, reboot or not.
03kubectl top pods is blank cluster-wide, and separately your HorizontalPodAutoscaler shows TARGETS: <unknown> and refuses to scale. How are the two connected?
Incorrect — the HPA doesn't measure anything itself, it reads the same API metrics-server serves.
Incorrect — the HPA reads the aggregated API without locking or draining it; both would work if metrics-server were healthy.
Incorrect — <unknown> is a missing-metrics symptom, and the blank kubectl top points at the same source.
Correct — the HPA is a thermostat wired to the same sensor as kubectl top, so restoring metrics-server recovers both together.
Network policiesApplication logs

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep