>_SecOpsLog_
CoursesKubernetes administrationA troubleshooting method

A troubleshooting method

Working outside-in, and stopping at the first no.

Intermediate8 min · lesson 61 of 65
In plain terms
Troubleshoot like a plumber tracing a leak from the street inward: is the water even on (the node), does it reach the house (the control plane), the right room (was the pod scheduled), the tap itself (the app)? Stop at the first “no.”

A pod in your prod namespace just flipped to CrashLoopBackOff, and the Slack channel is filling up. The instinct is to start changing things: restart it, redeploy, bump the memory limit. Resist that. Change something at random and you might get lucky, the pod goes green, and now you've buried the real break instead of fixing it. Next week it comes back and nobody remembers what you touched. Troubleshooting a cluster is more like tracing a water leak from the street back into the house. You don't rip open a wall first. You walk the pipe. Is the water even on, does it reach the building, then the right room, then the tap. You stop at the first place it runs dry, because that's where the break is. Every dry spot past that point is just a symptom of the first one.

Follow the pipe, stop at the first dry spot

Kubernetes hands you four commands, and they line up with those four checks. You run them with kubectl, the command-line tool that talks to the cluster's single front desk, the API server (Application Programming Interface server, the one door in and out of the cluster). First, kubectl get: is the object even there, and what state is it in? A Pod, the smallest thing Kubernetes runs, usually a single container, reports a status like Running, Pending, or CrashLoopBackOff. Second, kubectl describe: why is it in that state? Every object keeps a running diary at the bottom of describe called Events, and the answer is usually sitting right there. Third, kubectl logs: what did the app itself say before it fell over? Fourth, kubectl exec: open a shell inside the running container and test by hand, the way you'd walk into the room and turn the tap yourself. From in there you can check whether a config file is really where the app expects it, or curl a database the app swears it can't reach. You go in that order, and the moment one command says no, you stop and fix that. Don't read logs for a pod that never started.

1. get: what state is it in?
kubectl get pods -n prod
output
NAME                    READY   STATUS             RESTARTS        AGE
web-7d9f8c6b4-xl2kq     0/1     CrashLoopBackOff   6 (2m14s ago)   11m
web-7d9f8c6b4-p8m4t     1/1     Running            0               11m
api-5c8b0d9f77-k2wsz    0/1     Pending            0               4m

get tells you the state, not the reason. The web pod is crash-looping and the api pod is stuck Pending, but neither line says why. The READY column does say a little: 0/1 means the container isn't ready, either not running at all or up but failing its readiness check, and 1/1 means it's passing. That still won't tell you the cause, though, and guessing it from the status name is how you lose an afternoon. For the real answer, describe the object and read the Events at the bottom.

2. describe: why? (read the Events)
kubectl describe pod web-7d9f8c6b4-xl2kq -n prod
output (Events section only)
Events:
  Type     Reason     Age                 From     Message
  ----     ------     ----                ----     -------
  Normal   Pulled     11m                 kubelet  Successfully pulled image "web:1.8.2" in 812ms (812ms including waiting). Image size: 24917331 bytes.
  Normal   Created    11m (x7 over 11m)   kubelet  Created container web
  Normal   Started    11m (x7 over 11m)   kubelet  Started container web
  Warning  BackOff    92s (x24 over 11m)  kubelet  Back-off restarting failed container web in pod web-7d9f8c6b4-xl2kq_prod

Look at what describe does and doesn't tell you. It says the kubelet, the agent running on every node that starts and watches containers, keeps restarting the container after it fails. That's the symptom, not the cause. The container that failed is already gone, replaced by a fresh one, so its error isn't in the current logs. That's what --previous is for. Show me the logs of the container that died, not the one that just booted.

3. logs --previous: what did the app say?
kubectl logs web-7d9f8c6b4-xl2kq -n prod --previous
output
2026-07-16T09:14:02.118Z  INFO   starting web v1.8.2
2026-07-16T09:14:02.140Z  FATAL  config: environment variable DATABASE_URL is not set
panic: missing required configuration
 
goroutine 1 [running]:
main.mustEnv(...)

There's the first no. The app never got its DATABASE_URL, so it panics the instant it boots. That's a config problem, not a networking problem and not a Kubernetes bug. The fix lives in that missing environment variable, usually a bad or absent reference to a ConfigMap or a Secret (the small objects that hold config values and passwords for the pod to read). Set it correctly and the crash loop ends. You never had to touch the network, the node, or the control plane (the cluster's brain, the components that decide what runs where).

The symptom already names the layer

After a while you stop walking every hop, because the state alone points at the layer. Pending means the pod was accepted but never placed on a node. The scheduler, the component that works like a host seating diners, couldn't find a table that fits, so the pod waits. describe or the events give the real reason: not enough CPU, a taint the pod doesn't tolerate (a node fenced off unless the pod carries a matching pass), or a storage volume that won't bind. CrashLoopBackOff means it started and then died, again and again, which is almost always the app's own fault, so go straight to logs --previous. Running but unreachable is a Service problem before it's ever a network one. Check whether the Service has endpoints first. Empty endpoints mean no ready pod matches its selector, not a broken router. And weirdness across the whole cluster, everything timing out at once, usually points at DNS (Domain Name System, the cluster's phone book) or the control plane, not any single pod.

the timeline when several things fail at once
kubectl get events -n prod --sort-by=.lastTimestamp
output
LAST SEEN   TYPE      REASON             OBJECT                       MESSAGE
5m          Warning   FailedScheduling   pod/api-5c8b0d9f77-k2wsz     0/3 nodes are available: 3 Insufficient cpu. preemption: 0/3 nodes are available: 3 No preemption victims found.
2m          Warning   BackOff            pod/web-7d9f8c6b4-xl2kq      Back-off restarting failed container web in pod web-7d9f8c6b4-xl2kq_prod

Sorting events by time gives you the whole cluster's story on one screen. It's how you spot that the Pending api pod is really a capacity problem: three nodes, none of them with a spare core to give. Same method, wider lens.

One last place the pipe can run dry is the node itself. If a pod looks fine in describe and its logs are clean but it still won't come up, check the machine it landed on with kubectl get nodes. A node marked NotReady, or one flagged with disk or memory pressure, will quietly wreck every pod on it while each pod looks blameless on its own. Same walk, one hop further out toward the street.

Read the symptom, pick the layer
A workload is broken. Read the state first.
the state names the layer, so stop at the first no
Pending
Never scheduled
describe then FailedScheduling: no CPU, an untolerated taint, or an unbound volume
CrashLoopBackOff
App's own fault
kubectl logs --previous, read what it said before it died
Running, unreachable
Service, not network
get endpoints first: empty means selector or readiness, not kube-proxy
Whole cluster flaky
DNS or control plane
not one pod; check CoreDNS and the API server
describe tells you it is restarting, not why it is restarting
On a crash-looping pod, describe shows a BackOff event and plain kubectl logs shows the container that just started, which is often empty or only the first couple of boot lines before it dies again. Both can fool you into thinking nothing is wrong. The real error lives in the container that already exited, so reach for kubectl logs POD --previous. If a pod has restarted several times and its logs look suspiciously clean, you're almost certainly reading the wrong container. Add --previous and the actual stack trace shows up.

Write the hypothesis before you change anything. Blind deletes destroy evidence.

Time-box each layer. If DNS is unclear after ten minutes, you are probably not in a DNS failure.

Capture commands and outputs in the incident doc as you go. Memory lies under stress. Blind deletes destroy evidence.

Try this

Pick a broken lab app and force yourself to work outside-in: Service, endpoints, pod, events, node. Stop at the first failing layer and fix only that before moving on.

terminal
$ kubectl get pods -n prod
 
$ kubectl describe pod web-7d9f8c6b4-xl2kq -n prod
 
$ kubectl logs web-7d9f8c6b4-xl2kq -n prod --previous
 
$ kubectl get events -n prod --sort-by=.lastTimestamp

Takeaway

Troubleshooting is a funnel. Outside-in, one no at a time. Skipping layers is how you restart healthy pods for an hour.

Quick check
01A pod in prod has been Pending for ten minutes. You keep running kubectl logs on it and get nothing useful back. What's the right next move?
Incorrect — No. A Pending pod has no running container yet, so there are no logs to produce. Logs is the wrong hop entirely; you're waiting on output that can't exist until the pod is scheduled and started.
Correct — Pending means the scheduler never placed the pod. describe (or kubectl get events) shows exactly why: insufficient CPU, an untolerated taint, or an unbound volume. That's the first no, and it's the thing to fix.
Incorrect — No. The kubelet only runs pods the scheduler has already assigned to its node. This pod was never assigned, so restarting kubelets changes nothing and risks disrupting healthy pods.
Incorrect — No. The replacement lands in the same cluster with the same constraints and goes Pending too. You'd be recreating the symptom without ever reading why. Describe first.
02A pod has been CrashLoopBackOff with 8 restarts. You run plain kubectl logs on it and see only two startup lines, nothing that explains the crash. Why, and what shows the real error?
Incorrect — the container does write logs; the problem is that each restart is a brand-new container, so plain logs shows the newest boot, not the run that failed.
Incorrect — nothing rate-limits the stream here; you are simply reading the newest container instance instead of the one that died.
Correct — the failed container is gone and replaced by a fresh one, so --previous is what surfaces the actual panic or stack trace.
Incorrect — events show the BackOff symptom, not the app's fatal message, which lives in the previous container's logs.
03You walk a misbehaving pod through get, describe, logs, and exec. describe is clean, the logs are clean, exec works, yet the pod still won't run reliably. What's the next hop outward?
Incorrect — a redeploy that lands on the same sick machine just reproduces the problem; you still haven't found the cause.
Correct — a node that's NotReady or under pressure quietly wrecks every pod on it while each pod looks blameless on its own.
Incorrect — changing resources at random is exactly the guessing the method exists to prevent, and nothing pointed at memory.
Incorrect — routing was never implicated by any check, and a blanket restart risks disrupting healthy traffic.
Backup & restoreApplication failure

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep