When a Pod will not run: a debugging routine
Pending, ImagePullBackOff, CrashLoopBackOff.
cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/k8s-fund/troubleshoot.tar.gz && tar -xzf troubleshoot.tar.gz, which creates ~/lab/troubleshoot/. SHA-256: 42d62e90d9f110975e43d424d35b153ca466c65955305cdf7fccb41842a4ef29Sooner or later you apply a manifest and the app does not come up. The Deployment exists, kubectl said created, and still nothing answers. It is tempting to start guessing: delete the Pod, apply the file again. That sometimes hides the problem and rarely explains it.
A short routine works better, because Kubernetes already records most of what went wrong. Look at the Pod status first. Then read the events, which say what the scheduler and the kubelet tried. Then read the container's logs. For a Service that does not answer, check its EndpointSlice. Below you practise it on four broken apps at once.
Four broken apps
Each Deployment in this file has one mistake. Try to spot them before reading on.
# Four small apps. Each one has one mistake in it.apiVersion: apps/v1kind: Deploymentmetadata:name: reportspec:replicas: 1selector:matchLabels:app: reporttemplate:metadata:labels:app: reportspec:nodeSelector:disktype: ssdcontainers:- name: nginximage: nginx:1.30.6-alpine---apiVersion: apps/v1kind: Deploymentmetadata:name: shopspec:replicas: 1selector:matchLabels:app: shoptemplate:metadata:labels:app: shopspec:containers:- name: nginximage: nginx:1.30.66-alpine---apiVersion: apps/v1kind: Deploymentmetadata:name: workerspec:replicas: 1selector:matchLabels:app: workertemplate:metadata:labels:app: workerspec:containers:- name: workerimage: busybox:1.38.0command: ["sh", "-c", "echo worker starting; slep 3600"]---apiVersion: apps/v1kind: Deploymentmetadata:name: apispec:replicas: 1selector:matchLabels:app: apitemplate:metadata:labels:app: apispec:containers:- name: nginximage: nginx:1.30.6-alpine---apiVersion: v1kind: Servicemetadata:name: apispec:selector:app: apisports:- port: 80
Some of these problems take a minute to show. Run kubectl get pods every few seconds until shop shows ImagePullBackOff and worker shows a RESTARTS count of 3 or more. That is where the routine starts.
Step 1: the status
api looks healthy. report is Pending, which means it was never placed on a node. shop is ImagePullBackOff: the node could not get the image and is waiting before it tries again. worker is the sneaky one. It says Running and 1/1, but RESTARTS is 4. A container that crashes straight after starting is shown as Running for a moment after each restart and as CrashLoopBackOff (or Error) the rest of the time. A restart count that keeps growing is the sign to trust.
Step 2: the events
kubectl events lists what happened in the namespace, oldest first. --types=Warning leaves out the routine messages:
Read the MESSAGE column for each object. For report, the scheduler says 0/3 nodes are available: one node has a taint (the control plane, which kind keeps free of apps) and the other two didn't match Pod's node affinity/selector. For worker, the kubelet is backing off from restarting a failed container. For shop, the pull failed with nginx:1.30.66-alpine: not found: that tag does not exist. The (x3 over 88s) style counts show the kubelet retrying, with longer waits each time.
The same events also appear at the bottom of kubectl describe pod, and kubectl events --for narrows the list to one object:
Now the image story is complete: Scheduled on a node, Pulling, Failed with the reason, then BackOff. If the message had said pull access denied, the repository name would be the suspect, or a private registry that needs credentials. not found with a familiar repository usually means a typo in the tag.
Step 3: the logs
Events tell you the worker container keeps dying, not why. describe gives the exit code, and the logs give the reason:
Exit Code: 127 is the shell's code for "command not found", and the log line says it plainly: slep: not found. The command has a typo. --previous shows the logs of the container before the current one, which is what you need when the current one has just restarted and has not printed anything yet. Here both runs printed the same two lines.
Step 4: Services with no endpoints
api is running, yet nothing reaches it through its Service:
The EndpointSlice has <unset> endpoints, so the Service has no Pods behind it. The SELECTOR column shows why: the Service looks for app=apis, while the Pod carries app=api. No event or log points at this one, because nothing failed. The Service is correctly selecting nothing.
Fix all four
diff lists the four mistakes: an impossible nodeSelector, a tag with an extra digit, slep, and apis. After applying the fixed file, all four Deployments are 1/1 and the api EndpointSlice holds one address. The rollouts waited for the broken Pods to be replaced (old replicas are pending termination), which is how a fix should reach a Deployment: through its spec, not by deleting Pods by hand.
Debugging a container that has no shell
Many production images are distroless: they contain the app and nothing else, so kubectl exec ... sh has nothing to run. This Pod uses the tiny pause image, which holds a single program:
# A Pod whose image has no shell and no tools at all, like many distroless images.apiVersion: v1kind: Podmetadata:name: quietspec:containers:- name: appimage: registry.k8s.io/pause:3.10.2
exec failed with "sh": executable file not found in $PATH. kubectl debug added an ephemeral container with busybox to the running Pod instead, without restarting it. --target=app puts it in the same process namespace as the app container, so ps saw /pause as process 1. From such a container you can look at the app's processes, its network and, through /proc/1/root, its files. Ephemeral containers cannot be removed again; they go away with the Pod. Clean up with kubectl delete -f fixed.yaml -f quiet.yaml.
Hands-on
Here are four new apps with four new mistakes. Save the file as practice.yaml and name each cause before you read the answers below.
# Hands-on: four more apps, four more mistakes. Find each one before you read the answers.apiVersion: apps/v1kind: Deploymentmetadata:name: cartspec:replicas: 1selector:matchLabels:app: carttemplate:metadata:labels:app: cartspec:containers:- name: nginximage: nginx:1.30.6-alpineresources:requests:memory: 64Gi---apiVersion: apps/v1kind: Deploymentmetadata:name: searchspec:replicas: 1selector:matchLabels:app: searchtemplate:metadata:labels:app: searchspec:containers:- name: nginximage: ngnix:1.30.6-alpine---apiVersion: apps/v1kind: Deploymentmetadata:name: mailerspec:replicas: 1selector:matchLabels:app: mailertemplate:metadata:labels:app: mailerspec:containers:- name: mailerimage: busybox:1.38.0command: ["sh", "-c", "if [ -z \"$SMTP_HOST\" ]; then echo 'error: SMTP_HOST is not set'; exit 1; fi; sleep 3600"]---apiVersion: apps/v1kind: Deploymentmetadata:name: catalogspec:replicas: 1selector:matchLabels:app: catalogtemplate:metadata:labels:app: catalogspec:containers:- name: nginximage: nginx:1.30.6-alpine---apiVersion: v1kind: Servicemetadata:name: catalogspec:selector:app: catalogueports:- port: 80
1. Apply it: kubectl apply -f practice.yaml.
2. Check the status, repeating it every few seconds until search shows ImagePullBackOff and mailer shows CrashLoopBackOff or a restart count of 3 or more: kubectl get pods.
3. Read the warnings for the new apps only (older events from the first part are still there): kubectl events --types=Warning | grep -E "cart|search|mailer".
4. Read the log of the app that keeps restarting: kubectl logs deployment/mailer.
5. Check the Service: kubectl get endpointslices -l kubernetes.io/service-name=catalog && kubectl get service catalog -o wide.
6. Remove everything: kubectl delete -f practice.yaml && kubectl get deployments,services.
What you should see: in step 2, cart is Pending, search is in ImagePullBackOff, mailer is in CrashLoopBackOff with a growing restart count, and catalog is Running. Step 3 names two causes. cart gets 2 Insufficient memory, because it requests 64Gi and no node has that much. search gets pull access denied, repository does not exist, because the image name is misspelled ngnix. Step 4 prints error: SMTP_HOST is not set: the app needs an environment variable nobody gave it. In step 5 the EndpointSlice shows <unset> and the SELECTOR column shows app=catalogue, while the Pods carry app=catalog. After step 6 only the built-in kubernetes Service is left.
Remember
Status says which phase failed, events say what Kubernetes tried, logs say what the app said, and an empty EndpointSlice means a selector that matches nothing. Fix the manifest and apply it rather than deleting Pods, and reach for kubectl debug when the image has no shell.
Pending for ten minutes. Where do you look first?report and cart.kubectl get pods shows a Pod as Running with RESTARTS at 6 and rising. What is most likely going on?Running between crashes, and its logs said slep: not found.<unset>. Its Pods are Running and Ready. What do you compare?app=apis while the Pod had app=api, so nothing matched.