When a Pod will not run: a debugging routine

Pending, ImagePullBackOff, CrashLoopBackOff.

Beginner25 min · lesson 20 of 21
Lesson files
The scripts, test data and local test servers this lesson uses, exactly as they ran on the lab machine (5 files, 1 KB): troubleshoot.tar.gz. The lab VM shares no folders with your computer, so fetch them inside the VM: cd ~/lab && curl -fsSLO https://secopslog.com/lab-files/k8s-fund/troubleshoot.tar.gz && tar -xzf troubleshoot.tar.gz, which creates ~/lab/troubleshoot/. SHA-256: 42d62e90d9f110975e43d424d35b153ca466c65955305cdf7fccb41842a4ef29

Sooner or later you apply a manifest and the app does not come up. The Deployment exists, kubectl said created, and still nothing answers. It is tempting to start guessing: delete the Pod, apply the file again. That sometimes hides the problem and rarely explains it.

A short routine works better, because Kubernetes already records most of what went wrong. Look at the Pod status first. Then read the events, which say what the scheduler and the kubelet tried. Then read the container's logs. For a Service that does not answer, check its EndpointSlice. Below you practise it on four broken apps at once.

The routine
1kubectl get podswhich state is it in?2events / describewhat did Kubernetes try?3kubectl logswhat did the app say?4endpointslicesdoes the Service find Pods?
Most problems are named by step two.

Four broken apps

Each Deployment in this file has one mistake. Try to spot them before reading on.

broken.yaml
# Four small apps. Each one has one mistake in it.
apiVersion: apps/v1
kind: Deployment
metadata:
name: report
spec:
replicas: 1
selector:
matchLabels:
app: report
template:
metadata:
labels:
app: report
spec:
nodeSelector:
disktype: ssd
containers:
- name: nginx
image: nginx:1.30.6-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: shop
spec:
replicas: 1
selector:
matchLabels:
app: shop
template:
metadata:
labels:
app: shop
spec:
containers:
- name: nginx
image: nginx:1.30.66-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: worker
spec:
replicas: 1
selector:
matchLabels:
app: worker
template:
metadata:
labels:
app: worker
spec:
containers:
- name: worker
image: busybox:1.38.0
command: ["sh", "-c", "echo worker starting; slep 3600"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: api
spec:
replicas: 1
selector:
matchLabels:
app: api
template:
metadata:
labels:
app: api
spec:
containers:
- name: nginx
image: nginx:1.30.6-alpine
---
apiVersion: v1
kind: Service
metadata:
name: api
spec:
selector:
app: apis
ports:
- port: 80
ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl apply -f broken.yaml
deployment.apps/report created deployment.apps/shop created deployment.apps/worker created deployment.apps/api created service/api created

Some of these problems take a minute to show. Run kubectl get pods every few seconds until shop shows ImagePullBackOff and worker shows a RESTARTS count of 3 or more. That is where the routine starts.

Step 1: the status

ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl get pods
NAME READY STATUS RESTARTS AGE api-769f68d764-4cwq2 1/1 Running 0 94s report-d5657d7b8-qrhkg 0/1 Pending 0 94s shop-7cc9b4d854-kktxg 0/1 ImagePullBackOff 0 94s worker-56885d8f65-s76mh 1/1 Running 4 (49s ago) 94s

api looks healthy. report is Pending, which means it was never placed on a node. shop is ImagePullBackOff: the node could not get the image and is waiting before it tries again. worker is the sneaky one. It says Running and 1/1, but RESTARTS is 4. A container that crashes straight after starting is shown as Running for a moment after each restart and as CrashLoopBackOff (or Error) the rest of the time. A restart count that keeps growing is the sign to trust.

Step 2: the events

kubectl events lists what happened in the namespace, oldest first. --types=Warning leaves out the routine messages:

ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl events --types=Warning
LAST SEEN TYPE REASON OBJECT MESSAGE 94s Warning FailedScheduling Pod/report-d5657d7b8-qrhkg 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling. 49s (x3 over 88s) Warning BackOff Pod/worker-56885d8f65-s76mh Back-off restarting failed container worker in pod worker-56885d8f65-s76mh_default(099f695a-2ca6-4096-be02-84a6aaaae71e) 40s (x3 over 88s) Warning Failed Pod/shop-7cc9b4d854-kktxg Failed to pull image "nginx:1.30.66-alpine": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.30.66-alpine": failed to resolve reference "docker.io/library/nginx:1.30.66-alpine": docker.io/library/nginx:1.30.66-alpine: not found 40s (x3 over 88s) Warning Failed Pod/shop-7cc9b4d854-kktxg Error: ErrImagePull 12s (x4 over 88s) Warning Failed Pod/shop-7cc9b4d854-kktxg Error: ImagePullBackOff

Read the MESSAGE column for each object. For report, the scheduler says 0/3 nodes are available: one node has a taint (the control plane, which kind keeps free of apps) and the other two didn't match Pod's node affinity/selector. For worker, the kubelet is backing off from restarting a failed container. For shop, the pull failed with nginx:1.30.66-alpine: not found: that tag does not exist. The (x3 over 88s) style counts show the kubelet retrying, with longer waits each time.

The same events also appear at the bottom of kubectl describe pod, and kubectl events --for narrows the list to one object:

ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl describe pod -l app=report | tail -n 4
Events: Type Reason Age From Message ---- ------ ---- ---- ------- Warning FailedScheduling 94s default-scheduler 0/3 nodes are available: 1 node(s) had untolerated taint(s), 2 node(s) didn't match Pod's node affinity/selector. preemption: 0/3 nodes are available: 3 Preemption is not helpful for scheduling.
$ kubectl events --for "$(kubectl get pod -l app=shop -o name)"
LAST SEEN TYPE REASON OBJECT MESSAGE 94s Normal Scheduled Pod/shop-7cc9b4d854-kktxg Successfully assigned default/shop-7cc9b4d854-kktxg to lab-worker 42s (x3 over 93s) Normal Pulling Pod/shop-7cc9b4d854-kktxg Pulling image "nginx:1.30.66-alpine" 40s (x3 over 88s) Warning Failed Pod/shop-7cc9b4d854-kktxg Failed to pull image "nginx:1.30.66-alpine": rpc error: code = NotFound desc = failed to pull and unpack image "docker.io/library/nginx:1.30.66-alpine": failed to resolve reference "docker.io/library/nginx:1.30.66-alpine": docker.io/library/nginx:1.30.66-alpine: not found 40s (x3 over 88s) Warning Failed Pod/shop-7cc9b4d854-kktxg Error: ErrImagePull 12s (x4 over 88s) Normal BackOff Pod/shop-7cc9b4d854-kktxg Back-off pulling image "nginx:1.30.66-alpine" 12s (x4 over 88s) Warning Failed Pod/shop-7cc9b4d854-kktxg Error: ImagePullBackOff

Now the image story is complete: Scheduled on a node, Pulling, Failed with the reason, then BackOff. If the message had said pull access denied, the repository name would be the suspect, or a private registry that needs credentials. not found with a familiar repository usually means a typo in the tag.

Step 3: the logs

Events tell you the worker container keeps dying, not why. describe gives the exit code, and the logs give the reason:

ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl describe pod -l app=worker | grep -A 3 "Last State"
Last State: Terminated Reason: Error Exit Code: 127 Started: Wed, 11 Nov 2026 08:20:31 +0000
$ kubectl logs deployment/worker
worker starting sh: line 0: slep: not found
$ kubectl logs deployment/worker --previous
worker starting sh: line 0: slep: not found

Exit Code: 127 is the shell's code for "command not found", and the log line says it plainly: slep: not found. The command has a typo. --previous shows the logs of the container before the current one, which is what you need when the current one has just restarted and has not printed anything yet. Here both runs printed the same two lines.

Step 4: Services with no endpoints

api is running, yet nothing reaches it through its Service:

ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl get endpointslices -l kubernetes.io/service-name=api && kubectl get service api -o wide && kubectl get pods -l app=api --show-labels
NAME ADDRESSTYPE PORTS ENDPOINTS AGE api-nb864 IPv4 <unset> <unset> 95s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE SELECTOR api ClusterIP 10.96.67.54 <none> 80/TCP 95s app=apis NAME READY STATUS RESTARTS AGE LABELS api-769f68d764-4cwq2 1/1 Running 0 95s app=api,pod-template-hash=769f68d764

The EndpointSlice has <unset> endpoints, so the Service has no Pods behind it. The SELECTOR column shows why: the Service looks for app=apis, while the Pod carries app=api. No event or log points at this one, because nothing failed. The Service is correctly selecting nothing.

Fix all four

ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ diff broken.yaml fixed.yaml
1c1 < # Four small apps. Each one has one mistake in it. --- > # The same four apps with the mistakes fixed. 16,17d15 < nodeSelector: < disktype: ssd 38c36 < image: nginx:1.30.66-alpine --- > image: nginx:1.30.6-alpine 57c55 < command: ["sh", "-c", "echo worker starting; slep 3600"] --- > command: ["sh", "-c", "echo worker starting; sleep 3600"] 83c81 < app: apis --- > app: api
$ kubectl apply -f fixed.yaml && kubectl rollout status deployment/report && kubectl rollout status deployment/shop && kubectl rollout status deployment/worker && kubectl rollout status deployment/api
deployment.apps/report configured deployment.apps/shop configured deployment.apps/worker configured deployment.apps/api unchanged service/api configured Waiting for deployment "report" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "report" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "report" rollout to finish: 1 old replicas are pending termination... deployment "report" successfully rolled out deployment "shop" successfully rolled out Waiting for deployment "worker" rollout to finish: 1 old replicas are pending termination... Waiting for deployment "worker" rollout to finish: 1 old replicas are pending termination... deployment "worker" successfully rolled out deployment "api" successfully rolled out
$ kubectl get deployments && kubectl get endpointslices -l kubernetes.io/service-name=api
NAME READY UP-TO-DATE AVAILABLE AGE api 1/1 1 1 96s report 1/1 1 1 96s shop 1/1 1 1 96s worker 1/1 1 1 96s NAME ADDRESSTYPE PORTS ENDPOINTS AGE api-nb864 IPv4 80 10.244.1.3 96s

diff lists the four mistakes: an impossible nodeSelector, a tag with an extra digit, slep, and apis. After applying the fixed file, all four Deployments are 1/1 and the api EndpointSlice holds one address. The rollouts waited for the broken Pods to be replaced (old replicas are pending termination), which is how a fix should reach a Deployment: through its spec, not by deleting Pods by hand.

Debugging a container that has no shell

Many production images are distroless: they contain the app and nothing else, so kubectl exec ... sh has nothing to run. This Pod uses the tiny pause image, which holds a single program:

quiet.yaml
# A Pod whose image has no shell and no tools at all, like many distroless images.
apiVersion: v1
kind: Pod
metadata:
name: quiet
spec:
containers:
- name: app
image: registry.k8s.io/pause:3.10.2
ubuntu@secopslog-k8s:~/lab/troubleshoot · Kubernetes 1.37
$ kubectl apply -f quiet.yaml && kubectl wait --for=condition=Ready pod/quiet --timeout=90s
pod/quiet created pod/quiet condition met
$ kubectl exec quiet -- sh
error: Internal error occurred: Internal error occurred: error executing command in container: failed to exec in container: failed to start exec "9e558ab8aca1b4f5e59fd984a177d8e5094365430e08dd3a13099387fac2a423": OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH
$ kubectl debug quiet --image=busybox:1.38.0 --target=app --container=dbg -- ps kubectl wait pod/quiet --timeout=60s --for=jsonpath='{.status.ephemeralContainerStatuses[0].state.terminated.reason}'=Completed kubectl logs quiet -c dbg
Targeting container "app". If you don't see processes from this container it may be because the container runtime doesn't support this feature. pod/quiet condition met PID USER TIME COMMAND 1 65535 0:00 /pause 17 root 0:00 ps

exec failed with "sh": executable file not found in $PATH. kubectl debug added an ephemeral container with busybox to the running Pod instead, without restarting it. --target=app puts it in the same process namespace as the app container, so ps saw /pause as process 1. From such a container you can look at the app's processes, its network and, through /proc/1/root, its files. Ephemeral containers cannot be removed again; they go away with the Pod. Clean up with kubectl delete -f fixed.yaml -f quiet.yaml.

Hands-on

Here are four new apps with four new mistakes. Save the file as practice.yaml and name each cause before you read the answers below.

practice.yaml
# Hands-on: four more apps, four more mistakes. Find each one before you read the answers.
apiVersion: apps/v1
kind: Deployment
metadata:
name: cart
spec:
replicas: 1
selector:
matchLabels:
app: cart
template:
metadata:
labels:
app: cart
spec:
containers:
- name: nginx
image: nginx:1.30.6-alpine
resources:
requests:
memory: 64Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: search
spec:
replicas: 1
selector:
matchLabels:
app: search
template:
metadata:
labels:
app: search
spec:
containers:
- name: nginx
image: ngnix:1.30.6-alpine
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: mailer
spec:
replicas: 1
selector:
matchLabels:
app: mailer
template:
metadata:
labels:
app: mailer
spec:
containers:
- name: mailer
image: busybox:1.38.0
command: ["sh", "-c", "if [ -z \"$SMTP_HOST\" ]; then echo 'error: SMTP_HOST is not set'; exit 1; fi; sleep 3600"]
---
apiVersion: apps/v1
kind: Deployment
metadata:
name: catalog
spec:
replicas: 1
selector:
matchLabels:
app: catalog
template:
metadata:
labels:
app: catalog
spec:
containers:
- name: nginx
image: nginx:1.30.6-alpine
---
apiVersion: v1
kind: Service
metadata:
name: catalog
spec:
selector:
app: catalogue
ports:
- port: 80

1. Apply it: kubectl apply -f practice.yaml.

2. Check the status, repeating it every few seconds until search shows ImagePullBackOff and mailer shows CrashLoopBackOff or a restart count of 3 or more: kubectl get pods.

3. Read the warnings for the new apps only (older events from the first part are still there): kubectl events --types=Warning | grep -E "cart|search|mailer".

4. Read the log of the app that keeps restarting: kubectl logs deployment/mailer.

5. Check the Service: kubectl get endpointslices -l kubernetes.io/service-name=catalog && kubectl get service catalog -o wide.

6. Remove everything: kubectl delete -f practice.yaml && kubectl get deployments,services.

What you should see: in step 2, cart is Pending, search is in ImagePullBackOff, mailer is in CrashLoopBackOff with a growing restart count, and catalog is Running. Step 3 names two causes. cart gets 2 Insufficient memory, because it requests 64Gi and no node has that much. search gets pull access denied, repository does not exist, because the image name is misspelled ngnix. Step 4 prints error: SMTP_HOST is not set: the app needs an environment variable nobody gave it. In step 5 the EndpointSlice shows <unset> and the SELECTOR column shows app=catalogue, while the Pods carry app=catalog. After step 6 only the built-in kubernetes Service is left.

Remember

Status says which phase failed, events say what Kubernetes tried, logs say what the app said, and an empty EndpointSlice means a selector that matches nothing. Fix the manifest and apply it rather than deleting Pods, and reach for kubectl debug when the image has no shell.

Quick check
01A Pod has been Pending for ten minutes. Where do you look first?
Incorrect — A Pending Pod was never started, so there are no logs to read.
Correct — The scheduler records why no node fitted, as it did for report and cart.
Incorrect — A failed pull shows as ErrImagePull or ImagePullBackOff, after the Pod is scheduled.
Incorrect — A Pending Pod has not been sent to any node yet.
02kubectl get pods shows a Pod as Running with RESTARTS at 6 and rising. What is most likely going on?
Incorrect — A rollout creates new Pods. It does not raise the restart count of an old one.
Incorrect — The scheduler does not move running Pods. Restarts happen on the same node.
Incorrect — A failed pull shows ImagePullBackOff, and the container never runs at all.
Correct — The worker looked Running between crashes, and its logs said slep: not found.
03A Service gets no traffic, and its EndpointSlice shows <unset>. Its Pods are Running and Ready. What do you compare?
Correct — The Service wanted app=apis while the Pod had app=api, so nothing matched.
Incorrect — A wrong port still leaves the Pods listed as endpoints; the connection fails later.
Incorrect — The Pods already run, so the image was pulled fine.
Incorrect — Restarts have nothing to do with which Pods a Service selects.

Related