Application failure

Working a broken app from Service down to pod.

Checkout is down. That's the whole ticket. No stack trace, no attached logs, just a customer-facing feature that stopped and a clock that's now running against you. The fastest way through is boring and it always works. Start where the request enters the app and walk inward, one layer at a time, asking a simple question at each step. Is the phone even ringing? Is anyone at the desk to answer it? Are they well enough to talk? And what are they actually saying? In Kubernetes those four questions land on the Service, its endpoints, the pod's health, and the logs. Most of the time the answer is a mislabeled desk, not a sick employee.

The first move never changes. List the pods and read the STATUS column, because that one word names the broken layer before you've read a single log line. A pod is the smallest thing Kubernetes runs, one or more containers scheduled together as a unit.

$ kubectl get pods -l app=checkout
NAME                       READY   STATUS             RESTARTS      AGE
checkout-7c9f8b6d5-2xk4p   0/1     CrashLoopBackOff   6 (28s ago)   5m
checkout-7c9f8b6d5-9pm2t   0/1     CrashLoopBackOff   6 (15s ago)   5m

Read the state, it names the layer

Each status points at exactly one subsystem. Pending means the scheduler could not find a node to place the pod on, the way a full restaurant can't seat you. Describe the pod and read the FailedScheduling event: not enough CPU or memory to fit, a taint on the nodes you didn't tolerate, or a PersistentVolumeClaim (the pod's request for a slice of disk) that never bound to a volume. ImagePullBackOff means the kubelet, the Kubernetes agent running on every node, tried to download the container image and failed, usually a wrong tag or a private registry with no imagePullSecret (the login it needs to pull). CrashLoopBackOff is a different animal. The container starts fine, the process inside dies, and Kubernetes restarts it again and again, like an engine that turns over and stalls. That one is the application's own fault, and the evidence sits in the logs of the run that just died. CreateContainerConfigError means the pod refers to a ConfigMap or Secret that isn't there. OOMKilled means the container ran past its memory limit and the kernel shot it.

$ kubectl describe pod checkout-7c9f8b6d5-2xk4p
...
Events:
  Type     Reason   Age                 From     Message
  ----     ------   ----                ----     -------
  Normal   Pulled   5m                  kubelet  Successfully pulled image "checkout:1.8.2"
  Normal   Created  4m (x4 over 5m)     kubelet  Created container checkout
  Normal   Started  4m (x4 over 5m)     kubelet  Started container checkout
  Warning  BackOff  28s (x18 over 5m)   kubelet  Back-off restarting failed container checkout

The BackOff in the name is literal. After a crash the kubelet waits 10 seconds before the next try, then 20, then 40, doubling up to a five-minute ceiling. So a pod that's been failing a while updates slowly, sitting in a penalty box between attempts. The RESTARTS count and that (x18 over 5m) on the event tell you how long it's been going. The describe told you the pod is looping. It did not tell you why. For that you read the logs of the previous container, the dead one, because the live container is often mid-restart and hasn't printed anything useful yet.

$ kubectl logs checkout-7c9f8b6d5-2xk4p --previous
2026-07-16T09:14:02Z INFO  starting checkout v1.8.2
2026-07-16T09:14:02Z FATAL config: env DB_HOST is required but was empty
exit status 1

There's the confession. The app wants a DB_HOST environment variable, it's empty, so the process exits on startup every single time. Deleting the pod and hoping does nothing but restart the same crash. Fix the config that feeds that variable, the container stays up, and the READY column flips to 1/1. Notice the whole diagnosis took two commands and no guessing.

Running but unreachable: walk the Service to the pod

Now the harder case. The pods are Running, 1/1 Ready, and the app still can't be reached. Time to walk the connection path. A Service is a stable phone number for a group of pods that come and go. You dial the Service and it forwards you to whoever's on shift right now. That shift roster is the endpoints list. A pod earns a spot on it by doing two things: wearing the labels the Service is told to look for (its label selector), and passing its readiness probe, the health check that decides whether a pod is ready to take traffic. So the first question is whether anyone made the roster at all.

$ kubectl get endpoints checkout
NAME       ENDPOINTS   AGE
checkout   <none>      6d
 
$ kubectl get svc checkout -o jsonpath='{.spec.selector}'
{"app":"checkout-api"}
 
$ kubectl get pods -l app=checkout --show-labels
NAME                       READY   STATUS    RESTARTS   AGE   LABELS
checkout-7c9f8b6d5-2xk4p   1/1     Running   0          3m    app=checkout,pod-template-hash=7c9f8b6d5

Empty endpoints next to healthy, Ready pods is almost never a networking fault. The roster is empty because the Service is hunting for a label the pods don't wear. Here the Service selects app=checkout-api and the pods are labeled app=checkout. No match, no endpoints, no traffic. Fix the selector, or relabel the pods, and the roster fills the same second. Had you started by capturing packets on the Container Network Interface plugin that wires pod networking, or by restarting kube-proxy (the agent that turns the endpoint list into routing rules on each node), you'd have burned the afternoon. There was nothing there for it to route to.

One note on that first command. kubectl get endpoints reads the older Endpoints object, which Kubernetes deprecated in version 1.33 in favor of EndpointSlice, and whose output gets cut short once a Service has a lot of pods behind it. It still works and it is still the fastest glance, but the future-proof way to ask the same question is kubectl get endpointslices -l kubernetes.io/service-name=checkout, which lists every address the cluster actually holds.

When endpoints do exist and connections still fail, suspect the port. A Service forwards traffic to a targetPort on the pod. If that number doesn't match the port the container actually listens on, every connection is refused while everything still looks healthy in kubectl.

$ kubectl get svc checkout -o jsonpath='{.spec.ports[0].targetPort}'
8080
$ kubectl get pod checkout-7c9f8b6d5-2xk4p -o jsonpath='{.spec.containers[0].ports[0].containerPort}'
3000

targetPort 8080, containerPort 3000. The Service is knocking on a door the app never opened, which reads as connection refused. Line the two numbers up, then prove the fix from inside the cluster with a throwaway pod instead of trusting the YAML. A quick curl that returns 200 is the difference between thinking it works and knowing it does.

$ kubectl run tmp --rm -it --image=nicolaka/netshoot --restart=Never -- \
    curl -s -o /dev/null -w '%{http_code}\n' http://checkout.shop:80
200
pod "tmp" deleted

CrashLoopBackOff wants previous logs and exit codes. ImagePullBackOff wants registry auth and tag existence.

A Running pod with zero endpoints is often readiness or selectors — not a network partition.

Roll back if the last change correlates. Heroes debug forward; professionals also revert.

Try this

Break a Service selector or readiness probe on purpose, then diagnose from Service to endpoints to the pods' labels and READY column until you find your own sabotage.

# Build something healthy first, then break it on purpose.
$ kubectl create deployment shop --image=nginx --replicas=2
$ kubectl expose deployment shop --port=80 --target-port=80
$ kubectl get endpointslices -l kubernetes.io/service-name=shop
NAME         ADDRESSTYPE   PORTS   ENDPOINTS               AGE
shop-9wq4t   IPv4          80      10.244.1.7,10.244.2.4   20s
 
# Sabotage one: aim the Service at a label no pod is wearing.
$ kubectl patch svc shop -p '{"spec":{"selector":{"app":"shop-api"}}}'
service/shop patched
 
# Sabotage two: put the selector back, then give the pods a readiness
# check they cannot pass (nginx answers /nope with a 404).
$ kubectl patch svc shop -p '{"spec":{"selector":{"app":"shop"}}}'
$ kubectl patch deploy shop -p '{"spec":{"template":{"spec":{"containers":[{"name":"nginx","readinessProbe":{"httpGet":{"path":"/nope","port":80},"periodSeconds":5}}]}}}}'
deployment.apps/shop patched
 
# Now diagnose it cold, without scrolling back up this page.
# Both breaks empty the roster, so work out which one you are looking at.
# Clean up when you are done.
$ kubectl delete deployment,service shop

Takeaway

App failures usually live in labels, readiness, image pulls, or crashes. Start with kubectl get pods and let the STATUS word name the layer; walk the Service to its endpoints only when the pods are Ready and traffic still fails.

Related