Debugging running pods

exec, port-forward, and ephemeral debug containers.

kubectl logs gave you nothing useful. The events look clean. And the app is still misbehaving. At some point you stop reading about the Pod from the outside and go in. (A Pod is the smallest thing Kubernetes runs: one or more containers that share a network address and a slice of the node's resources.) Three tools get you inside. kubectl exec runs a command in a container that's already up. kubectl port-forward opens a private tunnel from your laptop to one of the Pod's ports. kubectl debug bolts on a whole extra container for the case where the one you care about has no shell to run. Which one you grab depends on what's actually blocking you, so it pays to know what each does under the hood.

exec: run a command inside the container

Think of a building manager unlocking an office so you can walk in and check something yourself, instead of phoning the tenant and hoping they describe it right. That's kubectl exec. It runs a program inside a container that's already up, and most of the time that program is a shell, so you land at a prompt inside the container. From there you can read its files, print its environment variables, or check whether it can actually reach the database. Logs only tell you what the app chose to say. exec answers the two questions logs never do: is the config file really there with the right values, and can this Pod resolve and reach the service it depends on? One gotcha with multi-container Pods. If you don't name a container with -c <name>, exec attaches to the default one, which is often not the one you meant.

Here's what happens when you hit enter. The command doesn't run on your laptop, and it doesn't run on the control plane. kubectl opens a streaming connection to the kube-apiserver (the single front door to the whole cluster). The apiserver hands it to the kubelet (the agent process running on the node that hosts your Pod). The kubelet tells the container runtime to run your command inside the container's namespaces, as the container's own user. That relay is the reason exec needs a live container to attach to. No running process, nothing to exec into, and you get a container not found error instead of a shell.

kubectl exec web-7d9f8-abcde -- env | grep -E 'DB_|PORT'
kubectl exec web-7d9f8-abcde -- nslookup postgres.data

DB_HOST=postgres.data.svc.cluster.local
DB_PORT=5432
Server:    10.96.0.10
Address:   10.96.0.10:53
 
Name:      postgres.data.svc.cluster.local
Address:   10.24.1.53

port-forward: reach a port without exposing it

Say you want a temporary private phone line from your desk straight to a machine locked in a server room, without ever listing that machine in the public directory. That's the job kubectl port-forward does. It opens a tunnel from a port on your laptop to a port on a Pod or a Service (a Service being Kubernetes' stable name for a group of Pods). Now your own tools can hit an internal database, an admin endpoint, or a /metrics page directly, with nothing exposed to the internet: no load balancer, no Ingress. The tunnel is not a direct connection to the Pod. Like exec, it is streamed through the API server down to the kubelet, which is why it works from a laptop that has no network route to the Pod at all, why it needs permission on pods/portforward, and why it lives only as long as the command runs. Close the terminal and it's gone, which is what you want for a quick look. Aim it at a Service and Kubernetes forwards to one Pod behind that Service. So if the tunnel dies on you mid-session, odds are that Pod restarted and took the connection with it.

kubectl port-forward svc/postgres 5432:5432 -n data

Forwarding from 127.0.0.1:5432 -> 5432
Forwarding from [::1]:5432 -> 5432
Handling connection for 5432

With that running, psql -h 127.0.0.1 -p 5432 on your laptop is talking to the in-cluster database as if it lived on localhost. Same trick for a Redis admin port, or a Prometheus metrics endpoint you'd never want reachable from outside.

kubectl debug: when there's no shell to exec into

Hardened images are the modern headache. A distroless or scratch image (one built with no operating-system tools inside) ships your app binary and almost nothing else. No sh. No ls. No cat. Fewer programs inside means a smaller attack surface, which is a genuine security win, right up until the second you try to debug the thing and exec falls over because there's no shell for it to run.

kubectl exec -it api-distroless-6b8f4-9wq2p -- sh

error: Internal error occurred: error executing command in container: failed to exec in container: OCI runtime exec failed: exec failed: unable to start container process: exec: "sh": executable file not found in $PATH: unknown

You can't add a shell to an image you can't rebuild on the spot, so you bring your own toolbox instead. That's the idea behind kubectl debug. It sends in an inspector that carries its own tools and stands right next to the sealed machine. In plainer terms, it attaches an ephemeral container (a temporary extra container) to the running Pod, built from busybox or some similar image that's full of diagnostic tools. Kubernetes wires it in through a dedicated corner of the Pod's API called the ephemeralcontainers subresource. That's been generally available since v1.25, so on any current cluster there's no feature gate to flip. The new container shares the Pod's network address. Add --target and it also joins the process namespace of one specific container (they share the same view of what's running), so ps from your debug shell lists that container's processes. What it doesn't get: a restart policy, health probes, or resource requests. And once it's attached you can't edit it or remove it. It rides along until the whole Pod gets recreated.

kubectl debug -it api-distroless-6b8f4-9wq2p --image=busybox:1.36 --target=api

Targeting container "api". If you don't see processes from this container it may be because the container runtime doesn't support this feature.
Defaulting debug container name to debugger-x7k2m.
If you don't see a command prompt, try pressing enter.
/ # ps
PID   USER     TIME  COMMAND
    1 65532     0:03 /app/server
   19 root      0:00 sh
   26 root      0:00 ps
/ # ls /app
ls: /app: No such file or directory
/ # cat /proc/1/root/app/config.yaml
listen: ":8080"
db: postgres.data.svc.cluster.local

Want to confirm the debug container actually landed? Read the ephemeralContainers field straight back off the Pod object. This is also how you catch old debug containers piling up on a long-lived Pod, because nothing ever cleans them up for you.

kubectl get pod api-distroless-6b8f4-9wq2p -o jsonpath='{.spec.ephemeralContainers[*].name}'

debugger-x7k2m

One case still trips people, and it is not the one you would guess. A Pod stuck in CrashLoopBackOff (a container that keeps failing, getting restarted, and failing again) still takes a debug container fine, because an ephemeral container attaches to the Pod, and the Pod itself is up the whole time. What fails is exec, which needs a live process, and --target aimed at the container that keeps dying, since there is no running process whose namespace it could join. A debug container on its own also can't see inside that container's filesystem. So when what you need is the crashing container's own files and environment with the restart loop switched off, kubectl debug <pod> --copy-to=<name> makes a fresh copy of the Pod, and in the same command you swap that container's start command for something that just idles, like a long sleep. The copy comes up and stays up, running the exact same image and config, so you can exec in and look around. Even before any of that, kubectl logs --previous hands you the last output of the container that just died, which is often the whole answer. When the trouble is the node itself and not any single Pod, kubectl debug node/<node> -it --image=busybox starts a Pod on that node that shares the host's network and process namespaces and mounts the host's whole filesystem at /host, so the node's own files are under /host/var/log, not /var/log. That Pod is not privileged by default; add --profile=sysadmin when you need it to be. Either way you can read the kubelet's logs or check for disk pressure without ever opening an SSH session (SSH being the usual way you'd remotely log into a server).

RBAC may deny pods/exec. That is often intentional in prod. Use a break-glass role with audit.

port-forward is not an Ingress. It is a temporary tunnel through the API server.

Ephemeral containers are standard on any current cluster, with no feature gate to turn on. They join the Pod's network, and with --target one container's process namespace, so treat permission to add them as sensitive.

Try this

exec into a running pod, port-forward a Service to localhost, then start an ephemeral debug container on a distroless workload.

$ kubectl exec web-7d9f8-abcde -- env | grep -E 'DB_|PORT'
$ kubectl exec web-7d9f8-abcde -- nslookup postgres.data
 
$ kubectl port-forward svc/postgres 5432:5432 -n data
 
$ kubectl exec -it api-distroless-6b8f4-9wq2p -- sh
 
$ kubectl debug -it api-distroless-6b8f4-9wq2p --image=busybox:1.36 --target=api
 
$ kubectl get pod api-distroless-6b8f4-9wq2p -o jsonpath='{.spec.ephemeralContainers[*].name}'

Takeaway

exec, port-forward, and ephemeral debug containers are the live toolkit. Prefer them over SSH to nodes for app issues.

Related