>_SecOpsLog_
CoursesKubernetes administrationApplication logs

Application logs

kubectl logs, streaming, and multi-container pods.

Beginner8 min · lesson 53 of 65
In plain terms
Logs are the diary each container writes as it works; kubectl logs reads that diary back to you. To search the whole fleet’s diaries at once, you copy them into a shared library (a log store).

Your pod keeps dying. kubectl get pods shows CrashLoopBackOff, which is Kubernetes telling you it has given up restarting a container that won't stay up and is now waiting longer between each try. You run kubectl logs on the pod and get two lines from a container that started a second ago, and nothing about why the last one died. The output that would explain the crash left with the container that produced it. A log is the running diary a program keeps while it works: every line it prints goes in, one after another. This lesson is about reading that diary well, knowing where it physically lives on the machine, and what to do once the pod (the smallest thing Kubernetes runs, one container or a few that share a fate) that wrote it is gone.

What kubectl logs is actually reading

A well-behaved container writes to stdout and stderr, the two output streams every program has by default (the same ones that print to your terminal when you run a command). It does not write to a file inside the container. Here is why that matters. The container runtime, the software on each node that actually starts and stops containers (usually containerd), captures those two streams and writes them to a file on the node's disk. When you run kubectl logs, the command never touches the container. It hits the pod's log subresource on the API server (the cluster's front door, GET /api/v1/namespaces/<ns>/pods/<pod>/log), which forwards the request to the kubelet (the Kubernetes agent running on the node that holds the pod), and the kubelet reads that file and streams it back. So kubectl logs is a remote tail of a file the runtime keeps for you. It also explains a common dead end: an app that writes to /var/log/app.log inside its own container is invisible to kubectl logs, because nothing is capturing that file.

read the current container's stdout/stderr
kubectl logs web-5d8f -c app --timestamps --tail=5
output
2026-07-16T09:14:02.118471Z  INFO  config loaded from /etc/app/config.yaml
2026-07-16T09:14:02.140233Z  INFO  connecting to postgres at db:5432
2026-07-16T09:14:02.361902Z  INFO  migrations up to date (schema v41)
2026-07-16T09:14:02.362550Z  INFO  listening on :8080
2026-07-16T09:14:02.362701Z  INFO  readiness probe endpoint /healthz ready

The options that decide whether you find the bug

The flags are the difference between a two-minute fix and an hour of guessing. -f follows the stream live, the same as tail -f. --tail=N, --since=15m, and --since-time bound how much you pull back, which you want on a service that logs thousands of lines a minute. -c names a container in a multi-container pod. Leave it off on a pod that also runs a sidecar (a helper container sitting next to the main one) and kubectl does not error. It picks the first container in the spec, or the one named by the pod's default-container annotation if that is set, and prints a "Defaulted container ..." line so you know which one you got. Grab the wrong container's log by accident and you will stare at a perfectly healthy sidecar while your app burns. An init container (a setup step that runs to completion before the main app starts) has its own log too; you name it explicitly, since it has already finished by the time the pod shows Running. --all-containers=true merges them all. The one flag that saves you most often is --previous (short -p). After a crash the current container is a fresh restart with almost nothing in it, so --previous reads the log of the instance that just died, which is where the actual error lives.

the crash is in the PREVIOUS instance
kubectl logs web-5d8f --previous -c app
output
2026-07-16T09:12:58.402Z  INFO  connecting to postgres at db:5432
2026-07-16T09:12:59.905Z  FATAL dial tcp 10.96.0.31:5432: connect: connection refused
panic: database unreachable, giving up after 3 attempts
 
goroutine 1 [running]:
main.mustConnect(...)
        /src/main.go:48
main.main()
        /src/main.go:22 +0x1a5

To tail a whole workload instead of one pod, point the selector at its label and let --prefix tag each line with the pod it came from. Kubernetes streams from up to five matching pods by default and refuses more, so it does not open a flood of concurrent streams against the API server and the kubelets at once. When you have more replicas than that, raise the cap explicitly.

tail every replica at once, labelled by pod
kubectl logs -l app=web --prefix --tail=2 --max-log-requests=10
output
[pod/web-5d8f/app] 2026-07-16T09:14:03Z  GET /healthz 200 0.4ms
[pod/web-5d8f/app] 2026-07-16T09:14:04Z  GET /orders  200 12.8ms
[pod/web-7bc2/app] 2026-07-16T09:14:03Z  GET /healthz 200 0.5ms
[pod/web-7bc2/app] 2026-07-16T09:14:04Z  POST /orders 201 22.1ms

Where the logs live, and why they vanish

The file the runtime writes has a predictable path on the node: /var/log/pods/<namespace>_<pod>_<uid>/<container>/<restart-count>.log. The number is the restart count, so 0.log is the first run of the container, 1.log the second, and --previous is really "read restart-count minus one." You can see the files directly by launching a debug pod on the node, which mounts the host filesystem under /host.

list the raw log files on the node
kubectl debug node/node-2 -it --image=busybox -- ls /host/var/log/pods/prod_web-5d8f_9f3a1c2b/app
output
Creating debugging pod node-debugger-node-2-k4p9 with container debugger on node node-2.
0.log  1.log  2.log

Two limits bite here. First, the kubelet rotates these files: by default it caps each container's log at 10 MB and keeps five of them (the containerLogMaxSize and containerLogMaxFiles settings). A busy container overwrites its own older lines within minutes, so kubectl logs shows only what is still on disk, never the full history, even for a pod that is running fine. Second, --previous reaches exactly one restart back. A pod that has crash-looped forty times will show you crash forty and has no way to show you crash one. Grab the logs the moment you notice the problem.

When the pod is gone, so are its logs

Everything above reads from one node's disk. Delete the pod, or let the kubelet evict it when the node runs low on memory, or lose the node to a hardware failure, and every one of those files goes with it. That is fine for "debug this pod right now" and useless for "what did checkout log when it broke last Tuesday." The fix is central aggregation. A logging agent runs as a DaemonSet (a controller that pins one copy of a pod to every node, the way a building puts a mailroom on every floor). Each copy tails the files under /var/log/pods and ships them to a store you can search across the whole fleet: Loki, Elasticsearch or OpenSearch, or a cloud logging service. Fluent Bit and Vector are the usual agents. One habit makes this pay off: log structured lines (JSON with consistent fields like level, msg, and a request id) to stdout instead of free-form text, so you can filter and correlate across services in the store rather than grepping prose. Now the logs outlive the pods, and last week's incident still has evidence.

Pick the log tool by the pod's state
You need a pod's logs. What state is it in?
the pod's condition decides the tool
running now
live tail of stdout/stderr
kubectl logs -f <pod> -c <c>
crash-looping
the dead instance holds the crash output
kubectl logs <pod> --previous
deleted / node gone
only aggregation kept it
query the central store
kubectl logs reads one node's on-disk file. --previous recovers a crashed instance; once the pod is gone, only a DaemonSet agent shipping to a central store still has the history.
kubectl logs isn't your full history, and --previous only goes back one restart
The kubelet rotates each container's log on the node, by default 10 MB per file and five files kept (containerLogMaxSize and containerLogMaxFiles). A chatty or fast-crashing container overwrites its own older lines within minutes, so kubectl logs shows only what is still on disk even while the pod is running. And --previous reaches exactly one restart back: a pod that has crash-looped fifty times will not show you the first failure. Grab the logs the moment you see the problem, and ship everything to a central store so the history survives both rotation and the pod itself.

Previous logs vanish when the kubelet rotates. Catch CrashLoop evidence early.

Do not log secrets. Once printed, they live in aggregator storage and tickets.

Node disk pressure kills log retention first. Full disks look like silent applications. Catch CrashLoop evidence early.

Try this

Fetch logs from a multi-container pod with -c, follow briefly with --tail, and grab previous logs from a crashed container if one exists.

terminal
$ kubectl logs web-5d8f -c app --timestamps --tail=5
 
$ kubectl logs web-5d8f --previous -c app
 
$ kubectl logs -l app=web --prefix --tail=2 --max-log-requests=10
 
$ kubectl debug node/node-2 -it --image=busybox -- ls /host/var/log/pods/prod_web-5d8f_9f3a1c2b/app

Takeaway

kubectl logs reads what the kubelet kept for a container. Multi-container pods need -c. Long-term search needs a log shipper.

Quick check
01A pod has been in CrashLoopBackOff for an hour. kubectl logs <pod> prints only two startup lines and nothing about the failure. What is the fastest way to see why it is dying?
Correct — the running container is a fresh restart with almost nothing in it; --previous reads the instance that actually crashed, where the error is.
Incorrect — follow just tails the current restart, which also dies before printing the cause, so you watch the same empty startup repeat.
Incorrect — that destroys the crash evidence still on the node, and the fresh pod will almost certainly crash the same way.
Incorrect — top shows CPU and memory, not the application error; it will not tell you why the process exits.
02An app writes its logs to /var/log/app.log inside its own container. The app is clearly running, but kubectl logs on the pod returns nothing. Why?
Incorrect — the 10 MB figure is the kubelet's on-node rotation of captured streams, not a limit on a file inside the container.
Correct — the runtime tees stdout/stderr to a node file that kubectl logs reads, so anything the app writes to its own filesystem stays invisible to it.
Incorrect — --timestamps only prepends times to captured stream output; it can't surface a file the runtime never captured.
Incorrect — that path is written by the runtime from the captured streams, and an app can't populate it by writing its own file.
03On a pod with an app container and a logging sidecar, you run kubectl logs with no -c. It prints a 'Defaulted container ...' line and shows healthy traffic, but you're chasing a crash. What is going on?
Incorrect — merging all containers is what --all-containers does; without it kubectl reads exactly one container.
Incorrect — the sidecar suppresses nothing; you simply weren't shown the app container's log.
Correct — with no -c kubectl picks a default and tells you which, so on a multi-container pod you can end up reading a healthy sidecar while the app burns.
Incorrect — that line only reports which container was auto-selected, not whether another container has output.
Monitoring cluster componentsEvents & the event stream

Related

What Kubernetes isKubernetes fundamentalsSecurity domains & threat modelKubernetes security & hardeningThe cluster attack surfaceKubernetes attack & defenseKubernetes interview questionsInterview prep