Inspect, logs and exec

Read a container's output, run commands inside it and pull facts from its config, even with no shell.

When a container misbehaves or disappears, Docker still holds what you need to find out why: the process output, the exit status and the full configuration, kept until the container is removed. Three commands read that evidence. docker logs replays what the process printed, docker exec runs a command inside a container that is still running, and docker inspect prints what Docker recorded about it. None of them needs a rebuild.

Use the main lab VM (secopslog-docker). Unpack the lesson files into ~/lab/inspectlogs; the only file you need from them is the noshell/ folder used near the end.

Something to look at

Start an nginx web server in the background and publish it on the VM's loopback address only, as in "docker run in depth". The long hex string is the container ID; docker ps shows its first 12 characters next to the name.

docker logs: what the process printed

A container's main process writes to standard output (stdout) and standard error (stderr) like any other program. Docker's logging driver captures both streams, so docker logs can replay them at any time, also after the container has stopped. The first block below comes from the image's entrypoint script, which prepares the configuration; the [notice] lines come from nginx itself. The middle of the output is shortened here.

Send one request and read only the newest line. --tail N limits the replay to the last N lines, and --timestamps prefixes each line with the time Docker received it, which matters when the application's own log lines carry no time or a different time zone. docker logs -f keeps following the output live until you press Ctrl+C.

One detail trips people up when they search logs with a pipe. docker logs writes the container's stdout to your stdout and the container's stderr to your stderr, keeping them separate. nginx writes its [notice] lines to stderr, so a plain pipe into grep never sees them. They go straight to the terminal, and grep -c counts 0. Redirect stderr into the pipe with 2>&1 and the count is right.

A container that exits at once

A common report is "the deploy succeeded, but docker ps is empty". This container runs a shell command with a typo in it. docker run -d prints an ID and returns, which only means the container was created and started. Plain docker ps lists running containers, so the dead one is missing. docker ps -a lists every container, including stopped ones, and its STATUS column carries the exit code.

Exited (127) and sh: ecoh: not found tell the same story: the shell could not find a program called ecoh. Read the exit code before you read anything else, because it tells you which layer failed. The table repeats the codes from "Images, containers and the core commands" and adds where to look next; the labs in this course produce all of them except the out-of-memory case.

docker exec: run a command inside

docker exec starts an extra process inside a running container, in the same filesystem, network and process namespaces as the main process. Use it for yes-or-no questions: which user the server runs as, whether a config file contains what you expect, which processes are alive. In the nginx image the master process runs as root and the workers as the nginx user. The listen [::]:80; line was added by the entrypoint script you saw in the logs.

The last command shows the limit: exec needs a running container, so it cannot help with lab-typo. For an interactive shell, add -i (keep stdin open) and -t (allocate a terminal) and name a shell the image contains. Type exit to leave; the container keeps running.

docker exec -it lab-web sh

docker inspect: what Docker recorded

docker inspect prints the container's full record as JSON: the command it ran, its state, its mounts, its network settings, its restart policy. The full output is a few hundred lines, so read it once with head to see the shape, then pull single values out with -f (or --format). The argument is a Go template: {{.State.Status}} walks from the top of the JSON into State, then Status. It works on stopped containers too, which is how you read lab-typo's exit code after the fact.

Older guides read a container's IP address from .NetworkSettings.IPAddress. Docker 29 removed that top-level field, and the template fails. A container can be attached to several networks and has one address per network, so the addresses live under .NetworkSettings.Networks, keyed by network name. range loops over that map. If you prefer jq (installed in the lab VM), the same path works there.

172.17.0.2 is an address on Docker's default bridge network; yours may differ, and it can change when the container is recreated, which is why "Container networking basics" connects containers by name instead. The port bindings confirm what docker ps summarised: container port 80 is published on 127.0.0.1:8080 only.

Where the logs are kept

The two values below explain a disk that fills up on a busy host. This engine uses the default json-file logging driver, and map[] means no options were set, so there is no max-size and no rotation. Every line the container prints is appended to one JSON file under /var/lib/docker/containers/<id>/ (readable only by root) until the container is removed. The file is deleted with the container, which is why docker rm throws away the evidence you were about to read.

Production hosts set rotation in the daemon configuration or per container; "Logging and log drivers" in Docker in depth covers max-size, max-file and the local driver.

When the image has no shell

Minimal images often ship without sh, ls or ps, because every tool left out is one an attacker cannot use either. The lesson files build such an image: one static binary and one HTML file on top of an empty base, standing in for a distroless application. You do not need to follow the two-stage Dockerfile yet; "Writing a Dockerfile" introduces the instructions and "BuildKit builds: stages, cache and mounts" in Docker in depth the stages.

# A stand-in for a distroless application image: one static binary, no shell, no package manager.
# Stage 1 fetches a statically linked busybox; stage 2 starts from an empty image (scratch).
FROM alpine:3.22 AS tools
RUN apk add --no-cache busybox-static
 
FROM scratch
COPY --from=tools /bin/busybox.static /sleep
COPY index.html /www/index.html
ENTRYPOINT ["/sleep", "3600"]

Exit code 127 again, this time because sh does not exist in the image. Trying bash will not help either. Two approaches work on any Docker Engine. The first starts a throwaway debug container from an image that has tools and joins it to the target's namespaces: --pid container:lab-noshell shares the process list, so the target's main process is PID 1 in the debug container too, and --network container:lab-noshell shares its network, so localhost means the target. Through /proc/1/root the debug container can also read the target's files.

The second is docker cp, which copies files out of (or into) a container from the outside and needs nothing inside the image. Docker Desktop adds a docker debug command that automates the first approach; it is part of Desktop, not of Docker Engine, so it is not in the lab VM. "Minimal bases: distroless, scratch, static and Alpine" in Advanced container security goes further with debug images.

Clean up

Try this

Run docker exec -it lab-web sh on a scratch host or disposable cluster and read the output against what this lesson described. Then change one input so it fails, and re-run: the error you get is the one you will meet in production.

Takeaway

If you keep one thing from inspect, logs and exec, keep “Clean up”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related