Container forensics and incident response

Contain, capture and diff a suspect container before you destroy it.

The reflex that ends an outage, kill it and let it come back clean, destroys the evidence of a compromise. A container keeps that evidence in places the quick fixes erase in seconds: live process memory and open sockets (gone when the process dies), the container's log and configuration (gone on docker rm with the default json-file driver), and the writable layer stacked on the read-only image (also gone on docker rm). The job in the first few minutes is preservation, in one fixed order: contain, capture, investigate, remediate. This lesson runs that order on a benign suspect on the main lab VM (secopslog-docker), in ~/lab/forensics. The suspect plants its own fake evidence and exploits nothing.

The suspect logs one line, drops a copy of a binary, adds a crontab line, opens a TCP connection to lab-c2 (a stand-in for an attacker's server on the same network) and then sleeps, with the fake command-and-control address in its environment:

Contain: freeze, record, then cut the network

Freeze first. docker pause stops every thread at once through the cgroup freezer. It acts underneath the process, so there is no signal for an attacker to trap and nothing gets a chance to run a clean-up, and a paused container refuses docker exec, which is why every capture step runs from the host:

Then write down what the daemon knows before anything changes it. The host-side PID is the number the kernel gives the container's main process (not the 1 it sees inside), and the next steps read the kernel's view of exactly that process. docker inspect is the full record of how the container was started: image digest, mounts, environment, restart policy, labels and network settings. docker logs is everything the process wrote to stdout and stderr. Both disappear with docker rm, so they go into the evidence directory now. The directory is mode 700; on a real host, write to a root-only location on separate storage and copy it off the host as soon as you can:

Record the connections next, from the host, with nsenter joining only the container's network namespace ("Namespaces from first principles" introduced the method), so the ss binary is the host's and cannot have been replaced by the attacker:

The ESTAB line is the open channel: the suspect's nc connected to 172.18.0.2:4444, the address of lab-c2. On a real incident that peer address is often the most valuable volatile fact you collect. (The LISTEN line on 127.0.0.11 is Docker's embedded DNS resolver.) Now cut the network:

Disconnecting removes the container's interface, so the connection can no longer carry traffic: anything sent on it fails and the socket times out, and the address that tied it to the network is gone. That is why the socket table was saved first. If the container is actively sending data out, cut first and accept the loss, or drop its traffic with a host firewall rule on its address, which stops the flow and leaves the socket state readable.

Capture from the host, not from inside

A compromised container controls its own userspace, so its ps, ls and cat can lie, and its own files can be edited. Read the kernel's view from the host instead, and know which parts of it you can trust. Kernel-maintained entries such as /proc/<pid>/exe, fd, maps and status cannot be forged from inside: exe is the running binary, and it still resolves even if the attacker unlinked the file (it then reads (deleted)). /proc/<pid>/cmdline and environ are different. The kernel reads them out of the process's own memory, so a process can overwrite them (argument rewriting is routine malware behaviour), and environ shows the environment block the process started with, not variables it set later. Treat those two as claims to corroborate. Pull the dropped artifact with docker cp, which reads the filesystem through the daemon and works on a paused container:

The live binary resolves, the dropped file comes out as an ELF executable, and the fake C2 address is read from the process's environment block.

Diff, export, commit, hash

The image ships read-only and never changes, so everything the container wrote sits in the thin writable layer on top. docker diff reads that layer directly, and a well-behaved app writes almost nothing, so a dropped binary and a new crontab stand out:

A is added, C is changed. Take the whole filesystem as a tarball for offline analysis, and commit a runnable snapshot you can detonate later in an isolated lab. docker save writes that snapshot to a file, so the evidence directory holds it too and does not depend on the image staying on this host:

The export holds the dropped binary and the new crontab. Both the export and the saved image are for a lab, never for re-running attacker code on a production host. Last, hash everything you took. A copied file is a copy; a copied file with a checksum recorded at collection time is something an auditor will still trust months later:

The subshell keeps your shell in ~/lab/forensics. The hashes vary from run to run (timestamps, PIDs and addresses differ); what matters is that SHA256SUMS is written now and travels with the files.

Memory needs CRIU, and the lab does not install it

Disk shows what was written down; a payload that lives only in RAM leaves nothing there. Capturing it means checkpointing the running process tree with CRIU, which docker checkpoint drives:

# experimental daemon + CRIU installed on the host
$ docker checkpoint create --leave-running suspicious cp-incident-42
$ docker checkpoint ls suspicious

docker checkpoint is experimental on Docker 29: the daemon must run in experimental mode, and CRIU must be installed on the host. CRIU supports x86_64, arm64 and other architectures; the lab VM simply does not install it, so this course does not run it. --checkpoint-dir chooses where the images land; without it they go under the container's directory in the daemon's data root, which docker rm deletes, so set it to your evidence storage. Without CRIU you stop at the filesystem, configuration, log and socket evidence above, which is enough to answer most of the case.

Investigate, then remediate

Lay the pieces side by side: docker diff and the recovered binary show what ran and what it dropped; the socket table and the environment show who it called; the logs and the inspect record show how it was started and what it printed; your host auditd or Falco trail shows when it arrived. Only then remediate, and remediate never means restart. Before removing anything, neutralise the restart policy so the daemon cannot bring a copy back over your evidence:

Reconstruct the timeline from the daemon's own event stream, then remove the container. The evidence directory is what survives:

docker events with --since and --until gives each state change with a Unix timestamp: the create and start, the pause, the docker cp (archive-path) and the export. It did not list the update or the commit here, so your own case notes with times are part of the record. Stream events to your log pipeline in production, because the daemon keeps only a short history and loses it on restart. After docker rm the container, its log file and its writable layer are gone, and the ten files in ir-42 are what is left. With the evidence off the box, patch the image or close the way in, rotate every credential the container could reach and treat them as burned, redeploy from a freshly built clean image, and turn the drop you found (a new binary under /tmp writing to /etc/crontab) into a Falco rule so the next attempt trips an alarm. Clean up:

Try this

Work through “Investigate, then remediate” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from container forensics and incident response, keep “Investigate, then remediate”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related