Container forensics & incident response

When a container is popped: contain, capture, investigate.

The reflex that ends an outage is the same reflex that ruins an investigation: kill the thing and let it come back clean. On an ordinary bad night that instinct is right. During a compromise it shreds the only record you have of how the attacker got in. A burglary is the better comparison, and the first officer through the door does not start tidying up. A container keeps its evidence in three places, and the quick fixes wipe all three in seconds. Restart it and you lose the live process state the kernel is holding in memory, plus every open socket the attacker was talking through. Run docker rm and you also lose the writable layer, the thin scratch space stacked on top of the read-only image where everything the container changed while it ran actually lives. In the first five minutes your job is preservation, not repair. Do not walk on the scene.

Work in one fixed order and never skip ahead: contain, capture, investigate, remediate. Teams that jump straight to remediate redeploy the same vulnerable image an hour later, having learned nothing about the way in. The good news is that containers make this easier than a bare host does. The image ships read-only and never changes. Everything the running container writes lands in a thin copy-on-write layer on top (copy-on-write = the container gets its own private copy of a file only at the moment it changes that file). Treat the image as the before photo and that top layer as the after. A well-behaved app writes almost nothing while it runs, so a dropped miner or an edited crontab (crontab = the file that tells Linux which commands to run on a schedule) stands out like footprints across fresh snow.

Contain: freeze it, do not fix it

Cut the network first. That stops the attacker pulling down a second stage and stops your data leaving while you work. Next, write down the host-side PID (PID = process ID, the number the kernel gives the container's main process on the host, which is not the 1 that process sees inside its own namespace). You will need that number in a minute to read the kernel's view of that exact process. Then pause. Freeze-frame a film and every actor stops mid-step, and nobody on screen notices that time stopped. docker pause does that to a container, through the cgroup freezer (cgroup = control group, the kernel feature that tracks and controls a set of processes as one unit). The freezer halts every thread at once, including threads wedged in an uninterruptible sleep, and it acts underneath the process, so the attacker cannot catch it or fire off an anti-forensics wipe the way they could catch an ordinary stop signal. One catch to remember: a paused container refuses docker exec, so everything you collect from here comes through the host, never from inside.

# 1. sever command-and-control / exfil, but leave the process alive
$ docker network disconnect appnet suspicious
 
# 2. record the host-side PID before anything can change
$ PID=$(docker inspect -f '{{.State.Pid}}' suspicious); echo "$PID"
48213
 
# 3. freeze every thread in place via the cgroup freezer (this is NOT a kill)
$ docker pause suspicious
suspicious
$ docker inspect -f '{{.State.Status}}' suspicious
paused

Capture the live state from the host, not from inside

Question a witness the burglar has already coached and you get the answer the burglar wants. That is what the container's own ps, netstat and ls are once someone owns the container, because owning the container means owning its userspace. Those binaries may have been swapped for versions that hide the attacker's process, or the attacker may have deleted their own binary one second after launching it. So ask the kernel instead. /proc is the process filesystem, a live view the kernel keeps for itself, and nothing inside the container can forge it. From the host, as root, /proc/<pid>/ gives you the truth about that process. The exe symlink still points at the payload even after the attacker unlinked it, because Linux keeps a deleted file on disk until the last open handle closes. environ shows the config values and keys the running process is holding in memory right now. fd lists every open file and socket, which is how you learn who it was calling. Collect all of it while the process is still alive, because the memory dump in the next step ends it.

# the process's own binary: attacker ran it, then unlinked it. kernel still has it.
$ sudo ls -l /proc/$PID/exe
lrwxrwxrwx 1 root root 0 Jul 16 09:12 /proc/48213/exe -> '/tmp/.x/miner (deleted)'
 
# recover the unlinked payload through the exe link; the kernel still holds the inode
$ sudo cp /proc/$PID/exe /evidence/miner.bin
 
# config and keys the running process is holding right now
$ sudo tr '\0' '\n' < /proc/$PID/environ | grep -Ei 'key|token|c2'
C2_HOST=185.220.101.47
 
# open sockets: who it was talking to
$ sudo ls -l /proc/$PID/fd | grep socket
lrwx------ 1 root root 64 Jul 16 09:12 7 -> 'socket:[91422]'

Capture memory, then the filesystem

Disk is the paper trail. It shows you what got written down, and it cannot show you a conversation that was only ever spoken out loud. A payload that lives purely in RAM is that conversation (RAM = the working memory a process uses while it runs, wiped the moment the process dies). To keep it you take a checkpoint. docker checkpoint drives CRIU (Checkpoint/Restore In Userspace, a tool that freeze-dries a running process tree onto disk) and writes the whole tree out, memory pages included. In-memory-only implants, decrypted keys and injected code show up there and nowhere else. Fair warning: checkpoint is still an experimental feature in Docker 27.x, so the daemon needs experimental mode turned on and CRIU installed on the host, and some workloads will not checkpoint at all. The default checkpoint stops the container once it finishes, which is fine at this point. You already cut the network, so a brief unpause to run the dump cannot leak anything. The stopped container still answers docker diff and docker export, so you can read the writable layer and pull the full root filesystem into a tarball for offline work in a lab.

# MEMORY image via CRIU. checkpoint does its own atomic freeze, so unpause first.
$ docker unpause suspicious && docker checkpoint create suspicious cp-incident-42
suspicious
cp-incident-42
$ CID=$(docker inspect -f '{{.Id}}' suspicious)
$ sudo ls /var/lib/docker/containers/$CID/checkpoints/cp-incident-42/
core-48213.img  fdinfo-2.img  mm-48213.img  pagemap-48213.img  pages-1.img

# writable layer vs the read-only image: A added, C changed, D deleted
$ docker diff suspicious
A /tmp/.x
A /tmp/.x/miner
C /etc/crontab
C /root/.ssh/authorized_keys
 
# whole root filesystem as a tarball for the lab (never run attacker code on prod)
$ docker export suspicious -o /evidence/suspicious-rootfs.tar
$ tar tf /evidence/suspicious-rootfs.tar | wc -l
14877
 
# a runnable snapshot image you can detonate safely in an isolated lab
$ docker commit suspicious evidence:incident-42
sha256:9b2f0c7a1e34d8...

Investigate, then kill the root cause

Now lay the pieces side by side. docker diff and the recovered binary tell you what ran and what it dropped. The checkpoint memory and the open sockets tell you what it was holding and who it was calling. Your host auditd log (auditd = the Linux audit daemon, which records system calls and file access) and your Falco trail (Falco watches system calls as they happen and alerts on the suspicious ones) tell you when it arrived and how. Only after that do you remediate, and remediate never means restart. Patch the image or close the way in, rotate every credential that container could reach and treat all of them as burned, then redeploy from a freshly built clean image. Turn the drop you found into a Falco rule so the next attempt trips an alarm on its first move. The pattern sitting right there in your docker diff output, a new binary under /tmp writing to /etc/crontab, is exactly what a runtime rule catches live the next time round.

# BEFORE you delete anything, check what would resurrect the scene
$ docker inspect -f '{{.HostConfig.RestartPolicy.Name}}' suspicious
unless-stopped
 
# neutralize the policy so the daemon can't bring back a clean copy over your evidence
$ docker update --restart=no suspicious
suspicious
 
# evidence is captured (/proc, checkpoint, diff, export, commit). now retire the original.
$ docker rm -f suspicious
suspicious

Here is the short version for your runbook. Isolate first: quarantine the network, then pause. Preserve next: commit and export, plus a memory image if the case needs one. Investigate after that, without walking on what you collected. Do not reach for docker rm -f until everything you need is already off the box.

Collect the boring metadata too, because it answers the questions people ask you a week later. The docker inspect output, the container logs, the image digest, every mounted volume, and the other containers sharing that network. Line all of it up against the host timeline from auditd, sysdig or your eBPF tooling (eBPF = a kernel feature that lets a tool watch system calls safely from inside the kernel), and the order of events stops being guesswork.

Rehearse this on a lab compromise before you ever need it for real. Somebody reading a runbook for the first time at 3am, with a manager standing behind them, makes exactly the mistake this lesson opens with. Talk to legal and compliance early too, because a real case may need chain of custody on those disk images, meaning a written record of who held each copy and when.

Copy the evidence somewhere the incident cannot reach, not onto the same disk you are about to rebuild. Record a checksum for every file as you take it, along with the time, in the ticket. A copied file is a copy. A copied file with a recorded checksum is something an auditor, or a court, will still trust six months from now.

Keep a running log while you work: one line per command, the time you ran it, and what came back. You will not remember the order afterwards, and the order is often the whole answer. That log is also what protects you when somebody asks why the container vanished at 09:14.

Once the incident closes, run the same checks as a routine drill after every change window. Confirm the controls you added are still on, paste the command and its output into the ticket, and refuse to sign the change off if the reading has drifted. Guardrails rot quietly, and a five minute check is what catches that.

Then give each workload the tightest scope it can still run with, and keep it there. Fewer capabilities, no docker socket mounted in, a read-only root filesystem wherever the app tolerates one. That habit will not stop every compromise, but it decides how far an attacker gets after the first one and how much of the story you can still read afterwards. It compounds across every host and every pipeline you own.

Try this

Run the containment sequence on something harmless long before you run it on a real incident. Pause a container, pull its logs and its inspect JSON, export the filesystem to a tarball, and only then remove it. Watch the order closely, because the order is the entire skill.

$ docker run -d --name suspect alpine sh -c 'while true; do echo beat; sleep 5; done'
$ docker pause suspect
$ docker logs suspect | tail -5
beat
beat
$ docker inspect suspect > /tmp/suspect-inspect.json
$ docker export suspect > /tmp/suspect-fs.tar
$ ls -l /tmp/suspect-inspect.json /tmp/suspect-fs.tar
-rw-r--r-- ... /tmp/suspect-inspect.json
-rw-r--r-- ... /tmp/suspect-fs.tar
$ docker rm -f suspect

Takeaway

Pause before you destroy, and preserve before you remediate. Capture the inspect output, the logs, the image digest and a filesystem export, and you can answer the three questions that decide the whole case: what ran, with what privileges, and what it touched.

Related