Container forensics & incident response
When a container is popped: contain, capture, investigate.
Prevention eventually fails, so the last advanced skill is what to do when a container is compromised. The instinct to immediately docker rm the container is exactly wrong — it destroys the evidence. Incident response for containers follows a deliberate order: contain the blast radius first, then capture state for analysis, then investigate, and only then remediate. Because containers are ephemeral and immutable by design, the forensic signal is unusually clean if you preserve it.
Contain without destroying
First, cut the container off without killing it: disconnect it from its networks (or move it to an isolated one) and pause it so its processes freeze in place, preserving memory and open connections for capture. docker pause uses the cgroup freezer, so the process is suspended mid-flight rather than terminated. Do not restart, and do not remove — a stopped container still holds its writable layer, but a paused one also holds live process state.
$ docker network disconnect appnet suspicious # sever its connectivity$ docker pause suspicious # freeze processes (do NOT kill)# the container is now isolated and frozen, ready to capture
Capture the evidence
The immutable-image model is a gift here: docker diff shows exactly what the container changed against its read-only image — every file an attacker dropped or modified stands out, because a well-behaved container changes almost nothing. Export the whole filesystem for offline analysis, save the process list and network state, and pull the logs. Each of these is a one-liner, and together they reconstruct what happened.
$ docker diff suspicious # every added/changed/deleted file (A/C/D)A /tmp/.x/minerC /etc/crontab$ docker export suspicious -o suspicious-fs.tar # full filesystem for offline analysis$ docker top suspicious # processes running inside$ docker logs suspicious > suspicious.log # captured output$ docker commit suspicious evidence:incident-42 # a snapshot image you can re-run in a lab
Investigate, then remediate the root cause
With the evidence captured, correlate: docker diff plus the process list tells you what ran; the logs and your Falco/audit trail tell you when and how it got in. The remediation is not “restart the container” — that ships the same vulnerable image straight back. Fix the root cause (patch the image, close the entry vector), rotate every credential the container could reach (it must be assumed compromised), and only then redeploy from a clean, rebuilt image. Feed what you learned back into a Falco rule so the next attempt trips an alarm.