Detecting escapes at runtime

The behavioral signatures and fast containment.

You bolted the front door. Someone can still walk around the house and test the windows. Runtime escape detection is the sound of that test: a mount command running inside a container, a process opening the container runtime's control socket, a write landing on a host path that no application should ever touch. Those noises are worth dropping everything for. They are a signal to go and look right now, not proof that the attacker already owns the machine.

Two words get thrown around loosely, so pin them down. Prevention removes the things that make an escape easy in the first place: privileged pods, host mounts, the runtime socket handed out as a volume. Detection covers what prevention could not, including chains nobody has seen before, capabilities you were not allowed to drop for business reasons, and kernel bugs with no patch yet. The pairing that works is a small number of high confidence rules wired straight to fast containment. Isolate the pod, cordon the node (mark it so the scheduler stops placing new work there), keep the evidence.

Signatures that describe behavior, not exploits

Escape kits rarely bring their own toolbox. They use what is already sitting in the image: python, openssl, a busybox wget. Security people call that living off the land, borrowing the tools the house already owns. A distroless image (one with no shell and no package manager, only your app and the libraries it needs) shrinks that toolbox and shrinks your alert noise at the same time, because any process starting at all becomes worth a second look.

If Tetragon (a runtime security tool built on eBPF, extended Berkeley Packet Filter, the kernel's safe in-kernel programming layer) kills the process first, make sure the event still reaches your SIEM (security information and event management system, the log pipeline your analysts search). Enforcement with no telemetry produces ghost incidents. Something died in the night and nobody can say what it was reaching for.

Run these commands on a staging cluster until they feel dull. The first time you read Falco output should not be the night a production pod opens the runtime socket.

An escape leaves a recognizable trail. A mount binary spawned inside a container. An open of docker.sock. A ptrace (process trace, the system call a debugger uses to read and steer another process) reaching across namespace boundaries. A sudden use of CAP_SYS_ADMIN, the capability that sits a short hop from full root on the host. A process running outside the control group, or cgroup, it was supposed to belong to. A write to release_agent under /proc/sys/kernel, an old cgroup feature that hands the kernel a program to run on the host. Falco ships rules for most of this. You still have to tune them to your own hostPath conventions, meaning the node directories your pods are genuinely allowed to mount.

- rule: Mount in container
  condition: spawned_process and container and proc.name = mount
  priority: WARNING
- rule: Runtime socket access
  condition: open_read and container and fd.name = /var/run/docker.sock
  priority: CRITICAL

Letting Tetragon stop the clearest steps

Some signals have no innocent reading. A pod in a non privileged tier opening the runtime socket is one of them. For those, a Tetragon TracingPolicy with a Sigkill action (the signal that ends a process immediately) stops the attempt inside the kernel while Falco pages a human with the surrounding context. Observe first still applies. Watch the rule in alert mode, confirm that no legitimate workload trips it, then switch the kill on. Escape paths tend to be the first rules that earn enforcement, because the cost of a false positive is one restarted pod and the cost of a miss is the node.

matchArgs:
- index: 0
  operator: Equal
  values: ["/var/run/docker.sock"]
matchActions:
- action: Sigkill

The containment playbook

When a Critical escape signal fires, order matters. Label the pod for quarantine. Apply a deny all NetworkPolicy (the Kubernetes rule that decides which pods may talk to which) so it can reach nothing else. Cordon the node if you suspect the host was already touched. Capture the Falco and Tetragon timeline plus the pod logs before you evict anything, because eviction deletes the exact thing you will want to read tomorrow morning. Notify the platform team and security leadership at the same time. This is a phone call, not a row in a ticket queue.

kubectl get pod build-agent-0 -n ci -o jsonpath="{.metadata.labels.quarantine}"
 
# example output:
true

Layer detection on top of prevention

Admission control takes the easy escapes off the table before a pod ever starts. Runtime detection covers whatever is left. The only way to know both halves work is to run the attack yourself. Mount the socket in staging, then check three things in order: did the alert fire, did containment actually happen, and did the event reach the SIEM inside your SLA (service level agreement, the response time you promised the business)?

Taming false positives in CI

CI (continuous integration, the automated build pipeline) really does touch sockets. Build agents mount things, run containers inside containers, and behave in ways that would be alarming anywhere else in the cluster. Scope your Critical rules to production labels, or exclude the specific build images by name. Do not switch the rule off across the whole cluster. A CI namespace with weak policy and a generous service account is precisely where an attacker would like to land.

falco --list=rules | grep -i mount
 
# example output:
- Mount launched in container
- Mount sensitive directory in container

Tabletops for escape alerts

Run a tabletop where you inject a Critical runtime socket access alert and time the response. How long until the node is cordoned? How long until the logs are preserved? How long before leadership hears about it? Walk through what Hubble (Cilium's network flow viewer) shows for that pod, what Tetragon already killed, and which service account token might now be in someone else's hands. Put a developer in the story who mounted the socket temporarily last quarter and never took it back out, because that teaches responders to chase the policy fix alongside the incident. Decide in advance who can authorize breaking glass to exec into a quarantined pod for live forensics, and when the answer is memory capture only.

kubectl get events -n prod --field-selector reason=FailedMount | tail -1
 
# example output:
Warning FailedMount pod/quarantine-demo ...

Your escape playbook should already name the forensic vendor and the disk snapshot API call for every cloud you run in. Looking those up while a container is reaching for the host burns the exact minutes you were trying to buy.

Field notes from real clusters

Prevention changes how often an attempt starts. Detection changes how long it runs before anybody notices. Admission can refuse a privileged pod today and still wave through the pod that was legitimately privileged for a storage driver, so the runtime rules stay armed either way.

Behavior ages better than exploit names. You cannot know tomorrow's bug, but you do know that a distroless API (application programming interface) pod should never call mount and never open /var/run/docker.sock. Write rules in the shape of "this workload identity should never do this". Rules keyed to a single CVE number (Common Vulnerabilities and Exposures, the public catalogue of known bugs) go stale the moment someone finds a new one.

Falco and Tetragon are happy together as long as their jobs are separate. Falco alerts broadly and brings context: the process, the pod, the rule that matched. Tetragon enforces narrowly on the loudest steps. Alert wide, kill thin.

Panic deletes feel decisive and destroy the case. Freeze or snapshot the pod filesystem if your platform allows it, pull the node logs, and resist the urge to remove the only copy of the evidence. Isolating the network identity stops the damage; deleting the pod only stops your ability to explain it.

An open of /var/run/containerd/containerd.sock from a pod that has no business seeing it is not a ticket for tomorrow. Isolate first, then ask whether it was a mis-deployed debug profile. Teams that decide to "wait for a second event" on socket access frequently spend that wait watching an attacker become root on the node. Put that severity in the rule metadata so the person on call does not have to improvise it at 3am.

The fields worth reading on an escape event: the process executable, the parent chain that spawned it, the pod's service account, the volume mounts from the pod spec, and whether the capabilities include SYS_ADMIN. Glue the runtime event to kubectl get pod -o yaml before you write "false positive" in the ticket. A good share of supposed false positives turn out to be real misconfigurations where the escape failed halfway, and those are still worth containing.

Classic misuse: a storage daemonset is noisy, so somebody disables the mount and pointer trace rules cluster wide. Scope the exception to that daemonset's labels instead. Escape paths love the gaps you carved for convenience. Second classic misuse: alerting only on known exploit tool names. Attackers rename binaries in seconds, while a mount call from a web pod stays suspicious for years.

Killing versus alerting is a real trade-off on suspected escape steps. A kill stops the breakout and can also take down a legitimate storage or security agent when your selectors are slightly wrong. There is a cost on the other side too: the broader the rule, the more syscalls you are watching on every node, and watching every file open is more expensive than watching one socket path. Start broad rules as alert plus rapid human isolation. Add the kill for the narrowest socket and mount patterns once your canary workloads have stayed clean for several weeks.

Line up every escape alert against the admission decisions from the same window. If that pod shape should never have been admitted, fix the gate and go talk to whoever owns the exception. Detection that never feeds back into prevention turns into a museum of the same recurring page. Make the feedback a required field on the incident form so it cannot quietly be skipped.

Watch out for alert packing. One breakout can generate ten related events, and ten separate pages will bury the responder rather than inform them. Deduplicate by pod uid (unique identifier) and rule family for a short window so on call sees one case instead of a storm. Storms lead to muting, and muting is how you miss the second stage.

Detection without a rehearsed handoff to IR (incident response, the process your team runs during a live security event) is expensive logging. Pair every high confidence escape rule with a runbook step your on call has actually practiced: cordon, snapshot, preserve the volatile data, then decide whether you destroy the pod or capture it.

Try this

Do this on a lab cluster or a single staging node. Read the output carefully. One look is not enough reason to go and change production policy.

$ kubectl -n falco logs ds/falco --tail=5 | grep -iE 'privileged|mount|escape'
04:55:01 Warning Launch Privileged Container (pod=attacker)
04:55:03 Warning Mount Kernel Module (pod=attacker)
$ kubectl get events -n default --field-selector reason=FailedMount | head -2
LAST SEEN   TYPE      REASON        OBJECT
12s         Warning   FailedMount   pod/escape-try

Takeaway

If you wire up one rule this week, make it the runtime socket open in production, routed as a page with a runbook link attached and containment steps that keep the evidence. The mount rules and the ptrace rules can follow once you have watched the first one behave for a couple of weeks.

Next up is incident response, where the alert you built has to turn into decisions under time pressure, and where you find out whether the loop from runtime alert back to admission policy actually closes.

Related