Sandboxes & runtime detection

gVisor, Kata, and catching what still gets through.

In February 2019 someone published CVE-2019-5736, and a single malicious image became a way to own an entire machine. (CVE stands for Common Vulnerabilities and Exposures, the public catalog of known security bugs; every entry gets its own ID number.) The trick was small and nasty. A container overwrote the host's own runc binary through /proc/self/exe, runc being the low-level program that actually starts every container. The next time an admin ran docker exec into that container, the host executed the attacker's code as real root. Every container on that box was sitting on one kernel and one runtime, so a bug in either one left the whole host a single exploit away. A sandboxed runtime is what you reach for when you want to pull that shared kernel out from under the container.

The furnace everyone shares

A normal container is a set of rooms in one big house. Namespaces keep the tenants apart (a namespace is a per-process view of the system, a one-way mirror that lets each tenant see only their own room). Every room still draws its heat from the same furnace, and that furnace is the host kernel. Anything a container genuinely does, opening a file, sending a packet, reading the clock, goes out as a system call (a syscall, the request a program makes to the kernel to do work on its behalf). That request runs code down in the shared furnace. Crack the furnace and you are not in a room any more. You are in the basement of the whole house. That is why "container escape" and "kernel exploit" so often turn out to be the same sentence.

Give the container its own kernel

gVisor and Kata Containers give the same answer: stop letting the container's syscalls reach the host kernel at all. Where they part ways is how far they take it.

gVisor slides a program called the Sentry in between the container and the host. Think of a hotel concierge who takes every request at the front desk and handles almost all of them personally, so guests never wander into the boiler room. The Sentry is a user-space kernel: an ordinary process that re-implements the Linux system-call interface itself. When your container calls open() or socket(), the Sentry answers, not the host. Only a small, fixed set of syscalls ever reaches the real kernel, and the Sentry is penned in by a tight seccomp filter of its own (seccomp is secure computing mode, a bouncer working from a written allow-list; syscalls that are not on the list do not get through). The runtime binary that wires all of this up is called runsc. You end up with two boundaries stacked: container to Sentry, then Sentry to host through a locked door.

Kata Containers goes heavier. It wraps each container or pod in a real, lightweight virtual machine with its own guest kernel, booted in milliseconds by a stripped-down hypervisor (the software that runs virtual machines). The container talks to that guest kernel, and the guest kernel is what the host keeps at arm's length. An attacker now has to break out of the container, then break out of a VM, before the host is even in sight. You pick Kata the same way you pick runc, because as far as Docker is concerned it is one more runtime name.

# which runtimes can this host launch?
$ docker info --format '{{.Runtimes}}'
map[io.containerd.runc.v2:{...} kata:{...} runc:{...} runsc:{...}]
 
# run a workload under gVisor instead of runc
$ docker run --rm --runtime=runsc alpine dmesg | head -3
[    0.000000] Starting gVisor...
[    0.216633] Preparing for the zombie uprising...
[    0.464884] Generating random numbers by fair dice roll...

# runc shares the host kernel, so the container sees the host's version
$ docker run --rm alpine uname -r
6.8.0-45-generic
 
# runsc: the container talks to the Sentry, which reports its own fixed version
$ docker run --rm --runtime=runsc alpine uname -r
4.19.0-gvisor
 
# kata: a real, separate guest kernel booted inside a micro-VM
$ docker run --rm --runtime=kata alpine uname -r
6.1.62

Three different kernels, one alpine image. That single field says the whole thing out loud. Under runc, a kernel-level escape lands on the host. Under runsc it lands inside the Sentry, a user-space process that still has to beat its own seccomp filter before it can touch anything real. Under Kata it lands inside a throwaway virtual machine you can delete. Each step shrinks the blast radius.

When prevention fails, watch the behavior

A sandbox shrinks the attack surface. It does not make you all-seeing. Somebody will eventually find a bug in the Sentry. Or you will keep a trusted service on plain runc because the sandbox broke it, and one day something gets in anyway. So you also watch for the behaviors an escape produces, live, and you alert the second one of them shows up.

Falco is the tool most teams use for that. It loads a probe into the kernel (a modern eBPF program, extended Berkeley Packet Filter, safe sandboxed code that the kernel runs on your behalf) and streams the syscalls you care about to a user-space engine that checks them against rules. Here is why it lands so hard on containers you have already hardened: the behaviors worth alerting on are behaviors that should be impossible. A distroless, non-root, read-only container ships no shell at all. So a shell starting inside one is an alarm, not noise. Every control you added earlier in this course doubles as a tripwire.

# a running production container that ships no shell of its own
$ docker run -d --name payments nginx
 
# someone gets in and spawns a shell (the classic first move after an escape)
$ docker exec -it payments bash
 
# Falco, tapping the kernel from the host, fires immediately:
$ sudo tail -f /var/log/falco/falco.log
15:42:07.881644229: Notice A shell was spawned in a container with an attached
 terminal (user=root user_loginuid=-1 container_id=9f3c2b1a7e44
 container_name=payments image=nginx:latest shell=bash parent=runc
 cmdline=bash terminal=34816) container_id=9f3c2b1a7e44

Pair every technique with its own fix and its own detection. Take the docker.sock pivot from earlier in this course: a container with the Docker socket bind-mounted into it can drive the host daemon and ask for a privileged container, which is an escape that never touches the kernel. The fix is to never mount the socket, and to run read-only. The detection is a rule that fires the moment anything inside a container so much as opens that path.

# add a rule that fires when a container opens the Docker socket
$ cat /etc/falco/rules.d/docker-sock.yaml
- rule: Docker socket opened inside a container
  desc: A container process opened /var/run/docker.sock (common escape pivot)
  condition: open_read and container and fd.name=/var/run/docker.sock
  output: >
    docker.sock opened (cmd=%proc.cmdline container=%container.name
    image=%container.image.repository)
  priority: CRITICAL
  tags: [container, filesystem, mitre_privilege_escalation]
 
$ sudo systemctl reload falco
 
# attacker reaches the socket that was carelessly mounted into 'app'
$ docker exec app cat /var/run/docker.sock >/dev/null 2>&1
 
$ sudo tail -n1 /var/log/falco/falco.log
15:48:12.004551123: Critical docker.sock opened (cmd=cat /var/run/docker.sock
 container=app image=library/myapp)

The bill for a sandbox arrives as performance and compatibility. gVisor adds latency to syscall-heavy work, and it returns ENOSYS (the kernel's "function not implemented" error) for syscalls it has not built yet, so a database reaching for io_uring, or a program poking at an obscure /proc file, can fail to start at all. Kata spends memory and boot time on a virtual machine per workload. So treat a sandbox as a targeted tool for code you did not write and cannot vouch for: customer-supplied builds, continuous integration jobs (CI, the pipeline that builds and tests every proposed change) running arbitrary pull requests, multi-tenant functions. For your own services, the controls from earlier lessons are the floor: minimal image, non-root, dropped capabilities, seccomp, read-only root filesystem. A sandbox is what you add when "shared kernel" is a line you cannot afford to have crossed, and Falco is how you find out the day something tries to cross it.

Rolling one out has an order to it. Find a workload where isolation matters more than the last few percent of throughput, run it under runsc in a lab, and read the logs for ENOSYS. That error is the sandbox telling you exactly which syscall the workload wanted and did not get. If the failures sit in a library you can swap out, the workload is a candidate. If they sit in the storage engine, it is not, and Kata is the better answer there, because a real guest kernel implements the whole syscall set instead of an emulated slice of it.

Detection deserves the same care. Falco ships useful defaults, and defaults on their own will not cover you. Write down the handful of things that should never happen inside your containers, then make sure a rule exists for each: a shell exec in a distroless image, a read of /etc/shadow on a host mount, an outbound connection to an address nobody has ever contacted before, a write into a bin directory. Then perform each behavior yourself in a lab and watch the alert land. An untested rule is a comfortable assumption dressed up as a control.

None of this replaces the boring work. A sandbox around a container that still runs as root, holds every capability, and has the Docker socket mounted into it is a stronger fence around a wide-open gate. Drop the capabilities, mount no socket, run non-root and read-only, and then, for the workloads that earn it, put a separate kernel underneath. The order matters, because the cheap controls stop the attacks that actually happen every week and the expensive one stops the rare attack that would otherwise end you.

After a change window, run the same check you ran the first time. Confirm the runtime is still what you set it to, confirm Falco is still loaded and its rules are still in place, paste the command and its output into the ticket, and refuse to close the change if the reading moved. Prefer the tightest scope the workload can still run under. That habit compounds across every host and every pipeline you own.

Try this

Ask the daemon which runtimes it can launch, then check what a running container actually got. If runc is the only name on the list, write down what trialling runsc or Kata in a lab would take: the package to install, the daemon config change, and the one workload you would move first.

$ docker info --format '{{range .Runtimes}}{{.}} {{end}}'
runc io.containerd.runc.v2
$ docker run -d --name rt alpine sleep 60 >/dev/null
$ docker inspect -f 'Runtime={{.HostConfig.Runtime}} Privileged={{.HostConfig.Privileged}}' rt
Runtime=runc Privileged=false
$ docker rm -f rt
$ # next: install runsc/kata in lab and re-run with --runtime=runsc

Takeaway

When a shared kernel is a risk you cannot accept, move that workload to gVisor or Kata and keep runtime detection running underneath it. A sandbox raises the price of an escape. Least privilege and socket hygiene still do the everyday work of making sure nobody has to pay it.

Related