Sandboxed runtimes: gVisor & Kata

A kernel boundary for untrusted workloads.

Container escapes are rare, but when one lands it's expensive. A workload breaks out of its container, and because every container on a node shares the same host kernel, the attacker isn't sitting in the next container over. They're on the node. That means the kubelet's credentials, every service account token mounted on that node, and the secrets of every neighbor pod are suddenly in reach, and from there it's a short hop to the API server. seccomp (secure computing mode) and AppArmor lower the odds by filtering which system calls a container is allowed to make. Useful, but they're still filtering calls that run against that one shared kernel. You've narrowed the door. It's the same door. When a workload is untrusted enough that a shared kernel is a risk you can't accept, a multi-tenant platform or any job running code you didn't write yourself, that pod needs its own kernel boundary.

Think of a soundproof booth. Two people can share a room and agree to keep their voices down, or you can put one of them in a booth where it doesn't matter how loud they get, because the room never hears it. A sandboxed runtime is that booth. There are two mainstream ones, and they draw the line in different places. gVisor runs a user-space kernel called runsc that sits in front of the container and answers its system calls (syscalls) itself. The application's calls never touch the host directly; runsc handles them and makes only a small, tightly filtered set of real host calls on the workload's behalf. Kata Containers goes further and boots each pod inside a lightweight virtual machine (VM) with its own real, separate kernel, so the host kernel sits behind a hypervisor. You expose either one through a RuntimeClass and opt a pod in with a single field, runtimeClassName. Nothing else in the pod spec changes.

Opt a pod in, then prove it landed

A RuntimeClass is just a name that points at a handler your nodes already have installed. You create the class once. After that, any pod that sets runtimeClassName: gvisor gets started with runsc instead of the default runc. The handler name (runsc here) has to match the runtime that containerd or CRI-O was configured to expose; the RuntimeClass is only the Kubernetes-facing label for it. Here's the class and a pod that uses it.

apiVersion: node.k8s.io/v1
kind: RuntimeClass
metadata: { name: gvisor }
handler: runsc          # the gVisor handler, already installed on the nodes
---
apiVersion: v1
kind: Pod
metadata: { name: untrusted, namespace: sandbox }
spec:
  runtimeClassName: gvisor
  containers:
  - { name: app, image: registry.internal/untrusted:1.0 }

Applying the manifest is the easy part. What matters is whether the pod actually ended up in the sandbox or quietly fell back to runc. A pod can silently miss the sandbox if the class name is misspelled or a mutating webhook stripped the field, so you check rather than assume. Run two checks. Ask the API server which runtime class the pod is bound to, then look at the kernel from inside the pod, where gVisor announces itself by name. If the class name doesn't come back, or dmesg shows an ordinary host kernel instead of the gVisor banner, the pod is on runc and the sandbox you thought you had doesn't exist.

$ kubectl get pod untrusted -n sandbox -o jsonpath='{.spec.runtimeClassName}{"\n"}'
gvisor
$ kubectl exec -n sandbox untrusted -- dmesg | grep -i gvisor
[    0.000000] Starting gVisor...
$ kubectl exec -n sandbox untrusted -- uname -r
4.4.0

Make the sandbox mandatory

Setting runtimeClassName is opt-in, and opt-in security has a familiar weakness. People forget. A developer copies an older manifest, the field isn't there, and the untrusted job is back on the shared kernel with nobody watching. Pod Security Admission (PSA), the gatekeeper built into Kubernetes, won't catch this. Its levels cover things like privilege and host access, but it has no control for runtime class at all. So you add a policy engine. It works like a bouncer with a guest list: any pod trying to get into the untrusted namespace has to show a runtimeClassName of gvisor, or it's turned away at the door. Kyverno enforces that at admission, before the pod is ever persisted to etcd. Run the policy in Audit mode first to see what it would block without breaking deploys, then switch failureAction to Enforce once the namespace is clean. Lock down the RuntimeClass objects themselves with RBAC (Role-Based Access Control) too, so a tenant can't invent a class that points at plain runc and name it gvisor.

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata: { name: require-sandbox }
spec:
  rules:
  - name: sandbox-in-untrusted-ns
    match:
      any:
      - resources:
          kinds: [Pod]
          namespaceSelector:
            matchLabels: { trust: untrusted }
    validate:
      failureAction: Enforce
      message: "pods in an untrusted namespace must set runtimeClassName to gvisor"
      pattern:
        spec:
          runtimeClassName: gvisor

$ kubectl label ns sandbox trust=untrusted
namespace/sandbox labeled
$ kubectl apply -f require-sandbox.yaml
clusterpolicy.kyverno.io/require-sandbox created
$ kubectl run rogue --image=registry.internal/untrusted:1.0 -n sandbox
Error from server: admission webhook "validate.kyverno.svc-fail" denied the request:
 
resource Pod/sandbox/rogue was blocked due to the following policies
 
require-sandbox:
  sandbox-in-untrusted-ns: 'pods in an untrusted namespace must set
    runtimeClassName to gvisor'

The tradeoff

Isolation costs something. Both runtimes add latency, and both trade some compatibility for safety. gVisor implements most of the Linux syscall surface but not every corner of it, so an application that reaches for an exotic syscall, direct hardware access, or a niche filesystem feature can misbehave under runsc. Kata pays for a real VM per pod: more memory, a slower cold start. That's why you don't flip an entire cluster into a sandbox. You aim it at the workloads that earn it, the risky tenant or the build job running code you can't read. gVisor is the runtime behind GKE Sandbox on Google Kubernetes Engine (GKE); Kata shows up where you want VM-grade isolation on your own hardware. A common pattern is to give the runtime its own node pool and pin the RuntimeClass scheduling to those nodes, so the overhead stays where the risk actually is.

Sandboxes cost CPU and break some syscalls. Measure before you force them on latency-sensitive paths.

Admission should reject untrusted workloads that omit runtimeClassName. Optional sandboxes become unused sandboxes.

Sandboxing complements seccomp; it does not replace RBAC, NetworkPolicy, or secret hygiene.

Untrusted tenancy includes customer code, plugin marketplaces, and CI jobs that build arbitrary Dockerfiles. Those workloads get the sandbox RuntimeClass even if they are "internal." Internal is not the same as trusted. Write the verification command next to the control in the same pull request, and keep the sample output so the next on-call person can tell pass from fail without guessing.

Try this

Schedule a RuntimeClass gVisor or Kata pod in a lab and confirm the runtimeClassName and node labels match what you expect.

$ kubectl get runtimeclass
NAME     HANDLER   AGE
gvisor   runsc     3d
kata     kata      3d
 
$ kubectl -n untrusted apply -f - <<'EOF'
apiVersion: v1
kind: Pod
metadata: { name: sandboxed }
spec:
  runtimeClassName: gvisor
  containers:
  - name: c
    image: busybox:1.36
    command: ["sleep","3600"]
EOF
pod/sandboxed created
 
$ kubectl -n untrusted get pod sandboxed -o jsonpath='{.spec.runtimeClassName}{"\n"}{.status.phase}{"\n"}'
gvisor
Running

Takeaway

When a shared kernel is unacceptable, sandbox with gVisor or Kata via RuntimeClass and make it mandatory for untrusted namespaces.

Related