Capabilities, namespaces & cgroups

Drop ALL, non-root, and the isolation primitives.

A submarine survives a hull breach because it is chopped into sealed compartments. Water fills one room, the crew shuts the hatches, the boat stays up. Leave one hatch swinging and the whole vessel floods. A Linux container is built on the same idea, out of namespaces, capabilities and cgroups (control groups). Hardening one means shutting the hatches an attacker expects to find open.

Three words in plain English before anything else. A container is an ordinary Linux process that has been handed a restricted view of the machine. Namespaces decide what it can see: its own process list, its own network, its own filesystem tree. Capabilities are the pieces that root's power gets chopped into. cgroups cap how much CPU and memory it gets, and how many processes it may spawn. You close the hatches first. Runtime detection is what catches somebody prying at them anyway.

Capabilities: root chopped into separate levers

A locksmith holding a master key does not care which locks you fitted. Ephemeral debug containers and kubectl debug work the same way. They can hand back privileges you carefully stripped from the main container. Decide up front who may attach a debug container to production, and which security profile those sessions inherit. An unrestricted debug attach is a privileged pod wearing a shorter name. Practise the commands on staging until they feel boring. Boring is reliable.

Old-school root is one giant bundle of powers: CAP_SYS_ADMIN, CAP_NET_RAW, CAP_SYS_PTRACE, CAP_DAC_READ_SEARCH and dozens more. Drop ALL of them at the pod level, then add back only what the workload can prove it needs. CAP_SYS_ADMIN is the one to fear. It re-enables mounting filesystems and playing games with namespaces, the two moves at the heart of most container escapes, so treat a request for it like a request for the master key. Then audit who in your cluster's RBAC (role-based access control, the rules for who may do what) can create pods with extra capabilities. Admission control should refuse the dangerous set outright in production.

securityContext:
  runAsNonRoot: true
  runAsUser: 10001
  allowPrivilegeEscalation: false
  readOnlyRootFilesystem: true
  capabilities:
    drop: ["ALL"]
    add: ["NET_BIND_SERVICE"]
  seccompProfile: { type: RuntimeDefault }

Namespaces, and the user namespace question

cgroups: limits that also slow an attacker down

cgroups put a ceiling on CPU, memory and the number of processes a container may spawn, which blunts fork bombs and memory exhaustion. A low pids.max stops the rapid fork chains some denial-of-service tricks rely on. Pair that with readOnlyRootFilesystem and an attacker cannot drop a binary onto the image layer and expect it to stay there. The only writable spots left are volumes you mounted on purpose. Mount those narrowly, and put runtime rules on writes to them.

kubectl describe pod api-7c9 -n prod | grep -A3 "Limits:"
 
# example output:
Limits:
  cpu:     2
  memory:  1Gi
  ephemeral-storage:  2Gi

privileged and hostPath: the hatches that sink the boat

privileged: true hands the container every capability and nearly every device on the node. At that point it is a host process with extra steps. hostPath mounts of /, /var/lib/kubelet, or the container runtime socket tell the same story: whoever controls that pod controls the node. Admission policy should refuse all of it in production namespaces. The repeat offenders are CI (continuous integration) and build pods that mount docker.sock "temporarily". Treat those with the same seriousness as production.

kubectl get pods -A -o json | jq -r '.items[] | select(.spec.containers[].securityContext.privileged==true) | .metadata.namespace+"/"+.metadata.name' | head -3
 
# example output:
build/jenkins-agent-4b
dev/docker-builder-0

When the code you run is not yours

Sometimes the workload is code you did not write and cannot vouch for: multi-tenant SaaS (software as a service, many customers sharing one set of machines), or containers your users upload themselves. That is where gVisor and Kata Containers earn their keep. gVisor catches syscalls (the requests a program makes to the kernel) inside a kernel of its own written in user space. Kata boots a small virtual machine per pod. Both shrink the amount of shared kernel an escape can reach, and both cost you performance. For ordinary internal apps, standard runc (the default low-level container runtime) plus strict admission is still the right answer. Raise the isolation tier for workloads that earn it, not for the whole fleet by reflex.

# restricted forbids: privileged, host namespaces, hostPath volumes,
# unsafe caps, runAsNonRoot required, seccomp RuntimeDefault required.

Namespace and capability hardening belongs in the paved road, not in a wiki page nobody opens. Your internal starter Helm charts should default to runAsNonRoot, drop ALL, readOnlyRootFS true and no hostPath, so a team has to write an explicit override to weaken any of it. Put the share of pods still matching that default on the same dashboard as your patch SLAs (service level agreements, the deadlines you commit to for fixing things), where leadership already looks. Every deviation gets a ticket link in the values.yaml override, so the next person reading Git blame finds out why.

Catching drift before an attacker does

Manifests rot. A weekly report listing pods with allowPrivilegeEscalation set true, no runAsNonRoot, or added capabilities finds the slippage while it is still cheap to fix. Include CronJobs and Helm hooks, because they run real containers and usually dodge the review that normal deployments get. Then pair the report with Falco rules on privilege changes inside running containers, so you also catch tampering that no static manifest could have predicted.

kubectl get pods -A -o json | jq "[.items[] | select(.spec.containers[].securityContext.allowPrivilegeEscalation!=false)] | length"
 
# example output:
7

Every pod on that list needs one of two answers: fix it, or write down the compensating controls (its own namespace, a deny-all network policy, a read-only workload). Risk you wrote down is risk somebody can argue with later. Risk you forgot about is the one that surprises you.

Notes from real clusters

A submarine stays dry because the hatches, the seals and the pressure doors all hold at the same time. A container stays contained for the same reason. Namespaces, cgroups, capabilities and MAC (mandatory access control, kernel-level rules such as AppArmor or SELinux) all have to agree. Leave one hatch open, privileged, a hostPath to /, or the Docker socket, and every other control you built becomes scenery on the attacker's tour.

Capabilities chop "root" into separate levers: CAP_NET_ADMIN for reconfiguring the network, CAP_SYS_PTRACE for attaching to other processes, CAP_SYS_ADMIN for nearly everything else. Dropping ALL and adding back only what you need is the modern default, and plenty of apps need nothing added at all. If a chart ships with privileged: true because that was easier, file it as a production bug even though the app runs fine.

Namespaces carve up the container's view of the machine: mount, pid, network, uts, ipc and user. The user namespace is the interesting one. It maps uid 0 (user ID zero, root) inside the container to a boring unprivileged user on the host, so "root in the container" stops meaning "root on the node". Where the kernel and runtime support it properly, that is a serious escape mitigator. Go and find out whether your cluster has it switched on. Vendor slides and node reality drift apart.

cgroups bound CPU, memory and sometimes the process count. Treat them as brakes as much as walls; they are denial-of-service protection first and a security control second. A fork bomb in a pod with no PID limit is a noisy neighbour that can also starve the very agents you rely on to notice it. Keep the limits realistic, and watch for pods that quietly delete them during a hotfix.

Privileged pods and hostPath mounts are the two settings that undo everything else. Privileged reopens nearly every capability and device path on the node. A hostPath into a sensitive directory lets the container rewrite the node's own state. The container runtime socket is a remote control for the machine. Admission should deny all three by default, with an exception process that is narrow and that expires on a date somebody wrote down.

Harder tiers exist for hostile code: gVisor, Kata Containers, Firecracker-style micro virtual machines. Reach for them when your tenants distrust each other, or when you run plugins that strangers wrote. Do not talk yourself into treating a default runc container as a virtual machine; it is not one. Label workloads by trust tier so the scheduler and your policies can place them correctly.

Audit drift on a schedule. A weekly job that lists privileged pods, hostPath volumes and CAP_SYS_ADMIN additions catches the YAML (the text format Kubernetes configuration is written in) that slipped through during a 2 a.m. hotfix. Publish the list somewhere platform and security both read. That list is what prevents the "wait, I thought we banned that" conversation in the middle of an incident.

One real failure worth carrying around. A debug chart was copied into production still carrying privileged: true and a hostPath to /var/run/containerd. Nobody looked twice, because the namespace was called "tools". Then somebody phished a deploy token, found that chart, and climbed it straight to the node. The postmortem had nothing to do with a clever new CVE (Common Vulnerabilities and Exposures, the public ID given to a known bug). It was about admission exceptions with no expiry date, and a namespace everybody assumed was harmless. Isolation policy has to cover the boring namespaces too.

Reading a pod securityContext out loud is an underrated skill, in interviews and in review. Scan for privileged, allowPrivilegeEscalation, capabilities.add, hostPID, hostNetwork, hostIPC, and any hostPath volumes. One bad field cancels four good ones. When you review a PR (pull request, the change somebody wants merged), comment on the combination rather than field by field. "Runs as non-root, but adds CAP_SYS_ADMIN" is still a red flag.

A common way to get this wrong is stamp collecting. Seccomp profile, tick. AppArmor, tick. Drop ALL capabilities, tick. Meanwhile the pod mounts the cloud metadata path and carries a service account token with half the cluster's permissions. The process jail looks beautiful and the identity still owns everything. Put short-lived tokens, narrow RBAC and default-deny networking alongside your Linux isolation, or you spent all that effort on the wrong boundary.

There is a genuine trade-off with user namespaces and nested isolation. They make escapes much harder, and they break some privileged helpers: certain CSI (Container Storage Interface, the plugin system for storage drivers) drivers, eBPF loaders, older runtimes. Check your runtime matrix before promising user namespaces across the whole cluster. Enabling it for clearly labelled workloads first beats flipping a fleet-wide switch and spending the next week rolling it back.

Explaining this to a product owner? Skip the namespace lecture and hand them a building. Admission is the front door. Namespaces and cgroups are the walls of the room. Capabilities, seccomp and MAC are the kinds of lock on each door. Falco or Tetragon are the cameras. Then ask which door somebody propped open for a demo and never came back to close. Most "isolation failures" are propped doors, not broken walls. Put the propped-door list in sprint review where people see it, instead of leaving it buried in YAML.

Two more places to look. hostProcess pods on Windows nodes, and the privileged DaemonSets that get waved through because they are "platform owned". Platform-owned does not mean risk-free. Inventory them, write down why each one needs what it has, and monitor them harder than anything else you run. Attackers read your DaemonSets too.

When somebody does land inside a pod, isolation decides whether you have a bad afternoon or a cluster-wide scramble. Rehearse break-glass access in a non-production namespace, so that at 3 a.m. your on-call already knows which of these controls are real walls and which are only warnings.

Try this

Run these on a lab cluster or a single staging node, never straight at production. Read what comes back before you change any policy.

$ kubectl get pod hardened -o jsonpath='{.status.containerStatuses[0].ready}{"\n"}'
true
$ kubectl exec hardened -- cat /proc/1/cgroup | head -2
0::/kubepods.slice/kubepods-pod123.scope
$ kubectl auth can-i list pods --as=system:serviceaccount:default:default
no

Takeaway

Isolation is a stack, and the weakest layer sets its height. privileged, a hostPath into the wrong directory, or a mounted runtime socket flattens capabilities, namespaces, cgroups and MAC all at once. That is why the drift audit matters more than the day you first hardened the chart.

Next: find one workload in staging that runs privileged for no good reason and take it away. Then move on to Falco, so you can watch what the workloads you left alone actually do.

Related