seccomp syscall filtering

RuntimeDefault and custom default-deny profiles.

A hotel key card opens your room and the lobby. It does not open the electrical closet, the roof hatch, or the safe behind the front desk. seccomp (secure computing mode, the Linux feature that decides which system calls a program is allowed to make) is that key card for a container. A system call, or syscall, is how a program asks the kernel to do real work: open a file, start a process, mount a disk. The container gets the doors it needs. The service entrances stay locked.

Three terms you will meet all lesson, in plain words. A seccomp profile is the list of syscalls a container may use, checked on every single call, which is how calls like mount and ptrace (the debugger syscall that lets one process read and steer another) get refused. RuntimeDefault is the curated profile Kubernetes asks the container runtime to apply. A Localhost profile is a JSON file (JavaScript Object Notation, the plain-text format the runtime reads) that you wrote yourself and placed on the node. Writing a profile is the easy half. Rolling it out without breaking someone's Tuesday is the other half.

seccomp-BPF: a checkpoint on every syscall

seccomp loads a small BPF program (Berkeley Packet Filter, a tiny sandboxed program the kernel runs on your behalf) that sees each syscall and returns a verdict: allow it, allow it and write an audit line, fail it with an error code (errno), kill the process, or hand the decision to a supervising process outside the container. A current x86_64 kernel offers well over four hundred syscalls. Most containers touch a few dozen. RuntimeDefault blocks the known-dangerous end of that list, leaves ordinary server work alone, and is maintained alongside the container runtime, so you inherit the updates. Plenty of clusters still run seccompProfile: Unconfined, which hands an attacker the whole catalogue.

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  securityContext:
    seccompProfile:
      type: RuntimeDefault
  containers:
  - name: app
    image: registry/acme/api:1.4.2

Find out what the workload actually calls

Before you tighten anything, watch the app under traffic that looks like production. The Security Profiles Operator (SPO, a Kubernetes add-on that records syscalls from live pods and writes the profile for you) does this properly. strace (a tool that prints every syscall a process makes) and eBPF (extended Berkeley Packet Filter) tracers give you a quick ad hoc view. A default-deny profile allows only what you recorded, which is a strong control on a high-value tier and a maintenance bill on any app that spawns helper processes once a quarter. Put RuntimeDefault everywhere first, through admission policy, then read what it already permits before you assume you can beat it. The number comes back in the hundreds: the runtime default names most of the syscall table as allowed and refuses the rest through its default action, and the rest is where mount, bpf, keyctl and kexec_load sit.

crictl inspect $(crictl ps -q --name api | head -1) | jq '.info.runtimeSpec.linux.seccomp
  | {defaultAction, architectures,
     allowed: ([.syscalls[] | select(.action == "SCMP_ACT_ALLOW") | .names[]] | length)}'
 
# example output (containerd RuntimeDefault; the count moves with the runtime version):
{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": [
    "SCMP_ARCH_X86_64",
    "SCMP_ARCH_X86",
    "SCMP_ARCH_X32"
  ],
  "allowed": 347
}

Writing a Localhost profile by hand

A Localhost profile is two pieces that have to line up: a JSON file sitting on the node, and a field in the pod that points at it. The kubelet resolves that pointer under its own seccomp directory, which is the kubelet root directory plus /seccomp, so /var/lib/kubelet/seccomp on a stock install. The path you write has to be relative, so localhostProfile: profiles/api-v1.json means the file must exist at /var/lib/kubelet/seccomp/profiles/api-v1.json on every node the pod could land on. Getting it there is your job, not the cluster's: bake it into the node image, run a DaemonSet that writes it through a hostPath mount, or let the Security Profiles Operator ship it for you. Miss one node and pods scheduled there sit in CreateContainerError, because the runtime will not start a container whose profile file it cannot read.

{
  "defaultAction": "SCMP_ACT_ERRNO",
  "architectures": ["SCMP_ARCH_X86_64", "SCMP_ARCH_X86", "SCMP_ARCH_X32"],
  "syscalls": [
    {
      "names": ["read","write","openat","close","mmap","brk","futex","connect","accept4"],
      "action": "SCMP_ACT_ALLOW"
    }
  ]
}

The architectures list is not decoration. Syscall numbers differ per architecture, so a filter built for x86_64 alone has nothing to say about the same call arriving through the 32-bit or x32 ABI (application binary interface, the calling convention a binary uses when it enters the kernel). With SCMP_ACT_ERRNO as the default action that omission is merely surprising, because calls from an uncovered architecture are refused as well. Flip the default to allow and keep a deny list instead, and leaving those entries out is the classic bypass: the attacker walks in through a numbering table your rules never covered. The runtime's own default profile lists all three for exactly that reason. One more thing to notice: nine allowed calls will not start a real program, so take the names from a recording rather than from this fragment. Then point the pod at the file.

apiVersion: v1
kind: Pod
metadata:
  name: api
spec:
  securityContext:
    seccompProfile:
      type: Localhost
      localhostProfile: profiles/api-v1.json
  containers:
  - name: app
    image: registry/acme/api:1.4.2

seccomp, capabilities and MAC do three different jobs

seccomp decides whether the syscall happens at all. Capabilities (the pieces the kernel carved the old all-powerful root account into, such as CAP_SYS_ADMIN) decide whether an allowed syscall may do the privileged version of its work. MAC (mandatory access control, meaning AppArmor or SELinux) decides which files and sockets that work is allowed to touch. A profile that allows mount buys an attacker nothing on a container that has dropped CAP_SYS_ADMIN: the call clears the filter, and the kernel's capability check is what turns it away. Allow mount and hand CAP_SYS_ADMIN back, and you reopened the escape path yourself. Depth here means RuntimeDefault, plus drop ALL capabilities, plus AppArmor or SELinux in enforcing mode. seccomp on its own is one layer, not the answer.

kubectl get pods -A -o json | jq -r '.items[] | select(.spec.securityContext.seccompProfile.type=="Unconfined" or (.spec.securityContext.seccompProfile|not)) | .metadata.namespace + "/" + .metadata.name' | head -5
 
# example output:
prod/payments-api-7c9
prod/legacy-batch-0
staging/tools-jumpbox

Admission is what makes it stick

Pod Security Admission (PSA, the gate built into Kubernetes that grades every pod against a named standard) already requires RuntimeDefault or tighter at the restricted level. Kyverno and Gatekeeper, policy engines that inspect objects on their way into the cluster, can reject Unconfined at deploy time and audit what is already running. Count compliance every month. One privileged debug pod sitting in prod with Unconfined seccomp is a kernel-wide opening waiting for a remote code execution bug in whatever it runs. If a namespace genuinely needs the relaxation, write it down, name an owner, and give it an expiry date.

kubectl label ns prod pod-security.kubernetes.io/enforce=restricted --overwrite
 
# example output:
namespace/prod labeled

Teams push back on seccomp after one obscure block in staging, and they remember it for years. Get ahead of that with the SRE (site reliability engineering) crew: record with strace or SPO during full integration runs, not the happy path your CI (continuous integration) suite exercises. Every syscall you add to an allowlist needs an owner and a review date, because allowlists rot exactly the way firewall rules do. Re-record when you upgrade a language runtime. JIT compilers (just-in-time, the part of a runtime that turns hot code into machine code while the program runs) and garbage collectors quietly reach for new syscalls, and you find out in production, because CI ran on a different glibc (the GNU C library) patch level than your nodes.

How seccomp shows up during an incident

A blocked syscall almost never announces itself. The app logs a generic permission error, not the words "seccomp blocked clone". Put a line in the runbook that sends anyone staring at an EPERM (the kernel's "operation not permitted" error) stack trace to check the seccomp profile early, especially right after a glibc or runtime upgrade brought in something new like io_uring (a newer, faster way for a program to queue input and output work with the kernel). One more rule about recordings: an SPO capture is not finished until you remove the ProfileRecording object and the SeccompProfile it writes reports Installed. Enforce a half-recorded profile on a production tier and your gap list becomes an incident.

journalctl -k | grep -i seccomp | tail -1
 
# nothing comes back on a stock node unless the filter was loaded with the log flag
# or the action was SCMP_ACT_LOG. When a record does land it reads like this:
audit: type=1326 audit(1721802000.441:901): auid=4294967295 uid=10001 gid=10001 ses=4294967295 pid=55102 comm="app" exe="/app/server" sig=0 arch=3221225534 syscall=425 compat=0 ip=0x7f... code=0x50000

You do not get that line for free. An errno refusal is silent unless the filter carried the log flag or the profile used SCMP_ACT_LOG, so an empty grep is not proof that seccomp stayed out of it. When a record does land, it ends with syscall=425, a number rather than a name. Numbers map to names through the audit architecture tables, and the mapping depends on the architecture, so keep that document current. The moment you can name the refused call, the argument stops being mystical and turns into a small decision: allow that one syscall, or fix the library reaching for it.

Field notes from real clusters

The value of seccomp is that the dangerous call never happens, not that somebody noticed it afterwards. A detection tool tells you a container tried to mount something. seccomp means the container asked and got a flat refusal before the kernel walked any of the interesting code paths. Whole families of exploit gadgets stop working once the entry point is closed.

So which workloads earn a hand-written profile? Two kinds. The ones where a recording found calls the runtime default refuses, and the ones holding something an attacker would spend a kernel bug on: the payment path, the signing service, the tenant that runs other people's code. The bill is recurring. Every base image bump, every language runtime upgrade, every new dependency can add a call, and the profile has to be re-recorded and re-reviewed before that change ships. Price that upkeep before you write the file, not after the first broken deploy.

An allowlist written from memory produces broken deploys and developers who stop trusting you. Build the first draft with SCMP_ACT_LOG as the default action instead: unmatched calls still run, and the kernel writes an audit line for each one, so you collect the gap list without breaking the workload. Swap the default to SCMP_ACT_ERRNO once that log has gone quiet across a full release cycle.

Mismatched levers read worse than no levers. Dropping CAP_SYS_ADMIN while leaving a wide-open syscall set tells two stories at once, and a tight profile on a privileged pod is theatre. Line them up: non-root user, dropped capabilities, RuntimeDefault or tighter, read-only root filesystem wherever the app can live with one.

Good intentions become fleet reality only through admission. PSA, Kyverno or OPA Gatekeeper (Open Policy Agent, a policy engine that answers yes or no for each incoming object) can require securityContext.seccompProfile.type: RuntimeDefault. Without that, one revert of one YAML manifest (the text format Kubernetes objects are written in) quietly puts Unconfined back. Back the policy at the node too: turn on the kubelet's seccompDefault setting (--seccomp-default=true, generally available since Kubernetes 1.27) and every pod that does not name a profile of its own comes up under RuntimeDefault, whatever the manifest lost on the way in. Make the control visible in the pull request and alert on violations.

Most seccomp incidents read as "the app started failing after the platform team tightened profiles". That is a process win if you have a fast exception path with an expiry date attached. It is a security loss if the exception hardens into permanent Unconfined. Track exceptions like fire extinguisher pins: numbered, owned, checked on a schedule.

Know the ceiling. seccomp has no idea what a Pod or a Secret is, and it does not read file paths. If read and openat are allowed and the secret is already mounted in the container, seccomp will watch the process read it without blinking. For the behaviours seccomp cannot name, you need MAC, network policy and runtime detection.

Watch what a wrong profile actually looks like. The app returns a strange errno, usually EPERM, on a call it has made a thousand times: clone with a particular flag, an io_uring setup, some niche setsockopt. Developers blame the platform. The platform blames the app. What ends the argument is a syscall trace from staging showing the refused call, plus a tracked exception with a date on it. Without that loop, somebody eventually sets Unconfined and the hardening story ends quietly.

A pod that fails under RuntimeDefault is the best teaching material you will get. Compare the seccompProfile in the pod YAML against the runtime's default profile path, then reproduce the failure with strace or a logging profile until you can read the denied number. Numbers become names through the audit logs or scmp_sys_resolver, and the answer depends on the architecture, so resolve it against the one in the audit line rather than from memory. "Blocked syscall 56, which is clone on x86_64" is a sentence that turns into a patch or a one-line allowlist edit. "Kubernetes is being weird" is not.

Two ways teams misuse this. One is the company-wide mega profile, which rots the day any team ships a new library. The other is a profile copied from a blog post and applied to binaries nobody measured, which blocks harmless calls and teaches everyone that opting out is the safe move. Both fail the same way: the file was written once, by somebody who is no longer looking at it.

The trade-off is worth saying out loud. A tight allowlist shrinks the exploit surface and raises your change-management cost, because every library upgrade can arrive with a new syscall. Budget profile review into dependency upgrades the same way you budget API contract tests. A maintained default beats an abandoned masterpiece.

One more failure mode, and it is sneaky. Admission demands RuntimeDefault, then a mutating webhook or an ancient Helm chart (the packaged templates many teams use to generate their manifests) resets the field to Unconfined after the check has already passed. Diff live pods against policy intent every week. If your Kyverno or OPA rule says required while the running pods say Unconfined, you have a paperwork program rather than a seccomp program. Close the loop with a report the owners cannot quietly ignore.

In incident reviews, seccomp keeps turning up as the quiet control that would have stopped a noisy chain of syscalls before Falco (the runtime detection tool that watches syscalls and alerts on them) ever needed to page on-call. Keep a lab pod that deliberately violates its profile, so your pipeline proves the deny path still works after every node image bump.

Try this

Run this on a lab cluster or a single staging node. Read what comes back and sit with it. A first look tells you what is true right now, not what to change in production.

$ kubectl run web --image=nginx:1.27 \
    --overrides='{"spec":{"securityContext":{"seccompProfile":{"type":"RuntimeDefault"}}}}'
pod/web created
$ kubectl exec web -- grep Seccomp /proc/1/status
Seccomp:	2
Seccomp_filters:	1
$ kubectl run bad-profile --image=nginx:1.27 \
    --overrides='{"spec":{"securityContext":{"seccompProfile":{"type":"Localhost","localhostProfile":"profiles/not-on-this-node.json"}}}}'
pod/bad-profile created
$ kubectl get pod bad-profile
NAME          READY   STATUS                 RESTARTS   AGE
bad-profile   0/1     CreateContainerError   0          14s
$ kubectl delete pod web bad-profile
pod "web" deleted
pod "bad-profile" deleted

Takeaway

seccomp trims the syscall menu before the LSM (Linux Security Module framework, where AppArmor and SELinux plug in) and before a single line of your app logic runs, which makes it the cheapest place to delete an attack technique. RuntimeDefault across the fleet, and a Localhost profile only where a recording proved you needed one.

Next: turn on RuntimeDefault through admission in one non-production namespace, live with it for a week, see what breaks, then widen. Mandatory access control is where the next lesson picks up, because the syscalls seccomp allows still need something watching which files they touch.

Related