Tetragon in-kernel enforcement

TracingPolicy and synchronous Sigkill.

A night guard who phones the police after the thief is already inside the vault has recorded a crime, not stopped one. Tetragon works the other way round. It can throw the lock the moment the wrong fingerprint touches the sensor, because it makes the decision inside the Linux kernel before the syscall (system call, the request a program makes to the kernel) has finished running.

A few words before the commands. Tetragon is the runtime security agent built by the Cilium team, and it runs on eBPF (extended Berkeley Packet Filter, a way to load tiny sandboxed programs into the Linux kernel). It watches process starts, file access, network activity and privilege changes, and it does that cheaply. You tell it what to care about with a TracingPolicy, a CRD (Custom Resource Definition, a Kubernetes object type you define yourself) that says which kernel events to trace and what to do when one matches, up to and including Sigkill: ending the process in the kernel. This lesson covers deploying it, writing those policies, and rolling them out without taking production down.

What it watches, and what it can stop

Scope your first policies to a single namespace. A namespace-scoped policy has a blast radius you can describe in one sentence, which matters while you are still learning what your own workloads do all day. Cluster-wide policies reach every pod on every node, including ones owned by teams you have never met. Grow the scope the way you grow permissions: only when something you actually need forces it.

Where you hook, and what you do when the hook fires, change the outcome completely. Killing a process at execve (the syscall that loads a new program) is not the same as denying it at file open. The first stops the program from ever running. The second lets it run and blocks one action. Pick the earliest hook that still says what you mean, and remember the sidecar containers you forgot were in the pod, because they run programs too.

Practice on staging until applying and removing a policy feels dull. Dull is what you want at three in the morning.

Tetragon does its filtering inside the kernel, so only the events you asked for ever cross into userspace. That is why the overhead stays low on a busy node. Within a TracingPolicy, selectors match on things like argument values and return codes, and matchActions say what happens on a match: Post records the event and nothing else, Sigkill ends the process. Because the policy is an ordinary Kubernetes object, it lives in Git beside your Deployments and goes through the same review.

Writing your first TracingPolicy

Start with kprobes (kernel probes, hooks attached to a named function inside the kernel) on functions that already sit on a security boundary. security_file_permission is a good first pick, since the kernel calls it on the way to touching a file. Tracepoints are the other option when you want a syscall by name. Then narrow it down: one path, one binary, one namespace. Leave the policy in Post-only mode until the number of matches per day stops moving.

apiVersion: cilium.io/v1alpha1
kind: TracingPolicy
metadata:
  name: block-shadow-read
spec:
  kprobes:
  - call: security_file_permission
    selectors:
    - matchArgs:
      - index: 0
        operator: Equal
        values: ["/etc/shadow"]
      matchActions:
      - action: Sigkill

Observe before you enforce

Swap Sigkill for Post and ship the events to your SIEM (Security Information and Event Management, the system your security team searches). Then wait. Count the kills that would have happened across a full working week, deploy days and backup windows and whatever an admin does on a Thursday afternoon. Every legitimate match you turn up is a fix to the selector, made while the policy is still harmless.

kubectl -n kube-system logs ds/tetragon | grep block-shadow | tail -2
 
# example output:
policy=block-shadow-read action=Post process=curl pid=44102
policy=block-shadow-read action=Post process=healthcheck pid=9921

Running Tetragon next to Falco

Plenty of teams keep Falco for wide coverage from a mature rule set and add Tetragon for cheap telemetry plus a small number of surgical kills on the assets they cannot afford to lose. That overlap is fine, as long as your runbook names which of the two signals wakes a human. Two Critical pages for one behavior, with no deduplication in the SIEM, teaches people to ignore both.

tetragon gettracingpolicy
 
# example output:
NAME                 AGE
block-shadow-read    2d
observe-shell-prod   5d

Upgrades and day-two operations

Read the Cilium agent compatibility matrix before you upgrade Tetragon, because the two move together. Afterwards, verify two things instead of assuming them: that the CRDs reconciled, and that the BPF programs actually attached to their hooks. The bpftool checks you already run for Falco work the same way here. An upgrade that quietly detaches a policy leaves you enforcing nothing while the dashboard still shows the policy as present.

matchActions:
- action: Post
  rateLimit: 1m
  rateLimitScope: process

Testing policies in CI

In CI (continuous integration, the pipeline that runs on every pull request), render the TracingPolicy manifests and validate them against the CRD schema with kubeconform, so a typo fails the build instead of the cluster. If you can spare the minutes, apply them to a kind cluster (kind runs a throwaway Kubernetes cluster inside Docker) with Tetragon installed and confirm they are accepted. Paste the staging match counts into the pull request description too. A reviewer who can see that a policy matched the health-check binary four thousand times last week will send it back for a narrower selector.

kubectl explain tracingpolicy.spec
 
# example output:
GROUP:   cilium.io
KIND:    TracingPolicy
VERSION: v1alpha1
FIELDS:  kprobes, tracepoints, podSelector, ...

The dual approval matters more than the folder. Security signs off on what the policy is meant to stop. Platform signs off on what else it might hit. A Sigkill policy that lands on a Friday afternoon with one approval has ruined enough weekends to count as a pattern rather than bad luck.

Field notes from real clusters

Tetragon's list of actions is wider than kill. It can post the event and stop there. It can send SIGKILL. It can override the value the kernel returns, so the program sees a plain failure instead of dying. It can deny at an LSM hook (Linux Security Module, the kernel's built-in place for security checks, the same machinery SELinux and AppArmor plug into). That range is what separates it from a detector that can only write a line to a log.

You write TracingPolicy or TracingPolicyNamespaced objects, and each one answers three questions: which hooks, which filters, which action. Start with the logging actions in a lab namespace. Watch the event stream until the selector stops surprising you. Add SIGKILL or an override only after that. Enforcement with no observe rehearsal is how you kill your own deployer Job and then spend an hour working out why nothing ships.

Write selectors in the language Kubernetes already speaks: namespace, labels, container name, binary path. A global kill on /bin/sh sounds decisive until an init container uses sh to copy a config file, or the person on call uses it to debug at 2am. Wrap the sensitive binary in a narrow rule with the known-good callers allowed, rather than banning a shell across a continent.

The two tools are good at different jobs. Falco brings a large community rule set and alerts that drop cleanly into a SIEM. Tetragon brings precise kernel enforcement and fits neatly into a cluster that already runs Cilium. Pointing both at every event on the node buys you nothing except CPU load and duplicated noise.

Operations come down to eBPF facts of life. Is BTF (BPF Type Format, the type information the kernel publishes so eBPF programs can find struct fields) present on the node? Is the memlock limit, the cap on how much memory a process can pin, high enough? Does the kernel version support the hooks you asked for? After a node is replaced, check that the agent reloaded its policies. A TracingPolicy that failed to apply after GitOps drift (GitOps means the cluster is driven from a Git repository, and drift is when the two disagree) is an enforcement outage that nothing pages you about, so watch CRD status and agent logs.

Test policies in CI against a pod that should trip them and a pod that should not. A throwaway kind cluster that runs one bad exec and expects a kill, then runs a normal deploy and expects success, catches the selector mistake that would otherwise surface on a Saturday. Keep those tests in the same directory as the policies so nobody edits one without the other.

Tell the application owners which of their processes can be killed, before you switch anything on. Enforcement that arrives unannounced reads as a platform outage and gets escalated like one. Detection that arrives unannounced reads as a ticket. The conversation is part of the control.

One team shipped a TracingPolicy matching binaries: [/usr/bin/curl] with SIGKILL. It looked precise on paper. Then the cluster autoscaler health checks started dying, along with a service mesh debug wrapper, across three namespaces. The rollback was ugly because GitOps reapplied the CRD faster than anyone could delete it. A full deploy cycle in observe mode, plus a canary namespace, would have shown all of that in advance. Enforcement with no dress rehearsal is a denial of service you inflicted on yourself.

When an event arrives, read four fields: the policy name, the process path, the pod labels, and the action. An action of Sigkill means the kernel already stopped it, so your note in the SIEM is the record of something that happened, not a decision waiting on you. A Post action means nothing was stopped and a human or another system still has to move. Runbooks that blur those two leave night on-call guessing whether the thing is still running.

The common own goal is copying an example policy off the internet straight into production. Examples are written wide on purpose, so they demonstrate something on anybody's cluster. Narrow them to your namespace labels and your container names first. Watch out too for two policies hooking the same function with different actions, which gets you kills that depend on ordering and a Friday spent explaining them.

Here is the honest trade-off against Falco. CRD-native enforcement is a strong fit when your cluster already runs Cilium, and a staffing burden when nobody owns TracingPolicy review. Begin with a handful of denies on the assets you genuinely care about, not a second full rule set running alongside the first. Ten policies you can audit beat two hundred you cannot.

Treat TracingPolicy files the way you treat production firewall rules: CODEOWNERS on the directory, required reviewers, a changelog anyone can read. A drive-by apply from someone's laptop is how a kill policy reaches production unreviewed. The pull request should show the move from observe to enforce as a visible diff. If your tooling cannot show you that diff, you are not ready to roll SIGKILL out widely.

Kprobes are not free. Each one adds work to a code path the kernel runs constantly, and a stack of chatty policies on hot syscalls turns up as CPU on your busiest nodes. Prefer a few hooks on sensitive binaries and namespaces over broad coverage. If node CPU climbs after a rollout, do not guess which policy caused it. Read the Tetragon metrics and pull the noisiest one first.

Try this

Run this on a lab cluster or one staging node. Read what comes back, and do not go changing a production policy off the strength of a first look.

$ kubectl -n kube-system get ds tetragon
NAME       DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
tetragon   3         3         3       3            3
$ kubectl exec -n kube-system ds/tetragon -- tetra getevents -o compact | head -2
process_exec  default/payments  /bin/sh
process_kprobe default/payments  security_file_permission

Takeaway

The thing worth carrying out of here: Post and Sigkill are the same policy with one word changed, and that word is the difference between a log line and a dead process. Earn the change with a week of match counts.

Next step: apply one namespaced observe policy in your lab, watch it for a day, then move on to Cilium network policy so process controls and network controls meet in the same cluster.

Related