eBPF for runtime security

Sandboxed kernel programs, maps, and the verifier.

A night watchman with binoculars can see who walks into the warehouse. By the time he keys the radio, the thief is already inside and moving. Wire a sensor into the door frame itself and the alarm fires the instant a hand touches the handle, with no holes drilled that might weaken the wall. That is the job eBPF does for Linux. It watches from inside the kernel, at the moment something happens, without putting the machine at risk. Every runtime security tool in this course leans on that.

A few words before the tools. eBPF (extended Berkeley Packet Filter, small programs that the Linux kernel checks for safety and then runs inside itself) is how Falco, Tetragon and Cilium watch system calls without loading a risky kernel module. Before you tune a single rule or roll out a DaemonSet (the Kubernetes object that puts one copy of a pod on every node), you want a clear picture of what eBPF attaches to, what the verifier checks, and why the industry ended up here.

Tiny programs bolted onto kernel hooks

The kernel ships with pre-drilled mounting points, and you bolt a program onto one of them: syscall tracepoints (fixed markers at system call entry and exit), kprobes (hooks on internal kernel functions), LSM (Linux Security Module, the kernel's built-in permission checkpoints) decision hooks, and the packet handling paths, tc and XDP. Every program you load goes through a static verifier first. The verifier walks the code and proves that loops terminate and that memory reads stay in bounds. Only then does the kernel accept it. That check is what makes running foreign logic in kernel context sane instead of reckless. Accepted programs are compiled to native machine code and hand data back to userspace through maps and ring buffers. Because the watching happens right at the syscall boundary, a container process cannot hide. To run a shell, open a file or dial out to the internet, it has to ask the kernel.

bpftool prog show | head -6
 
# example output:
467: tracepoint  name falco_execve  tag 3a1b9c2d  type tracepoint
     loaded_at 2026-07-18T09:14:22+0000  uid 0
     xlated 1840B  jited 920B  memlock 4096B
468: kprobe      name tetragon_kprobe  tag 7f02e811  type kprobe
     loaded_at 2026-07-18T09:14:23+0000  uid 0
469: tracing     name cilium_policy  tag 91ac0044  type tracing

On any node running Falco or Tetragon you will find dozens of loaded programs: execve tracepoints, file permission kprobes, policy programs from Cilium. That output tells you what is actually attached. Knowing that the kernel supports eBPF is a much weaker statement. When events go missing, your first move is to check that the program you expect loaded, and that it is bound to the hook you think it is.

Maps and ring buffers: how the kernel talks back

A program running inside the kernel cannot print to your terminal. It writes structured records into a map or a ring buffer, and a userspace agent (Falco, Tetragon, or the Cilium agent) picks them up. The agent adds the context the kernel never had, such as container and Kubernetes metadata, then runs your rules or ships the event to a SIEM (security information and event management, the platform where your alerts land). Ring buffers carry the high-volume stream of security events. Hash maps hold small pieces of state, along the lines of "have we seen this binary before". Observe in the kernel, decide in userspace, unless you attach at BPF-LSM. That one split is the source of the detection gap every runtime engineer eventually has to close.

bpftool map show | grep -E "ringbuf|hash"
 
# example output:
471: ringbuf  name events  flags 0x0
     key 0B  value 0B  max_entries 262144  memlock 1048576B
472: hash  name pid_cache  flags 0x0
     key 8B  value 128B  max_entries 16384  memlock 2097152B

Why the kernel module went away

Older runtime tools shipped an out-of-tree kernel module to capture syscalls. That held up fine until a bug in the module took a node down, or until you counted the kernel versions across your fleet and needed a separate build for each one. eBPF removes both headaches. The verifier refuses unsafe programs outright, and CO-RE (Compile Once, Run Everywhere) leans on BTF (BPF Type Format, the kernel describing its own data structures) so one probe binary bends to fit different kernel versions with no per-node recompile. That portability is why Falco defaults to an eBPF driver, and why Tetragon and Cilium were built on eBPF from the first line rather than on modules.

# Confirm BTF is available (required for portable probes):
ls /sys/kernel/btf/vmlinux && echo "BTF OK"
 
# Falco driver probe type in modern installs:
falco --version | grep -i driver
# Driver name: ebpf (CO-RE)

BPF LSM: where watching turns into stopping

A smoke detector tells you the kitchen is on fire. A sprinkler head does something about it. Observation and enforcement are two different jobs. The BPF LSM hook lets an eBPF program sit at the same decision points SELinux and AppArmor use, and answer allow or deny before the action completes. Tetragon Sigkill and BPF-LSM deny policies close the distance between "we saw a shell" and "the shell already ran". You do not need in-kernel enforcement on day one. Knowing the hook is there is what explains how a modern runtime stack can do anything beyond raise an alert.

cat /sys/kernel/security/lsm
 
# example output:
lockdown,capability,yama,apparmor,bpf

When bpf shows up in that list of active LSMs, the kernel is willing to load enforcement programs at security hooks. Add LSM programs on top of your observation probes (the tracepoints) only after you have measured how many false positives those probes produce. The lesson on enforcement walks that rollout path in order.

Running probes in production

Start with pinned versions. The Falco driver, the Tetragon agent and Cilium all have to match the kernel family your nodes actually run. Watch memlock as well (memlock is locked memory, the pages the kernel refuses to swap out, and eBPF programs and their maps live there). Once that budget is gone, no new probe can load. After every node kernel upgrade, confirm the probes came back before you call the upgrade finished. A probe that did not reload is a detection outage, not a cosmetic warning in a log.

kubectl -n falco logs ds/falco --tail=5 | grep -i ebpf
 
# example output:
2026-07-24T04:12:01.441Z INFO  falco: Opening 'ebpf' source with modern BPF probe
2026-07-24T04:12:01.892Z INFO  falco: Loaded 312 rules
2026-07-24T04:12:01.901Z INFO  falco: gRPC server listening on 0.0.0.0:5060

Following one exec from syscall to alert

A container spawns /bin/sh. The syscall tracepoint fires before userspace ever gets control back. The Falco or Tetragon probe reads the CPU registers, resolves the binary path, walks cgroup metadata (the kernel's grouping of processes, which is what a container really is) up to a container ID, and queues a record. Userspace fills in the rest from CRI (container runtime interface, how Kubernetes talks to the runtime on the node): pod name, namespace, image digest. Then policy runs. On a modern node, syscall entry to SIEM receipt lands in single-digit milliseconds. Hold onto that number, because it decides whether detect-and-respond is fast enough for a given threat model or whether you need the kernel itself to say no.

bpftool prog dump xlated name falco_execve 2>/dev/null | head -8
 
# example output:
int falco_execve(struct pt_regs *ctx):
 0: mov r1, ctx
 8: call bpf_get_current_pid_tgid
 16: stw [fp-4], r0
 ...

Reading bpftool output pays off during upgrades. If the bytecode loads but the attach fails, the hook point may have moved between kernel versions. That is the exact problem CO-RE solves with BTF relocations. Keep one staging node on the next kernel version and prove the probes attach there before the fleet follows.

Budgeting locked memory

Every program and every map holds memlock. A large fleet running Falco, Cilium and Tetragon side by side should watch MemAvailable in /proc/meminfo after the agents start. Linux defaults are often too tight, so agent containers may need a raised RLIMIT_MEMLOCK. Write the sysctl and ulimit settings into your node hardening baseline. Otherwise a security agent quietly fails to load an extra probe on the one day you need it, which is the day you are working an incident.

Field notes from real clusters

A kitchen during dinner service is loud, and counting plates as they leave the pass tells you nothing about the moment someone pulled a knife from the wrong drawer. Log scrapers count plates. A kernel hook watches the drawer. That timing difference is the whole reason runtime security lives down at the syscall layer instead of in application logs.

You do not need to write eBPF programs in C to run this stack well. What you do need is the shape of it in your head: what attaches, what holds state, and what breaks quietly after a kernel upgrade. When Falco goes silent after a node reboot, the question is almost never "did Kubernetes break". It is "did the probe load". That instinct saves hours.

Maps are the shared whiteboard between kernel and userspace. A ring buffer streams events across it. A hash map remembers things like "we already alerted on this binary". If events start dropping under load, look at ring buffer size and at whether the reader is keeping up, before you suspect anything else. Starving the consumer is a quiet and common failure. Your SIEM cannot alert on an event that never left the node.

CO-RE plus BTF is why one probe binary can ride across many kernel versions. Take BTF away and you are back to hard-coded struct offsets and a fresh build per kernel. Confirm that /sys/kernel/btf/vmlinux exists in every node image before you promise leadership portable detection.

Enforcement is a different promise than observation. A tracepoint that records execve does not stop the shell that execve started. BPF-LSM, Tetragon kill actions, or a userspace response path are choices you make on purpose, each with a false-positive budget attached. When someone asks whether eBPF stops attacks, answer with the hook and the mode the tool is running in, not with a yes or a no.

Teams get this wrong in a few predictable ways. They put Falco and Cilium on the same node without budgeting memlock, then wonder why the third probe refuses to load in the middle of an incident. They treat bpftool as optional trivia right up until a kernel upgrade silences detection. And they tell leadership that eBPF prevents attacks when everything they shipped is observe-only. Name the mode in every status update: observe, kill, or LSM deny.

Falco startup logs that announce the modern BPF probe and a rule count are a good sign, not proof. Believe them once a canary exec produces a live event. A Running pod with zero events for an hour on a busy node is a yellow flag. Back the log line up with bpftool prog show rather than trusting one userspace message. It is a boring check, and it catches the outages a green dashboard will happily hide.

One more operational tell. After a node joins the cluster, wait until bpftool prog show lists your security programs before you let sensitive workloads schedule there. Autoscaling that places pods first and loads probes second carves out a detection gap shaped exactly like a scale-up event, which is when an attacker would most like to be moving.

Try this

Run this on a lab cluster or a single staging node. Read the output and sit with it. Do not go changing production policy off a first look.

$ bpftool prog show | head -5
467: tracepoint  name falco_execve  tag 3a1b9c2d  type tracepoint
468: kprobe      name tetragon_kprobe  tag 7f02e811  type kprobe
$ ls /sys/kernel/btf/vmlinux && echo BTF_OK
BTF_OK
$ cat /sys/kernel/security/lsm
lockdown,capability,yama,apparmor,bpf
$ kubectl -n falco get ds falco
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE
falco  3         3         3       3            3

Takeaway

eBPF is the sensor layer underneath Falco, Tetragon and Cilium: verifier first, maps and ring buffers for the data, CO-RE so one binary survives a kernel upgrade. Watching a syscall and blocking a syscall are separate modes, and only one of them stops anything.

Next step: on one staging node, confirm BTF is present and list the loaded programs with bpftool. Then take the LSM lesson, where allow or deny before the action completes becomes possible.

Related