eBPF for security visibility

Probes, the verifier and bpftrace on real events.

eBPF lets you load a small program into the running kernel and attach it to an event: a process starting, a file opening, a TCP connection leaving. The program sees the event as it happens, from inside the kernel, which is why modern runtime sensors such as Falco are built on it. This lesson explains what eBPF is and why the kernel agrees to run your code, uses bpftrace one-liners to watch program execution, file opens and outbound connections made by an ordinary account, lists the programs and maps the kernel holds with bpftool, and covers who may load BPF and what eBPF cannot see. It is written from the defender's side and stands on its own; linux-perf/bpftools goes deeper into the tooling.

What eBPF is, and why the kernel trusts it

An eBPF program is bytecode for a small virtual instruction set, loaded with the bpf() system call. Before the kernel accepts it, the verifier (a checker inside the kernel) walks every path through the program and rejects it unless it can prove the program ends, reads only memory it is allowed to, and reaches kernel data only through a fixed set of helper functions. An accepted program is JIT-compiled (translated to native machine code) and attached to a hook. Maps are kernel-side key/value stores and queues that let a program keep state between events and pass results to a user-space tool.

BTF (BPF Type Format) describes the running kernel's data structures, so one compiled program can adapt to many kernel versions (CO-RE, compile once, run everywhere).

That is the difference from a kernel module, which runs with no such proof and can crash or subvert the kernel. The verifier is not a guarantee, though: bugs in it have been a recurring source of kernel privilege-escalation vulnerabilities, one reason both lab platforms keep unprivileged BPF switched off. And it proves memory safety, not good intent or low cost. A program that passes can still read what it is allowed to read, including other people's data; a probe on a busy code path adds work to every call; and some program types exist to change behaviour (deny an action, drop a packet).

Program types differ by hook. Tracing programs, the only kind this lesson loads, attach to tracepoints, kprobes (almost any kernel function), fentry/fexit (function entry and exit, cheaper than kprobes) and uprobes (functions inside a user-space program). LSM programs attach to the kernel's security hooks and can deny an action. XDP and tc programs process network packets, and cgroup programs filter what a group of processes may do. The lab host has what tracing needs.

Kernel 7.0 exposes its BTF at /sys/kernel/btf/vmlinux, and bpftrace 0.25 is installed (on the Rocky lab, bpftrace 0.24.2). kernel.unprivileged_bpf_disabled = 2 means accounts without privilege cannot use bpf() at all, a setting an administrator may change back at run time; 1 would lock it off until reboot, and 0 would allow some unprivileged program types. Both lab platforms default to 2. So the ordinary account's attempt to list programs fails.

Listing and loading programs both take privilege, as the Operation not permitted shows. For tracing, the capabilities are CAP_BPF for BPF operations plus CAP_PERFMON for performance and tracing hooks, split out of CAP_SYS_ADMIN in Linux 5.8 so a sensor need not run as full root. Anyone holding them can watch the whole machine. Walking the list of every loaded program, which bpftool prog list does, still needs CAP_SYS_ADMIN, so in practice the inventory below runs as root.

Attach points: tracepoints and their arguments

A tracepoint is a hook the kernel developers placed on purpose and keep stable across versions, such as the entry and exit of every system call. A kprobe can attach to almost any kernel function by name, but that name is not a promise. If a later kernel renames or removes the function, the probe fails to attach with an error when it loads. The dangerous case is quieter: the function still exists, but the compiler has inlined it at some call sites or the code path you cared about no longer calls it, so the probe attaches and simply stops firing for those events. For detections that must survive kernel upgrades, prefer tracepoints (or fentry with BTF). bpftrace -l lists attach points and -lv shows the arguments a tracepoint gives you.

sys_enter_openat hands a program the directory descriptor, a pointer to the filename, the flags and the mode, exactly the arguments of openat(2). The one-liners below read these as args.filename and so on.

Watching ordinary activity with bpftrace

A throwaway account, eb-dev, will do ordinary things while bpftrace watches only its UID. Each one-liner starts the account's commands in the background with a short delay, attaches, and ends itself after eight seconds with an interval probe; at a terminal you would leave that probe out and stop the tracer with Ctrl-C. The UID reaches the filter as bpftrace's first positional parameter: $1 in the script is the first argument after it, here $(id -u eb-dev), so no number is hard-coded and the same line works on your host.

Each line is one execve call: the parent's PID (the shell; curtask->real_parent->tgid reads it from the kernel's own task structure), the PID of the child the shell forked to run the command, comm (the name of the process making the call, still sh because the new program has not replaced it yet) and the program being run. The shell ran id and uname. This is the raw material of every "unexpected child process" detection, such as a web server whose child is a shell. Next, file opens and their results. The enter probe stores the filename pointer in a map keyed by thread ID, and the exit probe reads it back together with the return value, then deletes the entry (_ = discards the value delete() returns; bpftrace 0.25 prints a warning when that value is silently dropped).

Every cat first opens the loader cache and its shared libraries (most lines, elided here), and -2 is ENOENT, a translation file that does not exist. The line that matters is /etc/shadow -13: -13 is EACCES, an unprivileged account trying and failing to read the password hashes, while /etc/hostname 3 succeeded with file descriptor 3. Failed attempts on sensitive files are exactly what application logs never show, because the program wrote nothing; an audit rule on that file would record it (linux-det/auditpipe), but only for files you chose in advance. Last, outbound connections. The sock:inet_sock_set_state tracepoint fires on every TCP state change; state 2 is SYN_SENT, the moment a connection attempt leaves. A probe takes one predicate, so the state test and the UID test are joined with &&.

curl, running as eb-dev (UID 1002), tried to reach 192.0.2.10 on port 80. That address is from TEST-NET-1, reserved for documentation, so nothing answered; the kernel still reported the attempt, with the process and user behind it. A beacon to an attacker's server looks the same, which is why "who opened a connection to where" is a staple sensor signal.

Inventory: which programs is the kernel running?

Every loaded program and map is visible to root through bpftool. Knowing what is normally there is the baseline that makes an unexpected program stand out.

On this host systemd has loaded cgroup_device, cgroup_skb and cgroup_sysctl programs to enforce unit settings such as device access and IP filtering. The one tracepoint program, lima_ticker, belongs to the lab VM tool's guest agent, not to Ubuntu. Now start a tracer in the background, as a sensor would run, and look again while it is attached.

bpftrace's program appears as a tracepoint named after its probe, loaded by uid 0, with the IDs of the maps it uses. AT_connects is the @connects map, a per-CPU hash (one copy per CPU, so CPUs never contend) with room for 4096 keys. The kernel also audits every program load and unload by itself, with no rule needed, as long as auditing is switched on (here auditd switched it on).

The BPF record gives a program ID and op=LOAD (bpftrace loads more than one program, and tail kept its last); its SYSCALL record names the loader (comm=bpftrace) and the person behind it (auid=deploy, as linux-det/auditpipe explains). Now stop the tracer the way Ctrl-C would, with SIGINT, and look for the program's ID from the listing above.

The program left the kernel with its loader, and the audit log holds both ends of its life: prog-id=999 op=LOAD and op=UNLOAD. Forwarding these records gives you a history of every program ever loaded on the host. They depend on the audit subsystem being enabled, though, and on a default Ubuntu server, with no auditd, it is not.

The kernel initialised audit disabled at boot, and the 17 programs loaded since then (systemd's, plus the VM tool's lima_ticker) left no BPF record (type 1334) at all. Install auditd (linux-hard/auditd) before you rely on this history.

What eBPF does not see

Probes attached from inside the kernel are hard to evade from user space, but they are not the whole truth. An attacker with root can unload a sensor's programs, detach them, or load their own that tamper with the view, so a sensor needs its own liveness check and its alerts must leave the host. Events can be dropped: a ring buffer that fills faster than the user-space side reads it loses events, and tools report that as lost or dropped counts. A sys_enter probe reads arguments from user memory before the kernel does, so a determined attacker can change the buffer in between (a time-of-check, time-of-use race), which is why sensors such as Falco attach extra programs to cross-check it. Operations submitted through io_uring(7) do not pass through the per-call system-call tracepoints, so a file opened that way never fires sys_enter_openat; hooks deeper in the kernel (LSM or VFS functions) catch both. A process inside a container reports its host PID. A kprobe on an internal function can go silent on a kernel upgrade, as described above. And tracing has a cost: a probe on every openat runs on every call, whoever makes it, so on a busy production host keep ad-hoc tracing short and filtered inside the probe, as these one-liners are. Finally, tooling can lag the kernel: on Ubuntu 26.04 several bcc 0.35 tools compile their C at run time against the 7.0 kernel headers and currently fail, as execsnoop-bpfcc does here.

clang rejected a static assertion in the kernel's own fs.h header, so the tool never loaded; that is a packaging mismatch between bcc and this kernel, not a problem with eBPF. The bpftrace one-liners above, and the ready-made .bt tools the bpftrace package installs (/usr/sbin/execsnoop.bt, opensnoop.bt, tcpconnect.bt on Ubuntu), work. On RHEL the bcc tools live in /usr/share/bcc/tools and the same tool runs on the Rocky lab's 6.12 kernel.

Try this

On the Ubuntu lab host, create a throwaway account with sudo useradd -m eb-test. In one terminal, start the connection tracer for it without the interval probe, so it runs until you stop it: sudo bpftrace -e 'tracepoint:sock:inet_sock_set_state /args.newstate == 2 && uid == $1/ { printf("%-6s pid=%-7d uid=%-5d -> %s:%d\n", comm, pid, uid, ntop(args.daddr), args.dport); }' $(id -u eb-test). In a second terminal run sudo -u eb-test curl -s --connect-timeout 2 http://192.0.2.10/ and watch one line appear with curl, the UID and 192.0.2.10:80. While the tracer runs, find its program with sudo bpftool prog list | grep -A3 'name tracepoint_sock' and note the ID before the colon. Stop the tracer with Ctrl-C and confirm the program has gone from the list. Then find that ID twice, as op=LOAD and op=UNLOAD, with sudo ausearch -m BPF -ts recent -i | grep 'prog-id=<ID> '. Remove the account with sudo userdel -r eb-test.

Takeaway

Use tracepoints for anything you depend on, keep a baseline of bpftool prog list and forward the kernel's BPF audit records, and treat an unexplained loaded program as an intrusion until proven otherwise. eBPF shows what happens at the hooks you chose, so know which paths, such as io_uring, go around them.

Related