eBPF tracing fundamentals
Probes, maps, and the verifier.
A restaurant health inspector can't taste every dish or live in the kitchen. So they do the next best thing. They stand where the food passes, watch each plate go by, and write down anything wrong. eBPF (extended Berkeley Packet Filter, a name that is now mostly historical) lets you post an inspector exactly like that, except inside the Linux kernel (the core part of the operating system that talks directly to the hardware and controls every running program). You hand the kernel a tiny program, and it runs that program every time a chosen event happens: a process starting, a file opening, a network connection going out.
The program sees the event at the instant it occurs, in the one place userland malware (malicious code running as an ordinary program, out in user space rather than inside the kernel) finds hardest to lie about. Compare the alternatives. A tool like osquery asks the machine questions on a schedule, so it only catches state that survives until the next poll. Log parsing sees whatever the application decided to write, after the fact, and an attacker who owns the process owns its logs. An eBPF probe sees the raw syscall (system call, the request a program makes to ask the kernel to do real work like exec or connect) as it fires. That is ground truth, and it is why Falco, Tetragon, and Cilium are all built on it.
Three pieces make it work, and they are the whole lesson. Probes are where you hook in. Maps are how the program remembers things and reports them. The verifier is why the kernel is willing to run your code at all.
Probes: Where You Tap the Wire
A probe is a wiretap clip. You attach it to one specific wire, and it fires whenever a signal runs past. In eBPF terms a probe is an attach point, a named spot in running code where your program hooks in. Three kinds cover most security work. Tracepoints are stable hooks the kernel maintainers placed on purpose, like syscalls:sys_enter_execve; they rarely change, so a probe on one keeps working across kernel upgrades. Kprobes (kernel probes) attach dynamically to almost any internal kernel function by name, which is flexible but fragile. Uprobes (user-space probes) do the same trick inside a normal program, so you can read data before a library encrypts it. Two newer families matter for detection too: fentry/fexit hooks, which attach at function entry and exit more cheaply than kprobes, and BPF LSM (Linux Security Module) programs, which sit on the kernel's own security hooks. Those can allow or deny an action outright, where every other probe can only watch it happen.
comm is the name of the process that made the call, and args->filename is the program it is about to run. The nginx line is the one you care about. A web server has no reason to launch /bin/sh. That is a web shell, or a worker that was exploited a moment ago, and you caught it at the exec, before the shell did anything. The line under it, sh running curl, is the second stage reaching out to pull down more.
Maps: The Program's Memory and Its Mailbox
An eBPF program has amnesia. It wakes up when its event fires, runs for a few microseconds, and forgets everything. A map fixes this, and it does two jobs. It is the ledger the program writes to so a fact can survive from one event to the next, and it is the row of pigeonhole mailboxes it drops finished notes into for a user-space tool to collect later. Technically a map is a key-value store living in kernel memory. Common types are hash maps, per-CPU arrays (one private copy on each processor core, so cores never fight over the same slot), LRU (least-recently-used) hashes that evict old keys on their own, and the ring buffer (a first-in first-out queue added in kernel 5.8, now the usual way to stream events out to a waiting tool). Older tools used a perf event array for the same streaming job; the ring buffer replaced it because it keeps events in strict order and uses memory more efficiently.
In bpftrace, every variable that starts with @ is a map. This one keeps a running count of outbound connections per process, updated in the kernel on every call and printed when you stop it:
That tally lives in a hash map keyed by process name. You can see the real maps behind any loaded program with bpftool, including their type, key and value sizes, and capacity:
The hash map here holds up to 10240 entries of state. The ring buffer next to it is 256 kilobytes of queue that a detection tool drains continuously, which is how something like Falco gets a clean, ordered stream of events instead of polling the machine and hoping nothing slipped through between checks.
The Verifier: Frisked at the Door
The verifier is a bouncer who will not let you into the server room until you have proven, on paper, that you cannot break anything inside. You will definitely come back out (no getting stuck forever). You will only open the drawers you were assigned. You will never reach past them into someone else's data. No proof, no entry. In real terms the verifier is a static analysis inside the kernel that walks every possible path through your program before it loads. It checks that the program finishes (originally no loops at all; since kernel 5.3 it allows bounded loops, as long as the whole program stays under the verifier's budget of one million analyzed instructions for a privileged loader), that every pointer is tracked so you cannot read uninitialized memory or run off the end of a buffer, and that you never touch arbitrary kernel memory except through a fixed set of helper functions.
When a program passes, the kernel JIT-compiles it (Just-In-Time, translating the bytecode into native processor instructions) so it runs at close to the speed of hand-written kernel code. This is the whole reason eBPF is safer to load than a kernel module. A module is a guest handed the keys to the entire building; one bug crashes the machine, and a malicious one is a rootkit. Your eBPF program is proven harmless before a single instruction of it runs. Modern programs also carry BTF (BPF Type Format), a description of kernel data structures that lets one compiled binary run correctly across many kernel versions, an approach called CO-RE (Compile Once, Run Everywhere). It is what makes shipping a single detection agent to a fleet of mixed kernels workable.
You can watch the verifier say no. A classic mistake in a network program is reading packet bytes without first checking they exist. Load that, and the kernel refuses with a precise reason:
The fix is one line, an explicit check that data plus the header size is still inside the packet before you read it. That is the deal the verifier drives. It is stricter than a normal compiler, and its errors feel maddening the first few times, but every rejection is the kernel refusing to run something that could have hurt it. Nobody gets to skip the frisk.
A value of 2 means unprivileged loads are blocked, but a root user can switch them back on while the machine keeps running; that is the modern Ubuntu and Debian default. A 1 blocks them too and latches the setting shut, so nothing can flip it back on until you reboot. A 0 lets any local user load a program, which you do not want anywhere near a server.
Confirm the Probe Is Actually Running
A probe that failed to attach reports nothing, and nothing looks exactly like all-clear. So before you trust one in production, load it and check. bpftool lists every program the kernel is currently running, when it loaded, who loaded it, and which maps it feeds:
Confirm your program is there, that its type and attach point match what you wrote, and that the map IDs line up with the maps you expect it to fill (here 27 and 28, the same hash and ring buffer from earlier). If a program you never loaded is sitting in that list, someone else is on the same wire you are. Everything after this is turning that clean stream of events into alerts worth waking up for.
Try this
Work through “Confirm the Probe Is Actually Running” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: the wiretap points both ways. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.