The BPF toolkit (bcc & bpftrace)
The famous production-safe tracing tools.
A process ran on your web server at 3 in the morning. It spawned a shell, reached out to the network, grabbed something, and exited a few hundred milliseconds later. By the time your monitoring took its next sample, it was gone. top never showed it. A ps loop running once a second would have missed it too, because it started and died between two samples. The tool that catches that ghost, live, without slowing the box down, is built on eBPF.
Here's the mental model. The kernel (the core of the operating system, the part that talks to the hardware and hands out CPU, memory, disk, and network) is a busy factory floor. Every program that wants to open a file or send a packet has to go through it. The old way to watch a program, strace, is like a safety inspector who halts the machine at every single step to write down what it's doing. Accurate, but everything grinds to a crawl. eBPF (extended Berkeley Packet Filter, small programs you load into the running kernel) works differently. It's a set of tiny sensors you bolt onto the machinery. They report as work flows past them, and nothing stops.
Why This Is Safe To Run In Production
That difference is the whole reason these tools are trusted on live hosts. strace uses a kernel feature called ptrace (process trace), which stops the traced program every time it makes a system call (a request from a program to the kernel, like "open this file" or "send these bytes"). Each stop means two context switches, the CPU saving one job's state and loading another's. On a program making tens of thousands of syscalls a second, that tax can slow it down by ten times or worse. You do not want that on a production database.
eBPF pays almost none of that. Before the kernel will run your little program, a piece of code called the verifier reads it and proves it is safe: loops are bounded, every memory access stays in range, and the program always finishes. Anything that might loop forever or read wild memory is rejected at load time, not after it has already crashed the machine. The measuring then happens inside the kernel, and the numbers are added up in kernel data structures called maps. Only the finished summary gets copied out to you. Less copying, no stopping, tiny overhead.
bcc: The Ready-Made Tools
bcc (the BPF Compiler Collection) is a box of these sensors, already built. Dozens of small tools, each answering one classic question. You don't write any code. You run one by name. On Debian and Ubuntu they carry a -bpfcc suffix so they don't clash with older tools of the same name. execsnoop catches every process the moment it starts, including the short-lived ones top never sees. opensnoop shows exactly which files a program touches. biolatency draws disk latency as a histogram. tcplife logs every TCP connection with its size and how long it lived.
Read that top to bottom and you're watching an attack, not a batch job. A deploy hook shell spawns curl against 169.254.169.254 (the cloud metadata service that hands out temporary credentials to anything that asks), then a Python one-liner that opens a socket. That's credential theft followed by a reverse shell. execsnoop saw all three even though each lived for milliseconds. The RET column is the exec return value: 0 for success, a negative number if the exec failed.
Most of the I/O finishes in 128 to 255 microseconds. But look at the second little hump past 2 milliseconds. That bump is the slow tail that makes users complain while your average latency still looks fine. An average would smear those two behaviors into one number and hide the problem. A histogram splits them apart, so you can see the disk has two moods.
There's the metadata call again, this time as a finished connection: 1 KB pulled back from 169.254.169.254 in 3 milliseconds. tcplife gives you the whole story of each flow after it closes, so short connections that a packet capture might miss between polls still show up with their byte counts and duration.
bpftrace: One-Liners For The Questions No Tool Answers
When no ready-made tool fits, bpftrace lets you write your own probe in a single line. The language looks like awk (the old line-at-a-time text tool): a probe to attach to, then an action in braces. You choose where to hook, and you add things up as events fly past. Count syscalls per process. Draw a histogram of read sizes. Print every program that runs, with the file it launched. Same safe eBPF machinery underneath, so it's fine to fire off on a live server.
That second one is a size distribution: args->ret is how many bytes each read() handed back, and [0] means the call returned end-of-file. The same shape of one-liner turns into a security tripwire when you point it at process creation and print who ran what.
comm here is the calling program (still named sh, because the image hasn't been replaced yet), and str(args->filename) is the binary about to run. A root shell launching curl and then python3 in quick succession is the same intrusion you saw earlier, now caught with a single line you wrote yourself.
Where The Probes Attach
You'll keep hearing three words: tracepoint, kprobe, uprobe. These are the spots a probe can clip onto. A tracepoint is a stable inspection port the kernel developers built into the code on purpose. Because it's official, it stays in the same place across kernel versions and its fields are documented. Prefer these. A kprobe (kernel probe) can attach to almost any kernel function, even ones with no official port, which is powerful and fragile at once: function names and arguments can change between kernel versions, so a kprobe that worked last year might attach to nothing after an upgrade. A uprobe (user probe) does the same for functions inside a normal program, say one specific call inside nginx or a database. USDT (user statically defined tracing) is the tracepoint's user-space cousin, an official port the program's own authors added.
Modern tools tame the version problem with BTF (BPF Type Format, a map of the running kernel's data types that ships at /sys/kernel/btf/vmlinux) and CO-RE (Compile Once, Run Everywhere). Together they let a single prebuilt tool adjust itself to whatever kernel it lands on, without you dragging kernel source headers onto every host. That's what the newer libbpf-tools binaries rely on, and it's why one small static tool can now work across a fleet of mismatched kernels.
Catching An Attacker In The Act
Put two of these together and you can watch an intrusion unfold in real time. execsnoop shows the shell and its odd children. tcpconnect shows every new outbound connection the instant it's made, which is how you spot a reverse shell calling home or a beacon phoning a command-and-control server.
Line one is the giveaway. A python3 process, not your app, opening a connection to a random external address on port 4444 (a common default for reverse-shell frameworks). Your nginx talking to Postgres on 5432 is normal. The metadata call on port 80 you now recognize. One tool, and the anomaly stands out from the routine.
One honest caveat before you go wild with it. Almost no overhead is not the same as zero overhead. Bolt a uprobe or kprobe onto a function that fires millions of times a second (a hot lock, a per-packet handler) and you pay a small cost on every one of those calls, enough to measurably slow the target. Prefer tracepoints over kprobes, keep the probe's action cheap, and try it on a canary host before you point it at the busiest box you own.
Check Your Host Can Run It
Before you reach for any of this in the middle of an incident, confirm the host can actually load it. You want a reasonably modern kernel (bcc needs roughly 4.9 or newer, and the best CO-RE tooling wants a 5.x kernel), the BTF file present, and the privilege to load programs.
A modern kernel, the BTF file in place, and BTF compiled in. This host is ready. If that file is missing or the kernel is ancient, the tools may refuse to load. On a locked-down or old box, fall back to perf sampling and the /proc-based classics. And hold onto the honest limit here: every tool in this lesson observes. It finds the problem fast and points a finger right at it. Designing and applying the fix is still your job.
execsnoop-bpfcc from strace -f on the parent and from a one-second ps loop?-f flag exists to do exactly that. What ptrace costs you is speed, not visibility into children.tcpconnect prints three new outbound connections on a web host: python3 to 198.51.100.23 on port 4444, curl to 169.254.169.254 on port 80, and nginx to 10.0.0.9 on port 5432. Which line do you chase first, and for what reason?Try this
Work through “Check Your Host Can Run It” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: loading BPF is close to being root. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.