Tracing with perf & eBPF tools
See what a slow process is actually doing.
A process is pinned at 100% CPU (central processing unit, the chip that actually runs your code), or it's stuck and answering nothing, and top (the standard live view of running processes) only tells you it's busy, not what it's busy doing. Tracing is how you get behind the process and watch the real work: which functions burn the processor, which requests to the operating system it's blocked on, where the milliseconds actually go. There are three tools you reach for, in a fairly fixed order, and each one hands a defender something different.
Every slow program is waiting on the kernel
A program can do a lot on its own: arithmetic, shuffling bytes around in its own memory, looping. But the moment it needs anything from the outside world, reading a file, sending a network packet, asking for more memory, checking the clock, it has to ask the kernel (the core part of the operating system that controls the hardware and guards it from the programs on top). That request is a system call, or syscall for short. The program is like a tenant in a locked apartment, and the kernel is the building manager: anything involving the world outside the apartment means passing a note under the door and waiting for the manager to come back. Almost every slow program is slow because it's stuck waiting on one of those notes, a disk read, a lock, a reply from another server. Find the note it's stuck on and you've found the stall.
strace: a log of every request to the kernel
strace (short for 'system call trace') stands at that door and writes down every note. Point it at a running process with -p and the process ID, add -T to record how long each call took, and you get a live transcript.
Read that middle line. The process sat in futex (fast userspace mutex, the kernel primitive that locks live on top of) for 2.1 seconds. Everything else took microseconds. That single number is your answer: this process isn't computing anything, it's waiting on a lock somebody else is holding. No guessing, no reading source code, the transcript tells you exactly where the time went.
The -c flag skips the line-by-line firehose and prints a summary sorted by time spent. Here 91% of the wall-clock time is gone to futex: it's lock contention, not slow disk or slow network. Two more flags earn their keep. -f follows the process into any child it forks, so you don't lose the trail when a program spawns worker processes. And -e trace=openat,connect narrows the flood down to only the syscalls you care about, here file opens and outbound connections, which is often how you catch a program touching a file or an address it has no business touching.
Why strace can take down a busy service
Here's the catch, and it bites hard in production. strace works through a kernel feature called ptrace (process trace), the same machinery a debugger uses. ptrace doesn't watch from across the room. It stops the traced process on every single syscall, hands control to strace, waits for strace to record the call, then wakes the process back up. Every note under the door now goes through a middleman who pauses the tenant twice per note. On an idle or stuck process you'll never feel it. On a web server doing tens of thousands of syscalls a second, you've added two context switches (the CPU saving one task's state and loading another's) onto every one of them, and throughput can drop by 10x or worse.
perf: sampling the CPU instead of logging it
perf (the Linux kernel's built-in sampling profiler) takes the opposite approach. Instead of recording every event, it samples. A few thousand times a second it interrupts the CPU, notes which function is running at that instant, and adds up the tally. Think of a factory supervisor who doesn't shadow any one worker but glances up 4,000 times a second and jots down which station is active; after a minute, the station that showed up in the most glances is your bottleneck. Because perf only peeks periodically instead of intercepting every call, the overhead is small enough to run on a live box. perf top gives you that tally in real time, across the whole system.
A [.] means the function runs in userspace (ordinary program code); a [k] means it runs in the kernel. A third of the CPU is in compute_checksum, and it's your own binary. That's the hot loop, found without touching the source or adding a single print statement. To profile one command from start to finish rather than the whole system, record it and read the result back.
The -g flag captures call graphs, so perf report can also show you who called the hot function, not only which function was hot. One warning about symbols: if perf shows raw hex addresses instead of names like compute_checksum, the binary was stripped of its symbol table. Install its matching debug package (on Debian and Ubuntu these are the -dbgsym or debuginfo packages) so perf can translate addresses back into function names. When you need raw counters instead of a profile, perf stat -- ./cmd prints cache misses, branch mispredictions, and context-switch counts for a single run.
eBPF: sensors you load into the kernel
eBPF (extended Berkeley Packet Filter, the same technology you may have met in the security course) is the newest of the three and, for a defender, usually the most useful. Ignore the packet-filter name, it's historical. What eBPF actually gives you is the ability to load a tiny program of your own into the running kernel, attach it to a specific event, and have it run every time that event fires, safely. Before your program is let in, a verifier checks that it can't loop forever, can't crash the kernel, and can't read memory it shouldn't. It's the difference between tailing a suspect all day (strace) and bolting a motion sensor to one specific door that pings you only when that door opens (eBPF). Almost no overhead, and it runs down in the kernel where a watched process can't see it and can't switch it off.
You rarely write these programs by hand. Two toolkits ship ready-made ones. bcc (the BPF Compiler Collection) is a set of purpose-built tools; on Debian and Ubuntu they come from the bpfcc-tools package and are named with a -bpfcc suffix. bpftrace is a small language for writing one-liners when no canned tool fits. Start with execsnoop, which prints every new process the moment it executes.
A backup script is fine. The last line is not: a curl to a bare IP address, no hostname, writing a hidden file (the leading dot) into /tmp. That's the shape of a script pulling down a second-stage payload, and execsnoop caught it live because the syscall that starts every program (execve) fired and your sensor was attached to it. There's no polling gap for an attacker to slip through the way there is with a process list you check every few seconds.
biolatency draws disk I/O (input/output) latency as a histogram, one bar per speed bucket, in microseconds. Most reads land near half a millisecond, but notice the small cluster up at 16-32 milliseconds. Two separate humps mean a bimodal distribution: fast requests served from cache and a tail of slow ones hitting real disk. An average would have hidden that tail completely; the shape is the whole point.
That one line answers 'who is hammering the disk with file opens?' It attaches to the openat syscall and counts hits per program name (comm is the kernel's short name for the command). find opened 20,463 files while nginx opened barely a thousand. There's your answer, in a single command you can type from memory.
What a defender actually watches
Line these tools up and they stop being a performance kit and start being a detection kit. execsnoop shows every program that runs, which is how you spot a reverse shell or a cron job spawning a stranger. opensnoop-bpfcc -n sshd shows every file a named process opens, which is how you catch something reading /etc/shadow or an SSH private key. tcpconnect-bpfcc shows every outbound TCP (transmission control protocol) connection the moment it's made, which is how you catch a process beaconing to a command-and-control (C2) server it phones home to for orders.
There's that same curl again, this time caught reaching out to 198.51.100.7 on port 443. The real value for a defender is where these sensors live. They run inside the kernel, below the process being watched. An attacker running as an unprivileged user can't tell the sensor is there and can't turn it off, unlike a log file a foothold can quietly edit. To make that watch permanent instead of an interactive session, the same event streams feed tools like Falco and Tetragon, which turn eBPF probes into standing alert rules.
When you think you've fixed the stall, prove it with the same tool that found it. Rerun strace -c and watch the futex line's total seconds collapse toward zero. Rerun biolatency and check the slow 16-millisecond hump is gone from the histogram. Rerun perf top and confirm compute_checksum has dropped out of the top spot. A trace that shows the time is actually gone is the only proof that your change did anything.
Try this
Work through “What a defender actually watches” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: strace can turn a slow service into a dead one. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.