Flame graphs & CPU profiling

perf + FlameGraph: see the hot path at a glance.

Your server is pinned at 100 percent CPU (central processing unit, the chip that actually runs your code) and you cannot say why. The top command names the busy process. It says nothing about which lines of code inside that process are eating the cycles. A flame graph answers that question in one picture.

A flame graph works like an itemized receipt for time. A plain profiler hands you a total: the program spent 40 percent of its time in some function. A flame graph itemizes that total by the exact chain of calls that led to it, the way a receipt breaks a big bill into line items so the one expensive thing jumps out. perf (a command-line profiler that ships with Linux and rides on the kernel's built-in sampling machinery) gives you the hot functions as a flat list. A flame graph stacks the whole call path, so you see which function is hot and the exact route the program took to reach it. Brendan Gregg's FlameGraph scripts turned this into the standard way defenders and operators read a profile.

What perf is actually doing

Profiling here means sampling, and sampling is guesswork done carefully. A factory supervisor cannot watch every worker every second, so a hundred times an hour they walk the floor and note what each person is doing at that exact instant. Do it long enough and the tallies get accurate. If 40 percent of the snapshots catch someone at the packing station, packing is roughly 40 percent of the work. perf profiles a running machine the same way. Many times per second it interrupts every CPU, records the full call stack (the chain of function calls in flight at that instant), and moves on. Thousands of these snapshots add up to a statistical map of where the time goes.

-F 99 sets that rate to 99 samples per second on each CPU. The odd number is deliberate. Sample at a round 100 Hz (hertz, meaning times per second) and you risk marching in lockstep with timers and housekeeping jobs that also fire on round intervals, which biases the tally toward whatever runs on the tick. 99 keeps your sampling out of phase with those rhythms. -a means every CPU, system-wide. -g turns on call-graph recording, so each sample captures the whole stack, from the running function down to the entry point, instead of the top frame alone. The -- sleep 30 on the end is a trick. perf records system-wide for exactly as long as that throwaway sleep runs, which hands you a clean 30-second window.

That 2 is the gate. kernel.perf_event_paranoid controls who can profile without being root. A value of 2 or higher blocks unprivileged kernel profiling, and hardened distributions ship it at 3 or higher, which shuts out unprivileged users almost entirely. That is why every command below uses sudo. It also means the ability to run a system-wide profile is a privilege worth guarding, because a profile can reveal a lot about what a box is doing.

Record, fold, render

The recipe is three moves. Record raw samples with perf, fold each stack down to one line, then render those lines into a clickable image. First, grab the tools and record a window of the live system.

Before drawing anything, take the fast text look. perf report reads the perf.data file perf just wrote and ranks functions by how many samples landed on them. This is the flat list a flame graph will later give shape to, and on a compromised box it often ends the investigation on its own.

There is your answer, and it is a nasty one. kdevtmpfsi is a name chosen to look like a kernel device thread, but it is a real piece of Linux malware from the Kinsing miner family, and cryptonight_hash is the routine that mines Monero (a privacy-focused cryptocurrency). The [.] marks user-space code, [k] marks kernel code. No amount of process-name disguise hides this, because the profile is built from the actual functions the CPU ran, not from whatever the process called itself. Now fold and render, so you can hand a picture to whoever picks up the incident.

perf script dumps every sample as text, stackcollapse-perf.pl squashes each unique stack into a single semicolon-joined line with a count, and flamegraph.pl draws those lines. The folded file is worth peeking at once, because it demystifies the whole thing. It is only stacks and tallies.

Reading the picture

The SVG (scalable vector graphics, an image format that stays sharp at any zoom and stays clickable in a browser) is one self-contained file you can email to a colleague. Reading it is the real skill, and it comes down to two rules. Width is CPU time. The wider a box, the bigger the share of samples that caught that function running, which is the same as saying it burned more processor time. Height is depth. The stack grows upward, the bottom box is where execution entered, the top box is what was actually running when the sample fired. Left-to-right order carries no meaning at all. Boxes are sorted alphabetically so that identical stacks merge into one wide bar, which is what makes a hot path fat and obvious. Read the horizontal axis as a timeline and you will draw the wrong conclusion.

So two shapes tell most of the story. A wide flat plateau at the top is a leaf function spending CPU on its own work, and that function is where you optimize. A tall narrow tower is a deep chain of calls where the cost is smeared thinly across many layers, which usually points at an expensive abstraction or a function being called far too often. In a browser the picture is live. Click any box to zoom that subtree to full width, and use the search box to light up every place a function appears so you can total a name that is scattered across the graph.

When the flame graph is garbage

Walking a stack is like following a trail of breadcrumbs back to the door. A frame pointer (a CPU register that marks where the current function's stack frame begins) is that breadcrumb. Each frame points at the one below it, so perf can walk from the running function all the way down to the entry point. Optimizing compilers love to reuse that register for real work, so at the common -O2 setting they drop the breadcrumbs (the -fomit-frame-pointer behavior). Without them, perf grabs the top function and then loses the trail, and your flame graph turns into a field of stubby [unknown] and raw hex boxes that tell you nothing. This is the number one reason a flame graph comes out useless, and it is fixable.

You have two fixes. Rebuild or reinstall the program with frame pointers kept, using -fno-omit-frame-pointer. This is why Ubuntu 24.04 and recent Fedora now compile the whole distribution with frame pointers on by default, accepting a tiny slowdown in exchange for profiles that actually work. Or leave the binary alone and unwind with DWARF (the debug-info format compilers emit; it records how to rebuild the stack even after the breadcrumbs are gone). Add --call-graph dwarf to the record, and point perf at a debug-symbol server so it can name the functions it finds.

Notice the cost. Copying stack memory blew the same 30-second capture from about 4 megabytes to nearly 59, and at high enough sample rates DWARF unwinding starts dropping samples. That is the tradeoff. It works on optimized binaries without a rebuild, but it is heavy. The -E on sudo matters too, because it keeps your DEBUGINFOD_URLS variable when the command runs as root. Without it, the symbol server is never contacted.

Interpreted and JIT (just-in-time, where code is compiled to machine instructions while the program runs) languages need one more piece. perf sees the machine code the runtime generated, but the addresses mean nothing to it unless the runtime writes out a map. Node.js does this with --perf-basic-prof, which drops a /tmp/perf-<PID>.map file (PID is the process identifier) that perf reads to turn addresses into JavaScript function names. Java gives you two routes: run perf with -XX:+PreserveFramePointer and load perf-map-agent to write that same map, or skip perf and use async-profiler, which walks Java's own frames and emits folded output directly. For Python, the practical tool is py-spy, which reads the interpreter's frames from outside the process and can write folded output straight into flamegraph.pl. Without the runtime's help, those frames stay as bare hex. A flame graph is only ever as good as the stacks feeding it, so get the symbols right before you read anything into the shape.

Off-CPU time and before-versus-after

A CPU flame graph only shows work that was running. Think of timing a delivery driver by clocking only the minutes the van is actually moving. The twenty minutes idling at a loading dock never show up, even though that idle stretch is what made the delivery late. A CPU profiler has the same blind spot. It says nothing about a request that spent two seconds parked, waiting on a lock, a disk, or a slow database reply, because a waiting thread is off-CPU and never gets sampled. That waiting is the other half of most latency problems. An off-CPU flame graph fills the gap by drawing time spent blocked instead of time spent running, built from the kernel's scheduler events rather than CPU samples. The modern way to capture it is offcputime-bpfcc from the bcc tools (the BPF Compiler Collection, a set of ready-made programs built on eBPF, the kernel's in-house engine for running small tracing programs safely inside the kernel). It emits folded stacks you feed to the same flamegraph.pl.

The other high-value move is proving a fix worked. Record a folded profile before your change and another after, then draw the difference. difffolded.pl colors what got hotter and what got cooler, so a shrunken hot path is something you can see rather than something you claim.

For a defender, that same before-and-after is your verification loop after you kill a miner or patch a runaway service. Capture 30 seconds, confirm the hot path is gone, and keep the SVG as evidence. The moment a box goes hot for a reason nobody can name, the first move is one perf record -a -g and one folded flame graph. It gives you the function that is burning CPU, along with the process it lives in, and that is usually the whole investigation.

Try this

Work through “Off-CPU time and before-versus-after” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: perf.data can hold secrets, and needs root. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related