The USE method & finding bottlenecks
A repeatable way to locate the constraint.
Most performance work goes sideways the same way. Something is slow, so you log in, and within a minute you are already tuning: bumping a thread pool, adding memory, flipping a low-level system setting you read about once. You are guessing. Sometimes the guess lands and you look like a hero. More often it costs you an afternoon, because you spent it tuning a resource that was never the problem. The USE method is how you stop guessing and find the actual constraint first.
Three Questions, Every Resource
Start at a supermarket checkout, because the whole method fits there. Utilization is how many lanes are open and actively ringing up customers. Saturation is the line waiting behind them, the shoppers who want a lane and cannot have one yet. Errors are the transactions that fail: a declined card, a register that crashes, an item that will not scan. Ask those three questions about a checkout and you have the USE method whole: Utilization, Saturation, and Errors. You ask them about every physical resource the machine leans on, the CPU (central processing unit, the chip that actually runs your code), memory, disk, and network. And here is the part people miss. A store can be fully utilized, every lane busy, and still be perfectly healthy, as long as nobody is standing in line.
Now the same idea in kernel terms, where the kernel is the core part of the operating system that talks straight to the hardware. Utilization is the fraction of an interval a resource was busy doing work; a disk that spent 90% of the last second servicing requests is 90% utilized. Saturation is the work that piled up because the resource could not keep pace: the run queue of processes wanting a CPU, the depth of the disk's input/output (IO) request queue, the memory the kernel had to push out to swap (disk space it uses as overflow when physical memory fills). Errors are countable failures: a failed disk write, a dropped packet, a process the kernel killed for running the box out of memory. Of the three, saturation tracks pain most closely. Full utilization with no queue is a resource working hard and coping fine. A queue means something is waiting, and waiting is what your users actually feel.
The First Sixty Seconds
Before you reach for tracers and flame graphs (heavier tools that visualize exactly where time goes), spend one minute getting oriented. Brendan Gregg, the performance engineer who created the USE method and later wrote Netflix's sixty-second triage checklist, teaches a fixed opening sequence of cheap commands. Depth is not the goal here. Direction is. You want to know which resource deserves the next hour of your attention before you sink the hour.
Those three numbers are the load average over the last one, five, and fifteen minutes. Count them the way you would count people in the store: everyone who wants a register, both the shoppers already at one and the shoppers stuck in line behind them. Read left to right for the trend. 8.10, then 5.22, then 3.15 means the most recent minute is the busiest of the three, so load is building rather than draining. This is a four-core box, so a load of eight sounds alarming. Hold that thought. Load average is the single most misread number in Linux, and the next command tells you what it actually counts.
Ignore the first row; it is an average since boot, not the live picture. Watch the rows underneath. Two columns under procs matter: r is the run queue, the count of tasks that want a CPU right now, and b is tasks blocked waiting on IO, mostly disk and network. Under cpu, wa is iowait: the share of time the processor sat idle with nothing to do but wait for storage. Here r stays at one or two while wa holds above fifty. So the CPU is not the bottleneck. It is standing around waiting for the disk to answer. si and so, swap in and swap out, are both zero, so memory is not thrashing either. The story already points hard at storage.
mpstat breaks the same picture out per core, and that matters because an average hides a single-threaded bottleneck. One core pinned at 100% while the other three sit idle averages out to a comfortable-looking 25%, and you would never feel the pain in the summary. Here every core shows the same high %iowait, which confirms the load is spread and storage-bound, not stuck in one thread. If instead you saw a single CPU at 100% %usr with the rest idle, you would go hunting for a process trapped on one thread.
dmesg is the E in USE. The kernel logs its worst news here: storage failures, filesystem corruption, and the out-of-memory (OOM) killer choosing a process to sacrifice when memory runs dry. The -T flag prints human-readable timestamps instead of raw seconds since boot. One glance tells you whether you are chasing a tuning problem or a dying disk. A real IO error in this log rewrites the whole investigation, because no amount of tuning fixes hardware that is on its way out.
Saturation You Can Actually Measure
Utilization tells you a resource is busy. It does not tell you anyone is suffering for it. For that you want saturation, and modern Linux hands you a direct readout the old tools never had: Pressure Stall Information (PSI), living under /proc/pressure. Think of it as the store measuring how much of the last few minutes shoppers spent frozen in line instead of moving. PSI answers one blunt question per resource. What share of recent wall-clock time did tasks spend stalled, unable to make progress because they were stuck waiting on this resource?
Two lines, two meanings. some is the percentage of time at least one task was stalled waiting on IO. full is the percentage of time every non-idle task was stalled at once, which means the machine did no useful work at all while it waited. The three numbers are averages over the last 10, 60, and 300 seconds; total is the raw count of stalled microseconds since boot. An io some of 57 says that for well over half of the last ten seconds, something was stuck on the disk. That is saturation stated as a plain percentage, and it drops onto the USE method cleanly. Matching files sit at /proc/pressure/cpu and /proc/pressure/memory. When you want to alert on 'is this resource actually hurting,' PSI usually beats utilization as a signal, because it measures the waiting, not the busyness.
Name The Process
USE finds the constrained resource. It does not hand you the culprit using it. Once the io pressure line is screaming, you switch questions, from which resource to who is holding it down. iostat gives you the per-device view.
Most of those columns are noise during triage. Three earn your attention. %util is the share of the interval the device had at least one request in flight (utilization). aqu-sz is the average number of requests sitting in the queue (saturation). w_await is the average milliseconds a write waited, queue time included (latency, the number your users actually feel). Here sda sits at 98% util with a queue depth near ten and writes waiting almost 47 milliseconds each. The disk is pinned. Now go find the writer.
pidstat -d attributes disk throughput to individual processes. kB_wr/s is kilobytes written per second, and iodelay is how long that process sat blocked on IO, counted in clock ticks. postgres is writing eight megabytes a second and soaking up nearly all the iodelay. There is your answer: the database is the load, not the operating system and not the kernel. iotop shows the same per-process disk view in a live, top-style display if you would rather watch it move. From here the fix is a database question, a missing index driving full-table scans, an over-aggressive checkpoint, a runaway backfill, and you can go have that conversation carrying evidence instead of a hunch.
What An Attacker Looks Like Through USE
The reason this sweep is worth burning into muscle memory is that it does not care whether the cause is a bad query or a bad actor. Both surface the same way, as a saturated resource. An unauthorized coin miner does not announce itself. It shows up as CPU utilization pinned near 100% with a run queue climbing behind it, and when you ask who, you find a process with an odd name burning every core.
%CPU here totals across cores, so 391 on a four-core box is a single process eating nearly the whole machine. The name kdevtmpfsi is a well-known miner that dresses itself up to look like a kernel thread; a genuine kernel worker never burns user-space CPU like this. The same sweep catches other attacks in their native USE signature. A memory-exhaustion bug or a fork bomb shows up first as saturation, swap in and out climbing in vmstat, and then as errors, OOM kills landing in dmesg. A volumetric network flood shows up as the interface saturating and receive drops climbing, which you read straight off the counters.
On the RX (received) side, dropped climbing while errors stays at zero is the classic shape of a receive queue filling faster than the kernel can drain it, exactly what a flood or a misbehaving client produces. You did not set out to find an intrusion. You ran a performance sweep, and the saturated resource pointed right at it. That is the quiet security payoff of doing triage by method instead of by hunch.
One last habit. After you change something, run the exact same USE sweep again and watch the one number that was saturated. If the io pressure some line and iostat's aqu-sz both fall, you fixed the right thing. If they do not move, you did not, no matter how sensible the change looked on paper. The method that found the bottleneck is also the proof that your fix worked.
Try this
Work through “What An Attacker Looks Like Through USE” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: load average is not CPU usage. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.