CPU & memory: load, vmstat, pressure
Read load average, PSI, and memory reality.
A load average of 6 sounds scary until you know how many people are working the counter. One number with no context barely tells you anything. That holds for almost every CPU (central processing unit, the chips that actually run your code) and memory figure on a Linux box: the raw number is meaningless until you know what it counts and what you are measuring it against. This lesson walks through the small set of numbers that tell you whether a machine is really in trouble, and the ones that look alarming while being completely normal. For a defender, that gap is the difference between chasing a ghost and catching a cryptominer.
What Load Average Actually Counts
Load average is the number you see in uptime, top, and every dashboard, and it is the one people get wrong most often. A small coffee shop has a few baristas behind the counter. At any moment some customers are being served and others are stuck in line. Load average is roughly the size of that whole crowd, served plus waiting, smoothed out over time. The three figures are exponentially weighted moving averages over the last 1, 5, and 15 minutes, which is a formal way of saying recent seconds count for more than older ones. The comparison that matters is load against core count. A load of 6 on a 4-core box means more work wants to run than you have baristas, so a line forms. A load of 6 on a 64-core box is a slow afternoon.
Here is where even experienced people get caught. Linux does not only count tasks that want the CPU. It also counts tasks stuck in uninterruptible sleep, known as D state, which almost always means they are blocked waiting on storage or the network. So a machine can show a load of 8 while the CPU sits 90% idle, because the load is not coming from compute at all. It is coming from processes frozen while a slow disk catches up. When load is high, your first question is always the same: is this CPU demand, or is this IO (input/output, the work of reading and writing data to disk or network) waiting? Every tool below is there to answer it.
The fourth field in /proc/loadavg, 2/1043, is the number of tasks running or waiting to run at this instant, over the total number of processes and threads that exist on the box. The last value, 48123, is the process ID (PID) of the most recently created process. With four cores and a one-minute load of 6.12, this machine is oversubscribed on paper. Whether that actually hurts depends on why, and vmstat is where you start to see it.
vmstat: The One-Second Dashboard
vmstat (virtual memory statistics) is your live feed. Give it an interval and a count, and every line after the first is a fresh snapshot of that interval. Treat the first data line as scenery: it is an average since boot, not the current moment, so skip it. Read the rest the way you read a car dashboard while driving, glancing at what is happening right now.
The columns that earn their keep: r is the run queue, the number of threads that want the CPU this instant, read against your core count exactly like load. b is the tasks blocked in that uninterruptible D state. si and so are memory swapped in from disk and out to disk (steady nonzero here is the alarm we get to shortly). us, sy, id, and wa are the percentages of CPU time spent in user code, kernel code, idle, and waiting on IO. st is steal time, the slice of time a hypervisor took your virtual CPU away and handed it to someone else. On a cloud host, steady st points at a noisy neighbor or CPU throttling, and no amount of application tuning will fix it. For a defender, a box that suddenly pins us above 90 with a climbing run queue and nothing in your deploy history to explain it is worth a hard look.
Whose CPU Is It
Knowing the CPU is busy is not the same as knowing who is burning it. Picture a kitchen at full tilt. mpstat (multiprocessor statistics) tells you which burner is roaring: it breaks usage down per core, so you can see whether one core is pegged (a single-threaded hog) or all of them are hot (a parallel job or a genuinely busy server). pidstat (per-process statistics) tells you which cook is standing over that burner: it attributes CPU to individual processes, handing you a name and a PID. Run them together and you catch the classic intrusion tell, a process you do not recognize eating a whole core.
CPU 0 is pinned at 98% user time while the other three coast. That shape is a single-threaded process, and mpstat can see it but cannot name it. pidstat can.
There it is. xmrig, a well-known cryptocurrency miner, running as root and holding 99% of a core, while node, your actual web app, ticks along at a normal 30%. On a server whose job is serving web traffic, a miner sitting on a full core is an incident. You stop tuning and open a case. pidstat handed you the PID and the command name on a single line, which is exactly what you carry into that investigation.
Memory: Cache Is Not Used Up
Now memory, and the biggest misconception in Linux operations. A good cook keeps the ingredients they reach for out on the counter instead of running back to the pantry every thirty seconds. The counter looks full, but the instant they need the space, they sweep it clear. Linux treats your RAM (random-access memory, the fast working memory the CPU reads from directly) the same way. Any RAM a program is not holding, the kernel (the core of the operating system that talks to the hardware) fills with a cache of recently read files, because empty RAM helps no one. That cache is reclaimable the moment a program asks for memory. So the free column in free -h can look terrifyingly small on a machine that is perfectly healthy.
Look at the numbers. free reads 0.4Gi, which would send you into a panic if you stopped there. But available reads 8.1Gi, and available is the honest one: it estimates how much memory a new program could grab right now, counting all that reclaimable cache. The kernel publishes the same figure as MemAvailable in /proc/meminfo. Judge memory health by available and by swap activity, never by free alone. If available is comfortable, the box has memory to spare no matter how alarming free looks.
Pressure Stall Information: Measuring the Wait
Utilization has a blind spot. A resource can sit at 100% busy while nobody actually waits on it, and a resource can cause real pain long before it looks full. A kitchen running flat out is fine as long as every order still goes out on time. The number you actually care about is how long orders sat waiting. That is what Pressure Stall Information (PSI, a kernel feature that measures time lost to resource contention) gives you, across three files under /proc/pressure.
Each line reports the percentage of time tasks were stalled waiting on that resource, averaged over 10, 60, and 300 seconds, followed by a running total in microseconds since boot. The some line means at least one task was stalled. The full line means every task that could have run was stalled at the same moment, so nothing was getting done, which is the more serious of the two. CPU reports only some at the system level, because a whole machine does not fall into a full CPU stall the way it can for memory or IO. Here, io some sits at 42% over the last 10 seconds: for nearly half that window, something was stuck waiting on the disk. That is a real bottleneck, and it explains a high load average sitting next to idle CPUs far better than any utilization figure could.
PSI also works per cgroup (control group, the kernel feature that fences a set of processes together and caps what they can use), which is how containers and systemd (the program that starts and supervises services on a modern Linux) units get their own pressure readings. Point it at one service and you see exactly what that service is waiting on.
When Memory Runs Out
When available finally runs dry, two things happen, in order. First the kernel starts swapping: copying pages of memory out to disk to free up room. You see it as steady nonzero si and so in vmstat, with swpd climbing. A little swap sitting idle is harmless. Continuous swapping is the cliff, because disk is thousands of times slower than RAM, and a box that leans on swap under load thrashes and slows to a crawl. Size the RAM, or cap the workload, so the machine never depends on swap to function. If memory keeps shrinking past that point, the kernel calls in the OOM (out of memory) killer. Think of it as a bouncer in a dangerously overpacked room: it picks one person by a score and throws them out so everyone else can keep breathing. It chooses a process by its oom_score and terminates it to save the system. The evidence lands in the kernel log.
That one line is the whole story of a kill: which process (python3, PID 4823), how much it was holding (anon-rss of roughly 7.4Gi of anonymous memory, meaning heap and stack, the kind not backed by any file), and its oom_score_adj, the dial that biases the killer toward or away from a given process. If a service keeps getting OOM-killed, this log is your proof that memory ran out, rather than your code crashing on its own.
Modern Ubuntu and Fedora ship systemd-oomd, a userspace daemon that watches PSI and steps in before the kernel's hard OOM killer ever fires. Rather than waiting for memory to hit zero, it kills a whole cgroup once memory pressure or swap usage crosses a threshold you configure. That is usually what you want, but it means a process can be terminated while free memory still looks fine, so read its log before you blame your application.
Wire the pressure files into your monitoring, not the free number. An alert on io.pressure some crossing, say, 40% for a minute tells you a host is genuinely hurting, cleanly, with almost no false alarms. Then when load spikes, you already know whether to reach for pidstat and hunt a process or go check the disk. The metric that fires is the one that points you straight at the cause.
Try this
Work through “When Memory Runs Out” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: systemd-oomd Can Kill While RAM Looks Fine. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.