CPU & memory: load, vmstat, pressure

Read load average, PSI, and memory reality.

CPU and memory are the first resources to characterize, and Linux gives you precise, live views once you know which number means what. For CPU, the run queue (how many threads want to run) versus the core count tells you about saturation, and the split between user, system, and IO-wait time tells you what kind of work is consuming it. vmstat and mpstat are the workhorses; pidstat attributes it to processes.

$ vmstat 1 3
procs -----------memory---------- ---swap-- -----io---- -system-- ------cpu-----
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st
 6  1      0 210344  18232 3401220    0    0    12    88 2201 4410 71  9 12  8  0
#  └run queue        └ swap in/out (nonzero = memory pressure)   └us/sy/id/wa: user/sys/idle/iowait
$ pidstat 1 3                 # per-process CPU over time — who is actually burning it?

Memory: cache is not "used"

Linux deliberately uses spare RAM as disk cache, so the naive "free" number looks alarmingly low on a healthy system — that memory is reclaimable the instant a program needs it. The number that matters is "available" (free -h), which accounts for reclaimable cache. Real memory trouble shows as swapping (si/so in vmstat consistently nonzero) or, at the extreme, the OOM killer terminating a process, which you see in dmesg. Judge memory by available and by swap activity, not by "free."

$ free -h
               total        used        free      buff/cache   available
Mem:            15Gi       6.2Gi       0.4Gi         8.4Gi        8.1Gi
#  "free" is tiny (0.4Gi) but "available" is 8.1Gi — the system is HEALTHY (cache is reclaimable)
$ dmesg -T | grep -i "killed process"   # did the OOM killer fire? (real memory exhaustion)

Pressure Stall Information (PSI)

Modern kernels expose PSI — a direct measure of how much time tasks spent stalled waiting for CPU, memory, or IO, in /proc/pressure/. It answers "is this resource actually causing delays?" more honestly than utilization, because a resource can be 100% utilized without anyone waiting, or cause stalls before it looks full. PSI is the cleanest signal of real contention and is what container and cgroup resource management increasingly use.

$ cat /proc/pressure/cpu
some avg10=12.44 avg60=8.90 avg300=5.10 total=...   # % of time SOME task stalled on CPU
$ cat /proc/pressure/io
some avg10=42.10 ...    # high IO pressure = tasks are waiting on disk — the real bottleneck