Disk & network I/O
iostat, iotop, ss, and where time goes.
The USE method (Utilization, Saturation, Errors) tells you which resource is under pressure. It does not tell you which of your programs is causing it, or where the time is actually going. When the arrow points at the disk or the network, you switch tools and start hunting for the place the seconds are disappearing.
First, the mistake almost everyone makes. They watch throughput, the megabytes per second crossing the disk or the wire. It looks calm, so they decide the I/O (input/output, data moving to and from storage or the network) is healthy. Meanwhile the application is still crawling. Throughput was the wrong number to trust.
A supermarket checkout makes the difference obvious. Throughput is how many carts the store rings up every hour. Latency is how long you personally stand in line holding your milk. The store can scan carts at a furious pace (high throughput) while your one line barely moves (high latency), because twenty people are ahead of you. Users feel latency. They never once see throughput.
Reading iostat Without Fear
iostat (input/output statistics) reports what each disk device is doing. The extended view is wide and a little intimidating, which is why people bounce off it. You can ignore most of the columns. Four of them carry the whole story. The first report iostat prints covers everything since boot, so it blends a calm night and a busy afternoon into one average; the reports after it are the live interval you actually care about.
Read the second report, the live one. r/s and w/s are IOPS (input/output operations per second): 12 reads and 340 writes finished each second. r_await and w_await are the average time, in milliseconds, a request waited from the moment it was handed to the device until it came back, queue time plus service time together. Modern iostat splits the old single await column into a read number and a write number, and it dropped svctm (service time) entirely, so old cheat sheets that tell you to watch await and svctm are out of date. aqu-sz is the average queue length, the number of requests in flight or waiting at any instant. %util is the fraction of time the device had at least one request in flight.
Now the reading is easy. wkB/s says 43 MB/s, which sounds gentle. But %util is pinned at 99, aqu-sz sits near 11 (about eleven requests waiting or in flight at any instant), and every write waits 31 ms. That is a saturated disk. The throughput number stayed calm while the latency and the queue did the damage, exactly like your milk line.
Which Process Is Holding the Disk Under
iostat tells you the disk is drowning. It does not tell you whose hand is holding it under. iotop (I/O top) is the culprit finder. It attributes live read and write bytes to each process. Run it as root, and add -o so it lists only the processes actually doing I/O right now instead of every idle thread on the box.
The checkpointer thread is spending 92% of its time blocked waiting on I/O, and it is pushing most of that 43 MB/s. TID is the thread ID, the per-thread cousin of the PID (process ID). If you would rather log this over time than watch it live, pidstat -d 1 from the same sysstat package prints per-process disk stats every second without the full-screen interface.
sudo sysctl kernel.task_delayacct=1, and persist it in /etc/sysctl.d/ if you want it after reboot. Without it you get byte counts but not the wait percentages, which is exactly the signal you came for.The Network Side: Connection States Are a Diagnosis
A telephone switchboard makes the network states click. Every call is a line in some state: ringing, connected, being hung up, or waiting a polite moment before the plug is finally pulled. TCP (Transmission Control Protocol, the reliable delivery layer under most traffic) works the same way, and a pile of connections stuck in one state is usually the whole clue. ss (socket statistics) is the switchboard operator's view. Start with the summary.
Three states earn their own attention. TIME-WAIT is normal: it is the polite pause (about 60 seconds on Linux) after your side closes a connection, and a huge count means high connection churn, short-lived connections opened and closed in a storm, which can exhaust the pool of local ports. CLOSE-WAIT means the other end hung up and your application never called close, so a growing count is almost always a file-descriptor leak in your own code. And SYN-RECV is the one a defender watches closely.
SYN (synchronize) is the first packet of the TCP handshake. SYN-RECV means your server answered a SYN, sent its reply, and is waiting for the final acknowledgement that never arrives. Picture a prank caller who dials, waits for you to say hello, then stays silent and ties up your line. Thousands of half-open connections from scattered, unrelated source addresses is a SYN flood.
1287 half-open connections, each from a different, unrelated peer, all aimed at port 443. That is the shape of a spoofed SYN flood, not real users. The kernel's defense is SYN cookies, which you confirm with sysctl net.ipv4.tcp_syncookies (it should read 1 on a modern box). With cookies on, once the half-open backlog fills, the server stops storing state for each half-open connection and encodes it into the reply instead, so it rides out the flood.
Where Network Latency Actually Hides
Connection counts tell you the shape of the traffic. To see why a single connection feels slow, read the delivery receipt for that one socket. ss -i prints the kernel's internal TCP numbers per socket: RTT (round-trip time, how long a packet takes to go out and be acknowledged), the congestion window (how many packets the kernel dares to have in flight before it waits for acknowledgements), and the retransmit count (packets it had to send twice because the first copy was lost).
This connection is sick. rtt is 412 ms with big variance, retrans:0/18 means eighteen retransmissions over its life, and cwnd has collapsed to 2 packets because the kernel keeps hitting loss and backing off. The path is lossy or congested, and no amount of staring at the database will fix a network dropping roughly one packet in seventy. For the system-wide version of that retransmit signal, nstat (network statistics) reads the kernel's TCP counters.
TcpRetransSegs climbing steadily means packet loss somewhere on your paths. TcpExtListenDrops ties straight back to the flood section: it counts connections the kernel had to drop because the listen queue was full, which is what an undersized listen backlog does under a SYN storm or a genuine traffic spike.
Errors and Drops at the Interface
A NIC (network interface card) is a mail sorting room with a small inbox. When packets arrive faster than the CPU can pick them up, the inbox (the driver's receive ring buffer) overflows and letters hit the floor. ip -s link shows those counts per interface.
errors at 0 and dropped at 0, but overrun at 142, points at the inbox, not the cable. The overrun column is the card saying its receive ring (the small FIFO the hardware fills before the CPU drains it) ran out of room, so packets fell on the floor because the host could not keep up. Confirm it with the driver's own counters, ethtool -S eth0 (ethtool is the NIC's control tool; look for rx_fifo_errors, rx_missed_errors, or rx_no_buffer_count, depending on the driver), and if the number keeps climbing, enlarge the ring with ethtool -G eth0 rx 4096. Nonzero errors instead of overruns would point the other way, at the physical layer: a bad cable, a dying optic, CRC failures (CRC, the cyclic redundancy check that catches corrupted frames). For a live bandwidth picture while you work, iftop, nload, or bmon show per-connection and per-interface throughput in real time.
The Full Disk That du Cannot Explain
A library book gets pulled from the catalog, but nobody returns the physical copy, so its shelf slot stays occupied. The catalog swears the slot is free. This is the classic disk puzzle: df (disk free) says the filesystem is full, du (disk usage) adds up the visible files and finds far less. A file was deleted while a process still held it open, and Linux does not reclaim the blocks until the last open handle to it closes.
df sees 47G used, du can only find 21G, and lsof explains the gap. NLINK 0 (no directory entry points at this file anymore), the (deleted) tag, and a 25G size: a deleted log that rsyslogd still writes to on file descriptor 7 (fd, the small number the kernel uses to track an open file). Restarting the process frees the space. If you cannot restart it right now, you can truncate the file through the still-open descriptor in /proc (the kernel's live view of every process, laid out as files).
For a defender, this same mechanism does double duty as an incident-response signal. A process whose binary shows up as (deleted) in ls -l /proc/<pid>/exe is a hallmark of malware that unlinks itself from disk to hide while it keeps running from memory. The blocks stay pinned exactly like that log did, and you can recover the evidence straight from the open handle with sudo cp /proc/<pid>/exe /tmp/recovered.bin before you kill the process.
truncate -s 0 through /proc frees the blocks instantly, but it also destroys the log's contents and does not reset the writer's offset. The process keeps writing at byte 25000000000, so the file becomes sparse (a huge reported size backed by almost no real blocks) and any tool reading it may hit a wall of null bytes. Restarting the writer is the clean fix; reach for the /proc truncate only when you cannot afford a restart and you have already saved anything you need.Build the reflex and it pays off on every incident. Before you blame the disk, read await and aqu-sz, not the byte rate. Before you blame the network, read the connection states and the retransmit counter, not the byte rate. And when df and du disagree, reach for lsof +L1 before you delete things at random. The throughput graphs will look calm and lie to you. The latency, the queue, and the connection state will not.
Try this
Work through “The Full Disk That du Cannot Explain” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.
Takeaway
The trap worth remembering here: %util near 100 does not mean saturated on SSDs. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.