Disk & network I/O

iostat, iotop, ss, and where time goes.

When the USE triage points at disk or network, a second set of tools finds where the time goes. For disk, iostat shows per-device utilization and, crucially, latency and queue depth; iotop attributes IO to processes. The key metric is not throughput but %util and await (average wait per IO) — a device at 100% util with rising await is saturated, and whatever is queued behind it is what feels slow.

$ iostat -xz 1 3
Device   r/s   w/s   rkB/s   wkB/s  await  aqu-sz  %util
sda      12    340    512   44032   28.4     4.10   99.2
#                                    │        │       └ device ~100% busy = saturated
#                                    │        └ avg queue depth (things waiting)
#                                    └ avg ms per IO (latency) — climbing = trouble
$ iotop -o                       # which processes are doing the IO right now

Network throughput and connections

For network, ss (from the essentials course) inventories connections and their states; a pile of connections stuck in a particular state is often the clue. For throughput and errors, ip -s link shows per-interface packet and error counts, and tools like iftop or nload show live bandwidth. As with disk, you are looking for saturation (link near capacity) and errors (drops, retransmits) rather than just raw bytes.

$ ss -s                          # summary: total sockets, by state
$ ss -tn state time-wait | wc -l # a flood of TIME-WAIT? (connection churn)
$ ip -s link show eth0           # per-interface packets AND errors/drops
    RX: bytes  packets  errors  dropped
    ...        ...      0       142     # nonzero drops = a real network problem

The deleted-open-file disk trap

A recurring operational puzzle belongs here: df says a disk is full, but du cannot find the space. The usual cause is a file that was deleted while a process still holds it open — the space is not reclaimed until that process closes the file (or is restarted). lsof surfaces these, and it is one of those problems that looks baffling until you know the pattern, then takes thirty seconds to diagnose.

$ df -h /var        # 100% full
$ du -sh /var/*     # ...but nothing adds up to that
$ sudo lsof +L1 /var | head    # files with link-count 0 (deleted) still held open
COMMAND  PID   USER   ...  NLINK  NAME
rsyslogd 812   syslog ...  0      /var/log/old.log (deleted)   # restart it to free the space