Kernel tuning for performance

The sysctls that actually move the needle.

Your Linux kernel ships with a few hundred dials you can turn. They are called sysctls (system control settings, the kernel parameters you can read and change while the machine is running, where the kernel is the core of the operating system that manages memory, processes, and hardware). Almost all of them are set to values that work well for the general case, the same way a new oven comes pre-set to something reasonable before any baker touches it. Tuning is not turning every dial. It is finding the two or three that match your specific bottleneck, moving them on purpose, and proving the move helped.

The honest truth of performance work is that the defaults are good, and most speedups come from matching a small number of parameters to a real, measured constraint. A server carrying a thousand copied-in sysctls nobody understands is slower to reason about and easier to break, which for anyone doing security and operations work is its own kind of risk. So the loop is always the same: measure, find the constrained resource, change one thing, measure again.

How A Sysctl Actually Works

Every tunable is a light switch on a wall, and the wall is a special directory called /proc/sys (a live window into the kernel, dressed up to look like ordinary files). The kernel shows each setting as a file there. Reading the file tells you the current value, writing to the file changes it. The sysctl command is a friendlier front door to those same files, and the dotted name maps straight onto the path: vm.swappiness is the file /proc/sys/vm/swappiness (dots become slashes).

Writing directly to /proc/sys changes the value right now, but the change dies at the next reboot, the same way flipping a switch by hand does nothing to how the house is wired. To make a setting survive a reboot you write it into a file under /etc/sysctl.d/ and let the system apply it at boot. Runtime for testing, config file for permanence.

Measure Before You Touch Anything

The discipline that beats every clever setting is refusing to optimize something you have not measured. A frame for this is the USE method (check three things for each resource: Utilization, how busy it is; Saturation, how much work is queued waiting for it; and Errors, how often it fails). Walk CPU (central processing unit, the chip doing the actual computing), memory, disk, and network through those three questions and the real bottleneck usually names itself.

vmstat is a fast first look (it prints a line of system statistics every interval you ask for). The columns that matter most for tuning are si and so (memory swapped in from disk and out to disk per second) and wa (the share of CPU time stalled waiting on disk input/output). If si and so are stuck at zero and wa is low, swapping is not your problem, and no swappiness change will help you.

Memory And Swap

Picture RAM (random-access memory, the fast working memory the machine actually computes in) as the garage attached to a building, and swap as an overflow lot down the road. Swapping moves memory pages out to disk to free up fast memory. vm.swappiness is how eager the valet is to send cars to the overflow lot even when the garage still has spaces. The default is 60. On a server with plenty of memory that is often too eager, because pushing an application's pages to disk and fetching them back adds latency spikes you feel as random slowness.

Lower it. On a memory-rich server, vm.swappiness = 10 keeps application memory resident and only swaps under real pressure. Some operators go to 1, which is the minimum that still allows swapping when the kernel truly needs it.

The other memory dials worth knowing govern dirty pages (data your programs have changed in memory but that the kernel has not written out to disk yet). A kitchen sink fills with dirty dishes while you cook. vm.dirty_background_ratio is the point where a quiet background wash kicks off. vm.dirty_ratio is the point where the sink is so full that everyone has to stop and scrub before adding another plate, which is your writers blocking until the backlog clears. Lowering both smooths out write bursts instead of letting them pile into one big stall.

# Prefer keeping application memory in RAM; swap only under real pressure.
vm.swappiness = 10
 
# Start flushing changed pages to disk sooner, so write bursts stay smooth.
vm.dirty_background_ratio = 5
 
# Hard cap on unwritten pages before writers are forced to block.
vm.dirty_ratio = 15

Apply it with sysctl --system, which reads every file under the sysctl directories, sets each value live, and prints what it changed so you can confirm it took.

For a defender, swap is also a place secrets go to hide. When a page holding a decryption key or a plaintext password gets swapped, that data is written to disk, where it can outlive the process and turn up in a forensic image, or be read by anyone who later gets at the raw disk. Lower swappiness shrinks that window but does not close it. If a service handles secrets, the real controls are encrypted swap and having the process lock its sensitive pages into memory (the mlock system call, which pins a page in RAM) so they are never written out.

Network And Open-File Ceilings

A busy network service has a waiting line. When connections arrive faster than the program can accept them, the kernel holds the finished ones in a queue until the app picks them up. net.core.somaxconn is the longest that line is allowed to get before the kernel starts turning arrivals away. On modern kernels the default is 4096 (it used to be a cramped 128). For most services that is plenty, but a front-end taking thousands of new connections a second can need it raised. For links moving a lot of data over long distances, the socket buffer ceilings net.core.rmem_max and net.core.wmem_max (the largest receive and send buffers a socket may claim) can also bound throughput, but only raise those when a measurement of a fat, high-latency pipe points there.

The other common ceiling is open files. Every open file and every network socket costs the process a file descriptor (a small number the kernel hands out to track it, like a coat-check ticket for each open thing). There are two limits stacked on top of each other: a per-process cap you can see with ulimit -n, and a system-wide cap in fs.file-max. Hit either and new opens fail with the classic 'Too many open files' error (the EMFILE errno, short for error number), which shows up as dropped connections, failed logging, and services that wedge under load.

The three numbers from fs/file-nr are the handles currently in use, a legacy field that stays zero on modern kernels, and the system-wide maximum. The 1024 from ulimit is this shell's per-process limit, and it is the one most services run into first. syncookies deserves a note for defenders: a SYN flood (a denial-of-service attack that sends the first packet of the TCP (Transmission Control Protocol, the reliable connection protocol behind most internet traffic) handshake over and over without ever finishing it) tries to fill the half-open connection queue so real users cannot get in. net.ipv4.tcp_syncookies = 1 lets the kernel keep accepting genuine connections without storing state for the fakes, which is why it is on by default and should stay on.

Systemd Overrides The Shell's Limits

Here is where people lose an afternoon. You raise the open-file limit in /etc/security/limits.conf, log in, and ulimit -n confirms the new number. Then your service still dies at 1024. The reason: services started by systemd (the program that boots and supervises everything on a modern Linux) do not inherit your shell's limits at all. Their ceiling comes from the unit file's LimitNOFILE setting and nothing else. The right place to raise it is a drop-in override for that specific service.

[Service]
LimitNOFILE=65536

Reload the unit definitions, restart the service, then verify the limit landed on the running process by reading its /proc entry. Trust the process, not the config.

Verify The Change Did What You Think

Setting a value and assuming it worked is how tuning turns into superstition. Two checks close the loop. First, confirm the value is actually live by reading it back with sysctl. Second, confirm the effect on your real workload. For the accept queue there is a precise counter: every time the queue overflows and the kernel drops an arriving connection, it bumps ListenOverflows and ListenDrops. nstat (a small tool that dumps the kernel's network counters) shows them.

Zero and holding steady means your backlog is keeping up. A number that climbs while the service is under load is the kernel telling you, plainly, that connections are being thrown away, either because the queue is too short or because you are being flooded. That single counter turns 'the site feels slow' into a measurable, fixable fact, and it is the before-and-after signal that proves a somaxconn change earned its place.

Keep a short, commented file of the handful of sysctls you have measured into place, and nothing else. When the next engineer opens /etc/sysctl.d/99-perf.conf, every line should answer one question: what measurement put you here? A line that cannot answer that does not belong on the box.

Try this

Work through “Verify The Change Did What You Think” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: swappiness = 0 is not 'never swap'. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related