/proc, /sys & sysctl

The kernel as a filesystem you can read.

Most files on your computer are lumps of data sitting on a disk: a photo, a log file, last quarter's spreadsheet. The files under /proc and /sys are a different animal. Nothing is stored there. Each time you read one, the kernel (the core part of the operating system that talks directly to the hardware) builds the answer on the spot out of whatever is true at that instant. A factory control room puts every reading on one wall. Some gauges are read-only dials showing pressure, temperature, and load. Others are knobs an operator can turn to change how the line runs. /proc and /sys are that wall for your kernel, written out as plain text you read with cat and turn with echo.

This has a name you have probably heard: on Linux, almost everything is a file. It sounds like a slogan until you notice what it buys you. To ask how much memory a process is using, or whether a security mitigation is switched on, you do not reach for a special programming interface. You read a file. And the tools you lean on all day (ps, top, free, uptime) are themselves reading these same files and dressing up the result, so going to the source shows you what the kernel actually reports, before any tool rounds it off, summarizes it, or in the worst case gets tricked into lying to you.

A Filesystem That Lives Nowhere

Start by proving the 'nothing on disk' claim to yourself. Ask df, the disk-usage tool, how big these filesystems are.

Zero size, zero used, zero available. /proc and /sys are pseudo-filesystems (also called virtual filesystems): directory trees the kernel mounts at boot and fills in on demand, with no blocks on any drive behind them. Every open() and read() is really a function call into the kernel that returns freshly computed text. You can even see the guard rails on the mount.

Those options are worth a glance. nosuid, nodev, and noexec mean you cannot run programs or honor special device files out of /proc, a sensible default that shuts off a few tricks. Now the read-only dials. Three files answer the first questions you ask about any box: what processor it runs on, how much memory it has, how hard it is working.

Read that last line, /proc/loadavg, left to right: the system's load averaged over 1, 5, and 15 minutes, then 2/834, which means 2 tasks are runnable right now out of 834 that exist, and finally 28194, the process ID the kernel handed out most recently. free and uptime are formatters over meminfo and loadavg, nothing more.

Per-Process Ground Truth

Every running program gets its own numbered folder under /proc, named by its PID (process identifier, the number the kernel uses to track it). Think of it as a live case file the kernel keeps open for each process and rewrites continuously. ps and top skim a few pages of that file and print a neat table. When you need the truth instead of a summary, you open the folder yourself.

A quick tour of the entries that matter. cmdline is the exact command the process was started with. environ is its environment variables. exe links to the real binary (the compiled program file) on disk. cwd links to its working directory. fd/ lists every open file and socket (a live network connection) it holds. maps is its memory layout. limits shows its resource ceilings. status is a readable summary of all of it. Four of these pay for themselves during an incident.

Here is the one that bites teams in production. cmdline is world-readable. Any local user can read the full argument list (argv, the command-line arguments a program was started with) of any process on the box, including root's.

That process, PID 1337, belongs to another team. The webapp user read its database password out of thin air, no privileges needed, because someone passed the secret as a command-line flag. The kernel stores arguments separated by null bytes, which is why we translate them to newlines with tr. The operator takeaway: pass secrets through environment variables or files, never as arguments, and treat argv as public within the host.

The environment is stricter. environ is readable only by the process's owner and by root, so an unrelated user hits a wall.

Better, but not a safe. Root can read it, the process can leak it, and anything that can dump the process memory can pull it out. Environment variables beat command-line flags for secrets; they are not a vault. If you want a vault, mount the secret as a file with tight permissions.

One entry deserves a look on any host you suspect. exe is a symbolic link (a small file that points at another file) to the actual binary the process is executing.

A healthy daemon (a long-running background service) points at something like /usr/sbin/nginx. This one points into /dev/shm (a memory-backed scratch directory) at a hidden folder, and the kernel has tagged it (deleted). The binary unlinked itself from disk right after launching and now runs only from memory. That is a textbook pattern for a crypto-miner or fileless malware: ps shows a believable name, while /proc/<pid>/exe shows where the code really came from.

The everyday reads matter too. How many descriptors is it holding, how many is it allowed, how much memory is resident, how many threads.

A file descriptor is the numbered handle the kernel gives a process for each open file or socket. This process holds 17 and is allowed 1024 (its soft limit) up to 4096 (its hard ceiling). VmRSS is resident set size (RSS), the memory this process actually has in RAM (the physical working memory) right now, as opposed to memory it has reserved but not touched. Threads is the count of threads it is running. When a service is leaking descriptors, this is where you watch the number climb toward its limit, before it starts failing with 'Too many open files'.

Reading /proc directly is also how you catch a process hiding from ps. Because ps builds its list from /proc, a rootkit that only doctors ps output, or a process using odd tricks to stay off the list, can be exposed by walking /proc yourself and comparing. A PID with a live folder that never appears in ps is a lead worth pulling.

sysctl and /proc/sys: The Kernel's Control Panel

Everything under /proc/sys is a knob rather than a dial. These are the kernel's tunable settings, and unlike a CPU model or a load average, you are allowed to write to them. sysctl is the labeled control panel bolted over those same files. It exists because dotted names read more easily than long paths, and because of one trick worth burning into memory: a sysctl name is the file path under /proc/sys with the slashes turned into dots.

Same value, two doors. sysctl vm.swappiness and cat /proc/sys/vm/swappiness open the identical file. vm.swappiness sets how eager the kernel is to move memory out to swap (disk space the kernel uses as an overflow area when RAM fills up), on a scale from 0 to 100, where 60 is the stock setting. On a database host you often want it lower, so the kernel keeps hot data in RAM instead of paging it out to disk the moment memory gets tight.

That write is live and instant. It is also temporary. Reboot and you are back to 60, because sysctl -w only pokes the running kernel and forgets. To make a value survive a reboot, put it in a file the system reads on every boot.

# Persisted kernel tuning. Applied at boot and by `sysctl --system`.
vm.swappiness = 10
net.core.somaxconn = 1024

For a defender, /proc/sys is also where a large slice of host hardening lives, and where you confirm it stayed hardened. A few settings are worth reading on every box.

Read plainly: randomize_va_space = 2 means full ASLR (Address Space Layout Randomization, the kernel shuffling where code and data land in memory so an attacker cannot predict addresses), where 0 would mean it is off, a red flag. dmesg_restrict = 1 stops non-root users from reading the kernel log, which often leaks useful addresses. kptr_restrict = 1 hides kernel pointer values (internal memory addresses) from /proc for the same reason. rp_filter = 2 turns on reverse-path filtering in loose mode: the kernel drops a packet when its source address has no route back out through any interface, a cheap way to bin obviously spoofed (forged-source) traffic. Setting it to 1 is strict mode, which insists the return route match the exact interface the packet came in on.

Attackers write sysctls too. A common move after landing on a host is to turn it into a router so traffic can pivot through it, by setting net.ipv4.ip_forward to 1. Watching that one value is cheap. When you set it back, confirm the file itself, not only the command's echo, because the file is the ground truth.

/sys: Where the Hardware Shows Up

If /proc is mostly about processes and the live kernel, /sys is the building directory for hardware. sysfs (the filesystem mounted at /sys) is a tree the kernel builds to expose devices, drivers, and their settings as files. Your desktop, the udev device manager, and monitoring agents all read it constantly.

The first file is the MAC (Media Access Control) address, the hardware identifier tied to a network interface card. The second answers whether a disk spins: rotational = 0 means a solid-state drive, 1 means a spinning platter. On cloud hosts your names will differ (ens5 for the interface, nvme0n1 for the disk), but the files sit in the same place. This is how tooling learns your hardware without asking you.

The /sys entry security teams should know by name is the kernel's own report card on CPU hardware flaws.

Each file is the kernel's verdict on one CPU flaw: whether this machine is exposed and what mitigation is active. You will see Not affected, Mitigation: followed by the technique in use, or the single word Vulnerable, which is the one to hunt for across a fleet. It is one grep per host, no agent and no vendor scanner required.

Closing the Doors You Left Open

Remember that cmdline is world-readable, which lets any local user inventory every process and its arguments. On a shared or multi-tenant host you can narrow that view. Mounting /proc with the hidepid option hides other users' process folders from them.

# Hide other users' processes from unprivileged accounts.
proc  /proc  proc  defaults,hidepid=2  0  0

hidepid=2 (shown as invisible in /proc/mounts on modern kernels) means an unprivileged user sees only their own processes and cannot even tell that anyone else's exist. That shuts the argv-harvesting door from earlier for normal users. It does not stop root, and it can trip up monitoring agents that expect to see the whole process table, so roll it out and test before you depend on it.

The habit that pays off: when a tool tells you something surprising, find the file behind it and read that. cat /proc/sys/net/ipv4/ip_forward, ls -l /proc/<pid>/exe, grep . /sys/devices/system/cpu/vulnerabilities/*. The kernel is not keeping secrets. It has already written the answer to a file, and that file is current as of the instant you ask.

Try this

Work through “Closing the Doors You Left Open” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: there is no save button. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related