Linux capabilities: the pieces of root
The handful that actually decide an escape.
Old Unix had one master key, and root held it. If your user ID was 0, the kernel stopped checking permissions and you could do anything at all to the machine. Linux 2.2 took a hacksaw to that key. Today the kernel splits root's power into roughly 40 separate keys, and each one opens exactly one kind of privileged door: bind a port below 1024, load a kernel module, change any file's owner, watch and poke at another running process. Each of those keys is a capability, or cap for short, and together they are what people mean by Linux capabilities. Here is the part that surprises people. A container running "as root" does not get the whole ring. Docker hands it about 14 of the 40. Every hardening move in this lesson is you taking back the keys the app was never going to turn.
Read the ring instead of guessing at it
You never have to guess which keys are on there. Three tools will tell you, and they always agree, because all three read the same thing: what the kernel recorded for that process. capsh --print prints the capabilities of the process it runs inside, spelled out in names a person can read. getpcaps does the same for any PID (process ID, the number the kernel uses to keep track of a running program) you hand it, including a PID you spotted from the host. And /proc/<pid>/status shows the raw bitmasks the kernel actually stores, one bit per key. Start with a plain root container and ask it what it is carrying.
# what a default root container actually holds$ docker run --rm alpine sh -c 'apk add -q libcap; capsh --print' \| grep -E 'Current|Bounding'Current: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=epBounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap# 14 keys. full root on the host would list all ~40.
Before the jargon, stay with the keyring a moment longer. There are the keys you own. There is the one key you happen to be turning in a lock right this second. There are the keys that stay on the ring when you hand it to the next person. And there is the master list of keys the locksmith is ever willing to cut for you. Linux tracks five capability sets per process along exactly those lines: permitted (the keys the process owns and may draw on), effective (the subset the kernel checks when a syscall, meaning a system call, a program's request to the kernel, comes in), inheritable and ambient (what survives an execve, the call that swaps a new program into a running process), and bounding (the master list, the hard ceiling). The =ep that capsh printed is shorthand for "these caps sit in the effective and permitted sets." Bounding is the one that decides containment. A capability outside the bounding set can never be picked up later, not by running a setuid-root binary, not through a file that carries its own file capabilities. So read those 14 names twice. First as what the container holds right now. Then as the ceiling on what it could ever hold, unless somebody edits the run spec. CAP_SYS_ADMIN and CAP_SYS_MODULE are not on the ring, which is why the container cannot mount a filesystem or load a kernel module at all.
Audit from the host, not from inside the container
An attacker who lands inside a container controls everything you can see from inside it. Binaries get swapped, tools get hidden, output gets faked. The kernel owes the intruder nothing. From the host you can pin down the container's main PID and read the capability masks the kernel wrote for that process. That is how you audit a fleet of hundreds without trusting a single byte that came from inside the workload.
# from the host: find the container PID, read the caps the KERNEL recorded$ pid=$(docker inspect -f '{{.State.Pid}}' web)$ getpcaps "$pid"4213: cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap=ep$ grep -E 'CapEff|CapBnd' /proc/$pid/statusCapEff: 00000000a80425fbCapBnd: 00000000a80425fb# a80425fb is the fingerprint of the default set.
That hex string is worth committing to memory. 00000000a80425fb is Docker's default root set, and it comes back identical on every stock container. So dump CapEff across everything you have running. Any value that is not a80425fb means somebody added or dropped capabilities, and the ones with extra bits set are the ones you go look at first. A container started with --cap-drop ALL reads 0000000000000000 instead. Need to turn a raw mask back into names? capsh --decode=0xa80425fb prints them in one line, so a mask copied out of a log or an alert is never a mystery.
The four keys that open the host
Four keys are escape-grade, and there is a good reason none of them ships in the default set. CAP_SYS_ADMIN is so broad it picked up the nickname "the new root": it gates mount plus dozens of other kernel operations, and CVE-2022-0492 (a CVE is an entry in the public catalog of known software flaws) chained it through the cgroup release_agent feature (cgroup means control group, the kernel's accounting box around a set of processes) into a full container breakout on unpatched hosts. CAP_SYS_MODULE lets the container load a kernel module, which is total control of the host in one step. CAP_SYS_PTRACE, paired with a shared PID namespace (container and host looking at one process list), lets a process read and rewrite another process's memory. CAP_DAC_READ_SEARCH skips the file-read permission check and has been used to pull arbitrary files off the host. When a change request adds one of these because "the app would not start otherwise," that sentence is your cue to stop and ask what actually failed.
The fix is mechanical, and it is the same every time. Drop the whole ring, then hand back only the one key a proven need calls for. For most services that is nothing at all. The one you will legitimately hand back now and then is CAP_NET_BIND_SERVICE, which lets a process that is not root bind a port below 1024.
# drop the whole ring, hand back exactly one key$ docker run --rm --cap-drop ALL --cap-add NET_BIND_SERVICE \alpine sh -c 'apk add -q libcap; capsh --print' | grep CurrentCurrent: cap_net_bind_service=ep# verify the escape-grade keys are truly gone$ docker run --rm --cap-drop ALL alpine sh -c \'apk add -q libcap; capsh --print \| grep -qE "sys_admin|sys_module|dac_read" && echo PRESENT || echo "none present, good"'none present, good
The user ID and the keyring are two different locks
One more trap, and it catches careful teams. Running as a non-root UID (user ID, the number the kernel uses to decide who owns a process) and dropping capabilities are two separate controls. Neither one covers for the other. A container can run as UID 0 with an empty capability set, which is mostly harmless. It can also run as a friendly-looking non-root UID that somebody handed CAP_SYS_ADMIN, which is the opposite of harmless. getpcaps tells you the caps. id tells you the UID. An attacker needs only one of the two left open. The hardened baseline turns both locks at once: a non-root UID, cap-drop ALL, and nothing added back that you cannot name out loud in review.
# fleet detection: which running containers were granted EXTRA capabilities?$ docker ps -q | while read c; doprintf '%s add=%s drop=%s\n' \"$(docker inspect -f '{{.Name}}' "$c")" \"$(docker inspect -f '{{.HostConfig.CapAdd}}' "$c")" \"$(docker inspect -f '{{.HostConfig.CapDrop}}' "$c")"done/web add=[] drop=[ALL]/legacy-agent add=[SYS_ADMIN NET_ADMIN] drop=[] # investigate this one
Two numbers are worth carrying in your head: roughly forty capabilities exist, and a default Docker container holds about fourteen of them. Docker's default is already a compromise, enough for common server images to boot, well short of real root. The five names that should stop a change review cold are SYS_ADMIN, SYS_MODULE, SYS_PTRACE, NET_ADMIN and DAC_READ_SEARCH.
Reading CapEff out of /proc is your ground truth. A flag in a compose file records what somebody intended; the kernel bitmap records what actually happened, and over a few months those two drift apart. When an image "needs" SYS_ADMIN for a volume plugin or a nested build, give it a scoped sidecar or a dedicated node rather than widening every replica of the service.
Capability creep is popular with attackers because it arrives as a small YAML edit. Nobody reads a one-line diff with the care they give a new network route. One --cap-add from a debugging session that never got reverted is enough to reopen a path you spent a week closing.
The useful question in review is a narrow one: which operation fails without this capability? Ask for the error message, not the reasoning. If nobody on the thread can name the call that failed, the request is a guess, and the answer stays no until someone reproduces it.
Then there is the check you run after every change window. Re-read the mask, paste the command and its output straight into the ticket, and refuse to close the change if the reading moved. A control you verified once at rollout and never looked at again is a control you are only assuming you still have.
Aim for the tightest set the workload can genuinely run with, not the set that stops the complaints. On one host that habit is boring. Across a few hundred hosts and every pipeline that deploys to them, it is the difference between a contained incident and a bad week.
Try this
Run a default container, then run one with --cap-add SYS_ADMIN, and compare the two CapEff values. Exactly one bit moves. Learn to spot that bit and you can read a grant straight off a status file without decoding anything.
$ docker run --rm alpine sh -c 'grep CapEff /proc/1/status'CapEff: 00000000a80425fb$ docker run --rm --cap-add SYS_ADMIN alpine sh -c 'grep CapEff /proc/1/status'CapEff: 00000000a82425fb$ # CapEff bit for CAP_SYS_ADMIN is set in the second run — that is the review flag$ docker run --rm --cap-drop ALL --cap-add NET_BIND_SERVICE alpine sh -c 'grep CapEff /proc/1/status'CapEff: 0000000000000400
Takeaway
Treat capabilities as a set of individual keys, not a mood. Default-deny with cap-drop ALL, hand back only what the binary provably needs, and never leave SYS_ADMIN or SYS_MODULE on a general-purpose workload.