cgroups & resource containment

What resource isolation promises — and what it does not.

A fork bomb is thirteen characters long: :(){ :|:& };:. Drop it into a container with no limits and it chews through the host's process table in seconds. Every other container on that machine starts failing to spawn threads right alongside the victim. That is the failure mode cgroups exist to stop. Namespaces decide what a process can see. cgroups (control groups, the kernel feature that measures and caps how much of a machine a set of processes may use) decide what it can consume. Namespaces are the walls and one-way mirrors around each container's room. cgroups are the breaker panel in the basement: every apartment draws from the same supply, and the breaker is what stops one flat from browning out the whole building.

The limit lives in a file, not in Docker

cgroup v2 (version two of the control group interface, the single unified tree that now ships as the default on every mainstream Linux distribution) gives each container its own folder of small control files under /sys/fs/cgroup. You never write to those files yourself. The runtime does it the moment you pass a flag. --memory 256m turns into a plain byte count in a file called memory.max, which works out to 268435456 for 256 MiB (mebibytes, the binary cousin of megabytes). The kernel reads that file on the allocation path, every time the group asks for more memory, and refuses the request that would cross it. The container gets no say in this. The check happens one floor below it, in kernel code it has no door into.

$ docker run -d --name web \
    --memory 256m --memory-swap 256m \
    --cpus 1.5 --pids-limit 200 nginx:1.27
7f3c9b1e2a4d
 
# paths below use the systemd cgroup driver (the default on cgroup v2 hosts)
$ CID=$(docker inspect -f '{{.Id}}' web)
$ cat /sys/fs/cgroup/system.slice/docker-$CID.scope/memory.max
268435456              # 256 MiB, the hard ceiling the kernel enforces
$ cat /sys/fs/cgroup/system.slice/docker-$CID.scope/memory.current
7749632               # ~7 MiB actually in use right now
$ cat /sys/fs/cgroup/system.slice/docker-$CID.scope/pids.max
200

The same folder holds cpu.max, which --cpus 1.5 fills in as 150000 100000: 150 milliseconds of processor time in every 100 millisecond window, so one and a half cores' worth. Memory and CPU (central processing unit, the chip that actually runs your code) behave nothing alike when you hit the wall. Overrun memory and something dies. Overrun CPU and nothing dies, because the scheduler makes the group sit and wait its turn. The waiting leaves fingerprints. cpu.stat counts nr_throttled and throttled_usec, and watching those two numbers climb is how you tell 'my app is slow' apart from 'my app is capped'.

Hitting the ceiling: exit code 137

When a group's memory use would push past memory.max and the kernel cannot free up enough to make room, the OOM killer (out of memory killer, the kernel's last-resort process reaper) picks a process inside that group and hits it with SIGKILL (signal 9, the one a program cannot catch, block or ignore). The fattest target is usually PID 1 (process ID 1, the first process started inside the container), which is your actual application, and when PID 1 dies the whole container goes with it. The exit code is 137, which is 128 plus 9. So a 137 in your logs or your CI (continuous integration, the pipeline that builds and tests your code on every push) output almost always means 'this container wanted more memory than you gave it', not a crash in your code.

$ docker run --name hog --memory 64m --memory-swap 64m python:3.12-alpine \
    python -c 'x = bytearray(200 * 1024 * 1024)'   # grab and touch 200 MiB
$ echo $?
137
 
# the detection: Docker records exactly why the container died
$ docker inspect hog --format '{{.State.OOMKilled}} {{.State.ExitCode}}'
true 137

The kernel keeps its own tally. Every container's memory.events file carries an oom_kill counter, and the Docker daemon emits an oom action on docker events the instant one fires. Point an alert at either signal and a container that quietly restart-loops under memory pressure has nowhere left to hide.

The fork bomb, contained

pids.max does for processes what memory.max does for memory, and it earns its keep more often than people expect. Containers share one thing you might assume is private: the host kernel's single pool of process IDs. A fork bomb with no --pids-limit is not contained to its container at all. It sprints to drain the host's global PID space, and once that space is gone the host itself cannot fork. sshd (the secure shell daemon, the service you log in through) cannot open a session. Other containers cannot start workers. The box is down in every way that matters. Set a limit and the group hits its own wall first, with the damage staying inside it.

# alpine's shell is busybox ash (a POSIX sh), which won't accept ":" as a
# function name, so this is the same bomb with the function renamed to "b"
$ docker run -d --name bomb --pids-limit 100 alpine sh -c 'b(){ b|b& };b'
c4e1a9f0d5b7
$ BID=$(docker inspect -f '{{.Id}}' bomb)
 
$ cat /sys/fs/cgroup/system.slice/docker-$BID.scope/pids.current
100          # pinned at the cap; the host's PID space is untouched
$ cat /sys/fs/cgroup/system.slice/docker-$BID.scope/pids.events
max 5231     # fork attempts the kernel refused (the bomb, throttled)
$ docker rm -f bomb >/dev/null

What cgroups never promised you

cgroups bound consumption. They are not a security boundary, and they never claimed to be. A container can be metered to the byte and still read kernel memory, abuse a Linux capability (one of the fine-grained slices of root's power that the kernel hands out individually), or ride a runtime bug all the way to a full host takeover. There is a leak worth knowing about too. Plenty of tools answer 'how much memory do I have?' by reading /proc/meminfo, and those totals are not namespaced, so inside the container /proc reports the host's RAM (random access memory, the machine's physical working memory) instead of your memory.max. A JVM (Java Virtual Machine, the runtime that Java programs execute on) or a Node process that sizes its heap from that number budgets for memory it will never be allowed to touch, then takes a 137 the moment it grows into your cap.

$ docker run --rm --memory 256m alpine sh -c '\
    echo "proc  : $(grep MemTotal /proc/meminfo)"; \
    echo "cgroup: $(cat /sys/fs/cgroup/memory.max) bytes"'
proc  : MemTotal:       32784132 kB      # the HOST's 32 GiB, not your limit
cgroup: 268435456 bytes                  # the real 256 MiB cap, from the right file

cgroups and namespaces answer two different questions. Namespaces ask what this process can see. cgroups ask how much of the host it can take. Leave out memory, CPU and PID limits and one runaway container becomes a denial-of-service attack (flooding a system with work until it can no longer serve anyone) against every neighbour on the box, with no escape or exploit required.

The classic foot-gun is setting --memory while leaving swap unbounded, or forgetting on cgroup v2 that memory and swap are budgeted as one number. The process hits its RAM ceiling, spills into swap, and thrashes the host disk while a naive dashboard still shows a tidy limit that looks respected. Decide the total memory budget on purpose rather than by accident.

Circuit breakers, not door locks, is the mental model to carry out of here. A breaker stops one faulty toaster from blacking out the floor. It does nothing about the person with a master key who strolls into the electrical room. cgroups are the breaker. The locks live elsewhere: capabilities, seccomp (the kernel filter that decides which system calls a process is allowed to make), and user namespaces.

Every limit in this lesson is off by default. A bare docker run hands the container the whole machine: all the RAM, every core, as many processes as the kernel will give out. Nothing warns you about it. The container looks perfectly healthy right up to the day traffic doubles and it drags the node's other workloads down with it. A missing limit is a decision you made, whether or not you meant to make it.

Auditing what you already run takes a minute. docker inspect prints the values the daemon recorded, and fields like HostConfig.Memory and HostConfig.MemorySwap read 0 when nothing was set. The cgroup files on the host are your second opinion. If the daemon claims 256 MiB and memory.max reads max, believe the file, because the file is the copy the kernel consults.

Picking the number is the part nobody enjoys. Watch memory.current under real traffic for a week, take the peak, add headroom for the spike you did not happen to catch, and set your cap there. Too tight and you collect 137s at three in the morning. Too loose and the limit protects nobody. A cap a little above the observed peak beats both a round number chosen because it looked neat and a cap you never set at all.

Orchestrators put a friendlier face on these same files. A Kubernetes memory limit lands in memory.max on the node exactly the way --memory does, and a CPU limit becomes the same cpu.max pair. The vocabulary changes, the kernel does not. Everything here still applies when a YAML file is writing the flags for you.

CPU limits deserve a warning of their own, because throttling is invisible from inside the container. Latency graphs go bad, application logs stay clean, nothing crashes, and whoever is on call burns an hour chasing a ghost. nr_throttled climbing in cpu.stat ends that hunt in seconds. Scrape it alongside your usual metrics before you need it, not in the middle of the incident.

PID limits are the cheapest control on this page and the one people forget first. Most workloads live comfortably under a few hundred processes. Something that suddenly wants thousands is either badly broken or being driven by someone who is not you. A --pids-limit in the low hundreds costs a normal service nothing and turns a host-wide outage into one container having a bad afternoon.

In production the real work starts after the change window. Read the value back from the host once the change is in, paste the command and its output into the ticket, and refuse to close the change if the reading drifted from what you asked for. Choose the tightest budget the workload can actually live with. That habit compounds quietly across every host and every pipeline you own.

Try this

Start a deliberately greedy container, give it a cap it cannot possibly fit inside, and watch the kill land. Then run it again with swap pinned to the same number and compare how the host behaves. Seeing OOMKilled: true come back from a container you starved on purpose is the fastest way to make exit 137 stop feeling mysterious.

$ docker run -d --name hog --memory=64m --memory-swap=64m progrium/stress --vm 1 --vm-bytes 128M
$ docker events --since 0s --until 15s --filter container=hog
... oom ... die ...
$ docker inspect -f '{{.State.OOMKilled}} {{.HostConfig.Memory}} {{.HostConfig.MemorySwap}}' hog
true 67108864 67108864
$ docker rm -f hog
hog

Takeaway

cgroups bound the blast radius of a noisy neighbour. They do not stop a breakout. Set memory and swap as a single decision, add a PID cap anywhere a fork storm is a realistic risk, and treat an unlimited container on a shared host as an outage that has not picked its date yet.

Related