CoursesAdvanced Linux internals & toolingNamespaces & cgroups by hand

Namespaces & cgroups by hand

Build a container with no Docker.

Advanced14 min · lesson 11 of 17

There is no container object inside the Linux kernel (the core program that runs everything else and talks to the hardware). You can read the source and grep for one; it is not there. What is there are two older features that shipped years before Docker existed: namespaces and cgroups (control groups). Docker, Podman, and Kubernetes stack those two together and paint a friendly label on top. Scrape the label off and a container is one ordinary process that the kernel has been told to show less and allow less. Building one by hand takes twenty minutes and removes every bit of mystery, which is what you want as a defender, because you cannot harden or investigate something you believe is magic.

Two separate knobs do the work, and keeping them straight is the whole lesson. Namespaces control what a process can SEE. cgroups control what it can USE. One is a set of blinders. The other is a metered power strip that trips when you draw too much.

The two halves of a hand-built container
Namespaces: what a process can SEE
PID
its own process list; it becomes number 1
Mount
its own filesystem tree
UTS
its own hostname
Network
its own interfaces and routes
User
its own idea of who root is
cgroups: what a process can USE
memory.max
hard memory ceiling
cpu.max
share of processor time
io.max
disk bandwidth
pids.max
how many processes
Namespaces isolate the view. cgroups cap the resources. A runtime only wires both together.

Namespaces With Unshare

A namespace is a private copy of one kind of system resource. The process inside sees only its copy and treats that copy as the whole machine. Think of a hotel room whose walls are painted to look like the entire world: you can walk around, but you only ever see the room. Linux has several kinds, one per resource. PID (process identifier) namespaces give a process its own numbering of processes. Mount namespaces give it its own view of which filesystems are mounted where. UTS (Unix Timesharing System, an old name that now only covers the hostname fields) gives it its own hostname. Network namespaces give it its own interfaces, routes, and firewall rules.

The unshare command runs a program after splitting off (un-sharing) the namespaces you name. Give a shell its own PID, mount, UTS, and network namespaces in a single call:

~/secopslog — bash
$ sudo unshare --pid --mount --uts --net --fork --mount-proc /bin/bash root@host:/# echo $$ root@host:/# hostname box; hostname root@host:/# ps -ef root@host:/# ip addr
1 box UID PID PPID C STIME TTY TIME CMD root 1 0 0 12:04 pts/0 00:00:00 /bin/bash root 7 1 0 12:04 pts/0 00:00:00 ps -ef 1: lo: <LOOPBACK> mtu 65536 qdisc noop state DOWN group default qlen 1000 link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00

Read the flags left to right. --pid, --mount, --uts, and --net each ask for a fresh namespace of that kind. --fork matters more than it looks: a PID namespace only takes effect for children of the process that created it, so unshare forks a child to become PID 1 inside. Without --fork you get errors and a shell that is not actually PID 1. --mount-proc mounts a brand-new /proc (the virtual filesystem the kernel uses to expose process information) inside the mount namespace, so ps reads the new process list instead of the host's. Skip it and ps still shows every process on the machine, because it is reading the host's /proc. The empty network namespace is the clearest tell: one loopback interface, down, and nothing else.

Watching It From The Host

Isolation runs one way, like a one-way mirror in an interview room. The process inside cannot see out, but you, standing on the host, can see everything about it. That asymmetry is what makes containers investigable. Every namespace a process belongs to shows up as a file under /proc/<PID>/ns, and each file points at an inode number (the unique ID the kernel puts on a filesystem object, here standing in as the namespace's real identity), like a license plate on the namespace. Two processes carrying the same plate are in the same namespace.

~/secopslog — bash
$ for n in pid net mnt uts; do printf '%s ' $n; readlink /proc/self/ns/$n; done
pid pid:[4026531836] net net:[4026531992] mnt mnt:[4026531840] uts uts:[4026531838]

Those are the host's own namespace inodes. From a second terminal on the host, lsns lists namespaces and who lives in them. Your unshared shell, PID 1 to itself, shows up here with its true host PID and its own pid-namespace inode:

~/secopslog — bash
$ sudo lsns -t pid
NS TYPE NPROCS PID USER COMMAND 4026531836 pid 243 1 root /sbin/init 4026532189 pid 1 18542 root /bin/bash

That is the defender's lever. A process cannot hide from lsns or from /proc by entering a namespace, because the kernel still tracks it from the outside. If malware spawns itself into fresh namespaces to masquerade as an innocent PID 1, lsns and /proc/<PID>/cgroup still hand you the real tree, its host PID, and its parent. A process that claims to be PID 1 but sits at host PID 18542 with a non-initial pid inode is a process worth a second look.

cgroups Apply The Limits

Namespaces changed what the shell can see. It can still use the whole machine's memory and every CPU (central processing unit) core. cgroups fix that. A cgroup (control group) is a labelled bucket you drop processes into, with control files that cap how much of each resource the bucket may draw. It behaves like the breaker panel in a house: each circuit has a rating, and pull too much through one and it trips instead of melting the wiring.

Modern systems use cgroup v2 (control groups version 2), one unified tree mounted at /sys/fs/cgroup. First, see which resource controllers the kernel offers and which of them the parent hands down to its children:

~/secopslog — bash
$ cat /sys/fs/cgroup/cgroup.controllers cat /sys/fs/cgroup/cgroup.subtree_control
cpuset cpu io memory hugetlb pids rdma misc cpu io memory pids

memory, cpu, and pids appear in cgroup.subtree_control, which is why child groups you create will have memory.max and cpu.max files. This is a real gotcha: if a controller is missing from the parent's subtree_control, the matching control file never appears in the child, and your write fails with 'No such file or directory'. Create a group, cap its memory and CPU, and read the memory ceiling back:

~/secopslog — bash
$ sudo mkdir /sys/fs/cgroup/demo echo "100M" | sudo tee /sys/fs/cgroup/demo/memory.max echo "50000 100000" | sudo tee /sys/fs/cgroup/demo/cpu.max cat /sys/fs/cgroup/demo/memory.max
100M 50000 100000 104857600

Notice the memory ceiling reads back as 104857600, not 100M. The kernel accepts human units on the way in and stores plain bytes, rounded to page size, so always trust the number you read back over the one you wrote. cpu.max reads '50000 100000', meaning 50,000 microseconds of CPU time allowed per 100,000-microsecond window, which is half of one core.

Nothing is limited yet, because the group is empty. Put a process in by writing its PID to cgroup.procs, then prove the ceiling is real by trying to allocate past it. Running tail on /dev/zero (an endless stream of zero bytes) makes the tool try to buffer forever, which is a fine memory bomb for a test:

~/secopslog — bash
$ echo $$ | sudo tee /sys/fs/cgroup/demo/cgroup.procs cat /proc/self/cgroup tail /dev/zero cat /sys/fs/cgroup/demo/memory.events
26814 0::/demo Killed low 0 high 0 max 4 oom 1 oom_kill 1 oom_group_kill 0

The kernel let the process grow to 100 megabytes, then the OOM (out-of-memory) killer stepped in, and memory.events recorded oom_kill 1. That counter is the one to watch in production. If a service keeps dying and memory.events shows oom_kill climbing, the process is hitting its cgroup ceiling, not a host-wide shortage, and the fix is the limit, not more RAM (random-access memory, the fast working memory the process actually runs in). When you are finished with the group entirely, get every process out of it (the shell leaves when it exits, or move its PID back to the root group with echo $$ | sudo tee /sys/fs/cgroup/cgroup.procs) and run sudo rmdir /sys/fs/cgroup/demo. rmdir refuses while any process is still inside.

Confirm The Limit Actually Bit

Setting a limit and checking that it fires are different jobs. For CPU, the proof lives in cpu.stat. Burn a core inside the group and watch the throttle counters climb:

~/secopslog — bash
$ yes > /dev/null & sleep 5; cat /sys/fs/cgroup/demo/cpu.stat
[1] 26840 usage_usec 2510334 user_usec 2500120 system_usec 10214 nr_periods 51 nr_throttled 48 throttled_usec 2431990

nr_throttled counts the scheduling windows in which the group hit its cap and got parked. Forty-eight of fifty-one periods throttled means the 50 percent limit is doing its job. If you set a cpu.max and nr_throttled stays at zero under load, the process never reached the cap and your limit is a no-op. cpu.stat and memory.events are how you verify a change worked instead of hoping it did.

Isolation is not a security boundary
Everything here shares one kernel. Namespaces hide the host and cgroups ration resources, but the process still calls straight into the same kernel as everything else on the box. A hand-rolled unshare shell has no seccomp (secure computing mode, a filter on which system calls a process may make), drops no capabilities (the fine-grained pieces of root's power), and adds no user namespace, so root inside is root on the host the moment it escapes a mount or trips a kernel bug. This is superb for learning and unsafe for running untrusted code. Use a hardened runtime for that, and keep the kernel patched, because the kernel is the wall every container leans on.
Quick check
01A shell runs with fresh PID, mount, UTS, and network namespaces plus a cgroup capping memory at 100M. Code inside it triggers a kernel privilege-escalation bug. What does that setup actually do about it?
Incorrect — Hiding host processes changes what the exploit can look at, not what the kernel will run for it. Attacking the kernel needs no view of the host at all.
Incorrect — A memory cap governs how many pages the group may hold. It has no say over which kernel code paths the process gets to reach.
Correct — Both features sit above the kernel and depend on it staying honest. When the kernel is the thing that breaks, neither one is anywhere in the path.
Incorrect — Remapping identities takes a user namespace, and this command asked for four namespaces that do not include one. Root inside stays root outside.
02You run sudo unshare --pid --mount --uts --net --fork --mount-proc /bin/bash. What is --mount-proc doing, and what goes wrong without it?
Correct — ps builds its listing by reading /proc rather than asking the kernel directly, so a private copy is what narrows the output down to the namespace itself.
Incorrect — That fork belongs to --fork, which exists because a PID namespace only applies to children of the process that created it.
Incorrect — The renumbering comes from --pid. Your shell can be process 1 in its own namespace and still list every host process while it reads the host's /proc.
Incorrect — The empty network stack with its single loopback arrives with --net. Mounting /proc has nothing to do with interfaces or routing tables.
03You mkdir /sys/fs/cgroup/demo without error, then echo 100M > demo/memory.max returns 'No such file or directory' even though the directory is sitting right there. What explains it?
Incorrect — Membership decides which processes a limit applies to. It does not conjure control files, and an empty group still carries every file it is entitled to.
Correct — A child only receives control files for controllers its parent enabled. Read the parent's cgroup.subtree_control and you will find memory absent from the list.
Incorrect — That mistake gives you a permission error when the shell opens the file. Here the kernel is telling you the path does not exist yet.
Incorrect — Human units are accepted going in. Write 100M into a working memory.max, read it back, and you get 104857600, the same ceiling stored in bytes.

A real container is these same files. Point the tools at a running Docker process and you see the same kind of inodes and the same control-group path. Here, nginx runs as host PID 31002:

~/secopslog — bash
$ docker inspect -f '{{.State.Pid}}' web sudo lsns -p 31002 cat /proc/31002/cgroup
31002 NS TYPE NPROCS PID USER COMMAND 4026531834 time 248 1 root /sbin/init 4026531837 user 248 1 root /sbin/init 4026532200 cgroup 6 31002 root nginx: master process 4026532201 mnt 6 31002 root nginx: master process 4026532202 uts 6 31002 root nginx: master process 4026532205 ipc 6 31002 root nginx: master process 4026532206 pid 6 31002 root nginx: master process 4026532209 net 6 31002 root nginx: master process 0::/system.slice/docker-9f3c2a7b4e1d8a6f0c5b3e2d1a4f6c8b7e9d0a2c4f6b8d1e3a5c7f9b0d2e4a6c.scope

Look at the time and user rows: their inodes match the host's initial namespaces and point at PID 1, which is the kernel telling you Docker did not isolate them. Root in that container is root on the host if it breaks out, unless someone turned on user-namespace remapping. The mount, PID, UTS, IPC (inter-process communication), and network rows carry their own high inodes, so those are private, and the last line shows the systemd scope (systemd is the service manager that starts and supervises processes on most modern Linux) that holds the limits. Same primitives you set by hand, wired together by a runtime, and every one of them readable from /proc and /sys/fs/cgroup the moment an incident lands on you.

Try this

Work through “Confirm The Limit Actually Bit” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

The trap worth remembering here: isolation is not a security boundary. Check that on your own systems before you need to, because it is cheaper to find on a quiet afternoon than during an incident.

Related