Read-only root filesystem

Immutable at runtime, with explicit writable paths.

Break into a house and the first thing you look for is somewhere to stash your gear. An attacker who gets code running inside a container does the same thing, and the stash is disk. They drop a reverse shell (a small program that dials back out to the attacker and hands them a command prompt), a crypto miner, a stolen SSH key (SSH = secure shell, the normal way people log into a Linux machine remotely), or a tweaked entrypoint (the command a container runs the moment it starts) that relaunches their tooling after the next restart. Persistence lives on the filesystem. A read-only root filesystem takes the stash away. Mount the container's rootfs (root filesystem: everything the container sees under /) read-only and the attacker still has code execution, but nowhere to write anything down. Nothing they drop survives a restart. Nothing they edit sticks. Better still, the failed write is a signal you can alert on, because a healthy container almost never writes to its own program files.

$ docker run -d --name web --read-only \
    --tmpfs /tmp:rw,noexec,nosuid,nodev,size=64m \
    -v webdata:/var/lib/app \
    alpine:3.20 sleep infinity
9c1f2a7b3e14
 
# attacker has a shell and tries to plant a backdoor in the image
$ docker exec web sh -c 'echo pwned > /usr/local/bin/backdoor'
sh: can't create /usr/local/bin/backdoor: Read-only file system
$ echo $?
1

What read-only actually locks

To see why that write failed, you need to know what a container's root filesystem really is. A container image is a stack of layers, like a pile of transparent sheets on a desk, each sheet recording only what changed from the sheet below it. When the container starts, the runtime lays down the image sheets, which are read-only, then drops one blank sheet on top for the running process to scribble on. Everything you see at / is all of those sheets merged into a single view. That merged view is an overlay filesystem, and the blank top sheet is the upper layer. Normally your writes land there. The --read-only flag mounts the whole rootfs read-only, so a write to any path that came out of the image is refused by the kernel (the core of the operating system, which decides what a program is allowed to do) with the error EROFS, short for read-only filesystem. You see it as "Read-only file system". That covers /, /usr, /etc, /bin, your app directory, everything baked into the image. What it does not cover is any mount you add on top. That gap is where most mistakes hide.

# fleet audit: which running containers are NOT immutable?
$ docker inspect $(docker ps -q) \
    --format '{{ .Name }} readonly={{ .HostConfig.ReadonlyRootfs }}'
/web readonly=true
/legacy-api readonly=false     # <-- flag this one
 
# runtime proof from inside: the merged root really is mounted ro
$ docker exec web sh -c "grep ' / ' /proc/mounts"
overlay / overlay ro,relatime,lowerdir=/var/lib/docker/overlay2/l/ABC...,upperdir=...,workdir=... 0 0

Those two commands answer two different questions. The docker inspect line reads the declared config, the setting Docker was asked for, so you can sweep every container on a host and catch the one that shipped without --read-only. The /proc/mounts line is the truth from inside the container while it runs, because /proc/mounts is the kernel's own live list of every mount currently in place. The little ro token on the overlay mount proves the kernel is really enforcing it, not that somebody typed the flag and hoped. Once the container is running you get a third signal for free. A normal workload never writes to its own binaries, so a failed write to a read-only path is loud. Point a Falco rule at writes under an etc or bin directory (Falco is a runtime security tool that watches what containers do and raises alerts), or set an auditd watch (auditd = the Linux audit daemon, the operating system's own logging service) that catches the EROFS return code. Now "an attacker touched the filesystem" reaches you as a page tonight instead of a nasty discovery weeks later.

Give writes back on purpose

Almost every real app writes somewhere. It wants a scratch directory, a PID file (PID = process ID, the number the kernel gives a running program, usually written to a small file so other tools can find it), a socket, a cache. So you don't switch read-only on and cross your fingers. You switch it on and then hand back exactly the paths that must be writable, one at a time, the way a landlord hands over one key per door instead of the master key. Two tools do the handing back. A tmpfs (temporary filesystem) is a small filesystem that lives in RAM (random access memory, the fast memory that empties when the power goes): quick, writable, and wiped the second the container stops, which makes it right for temp files and runtime scratch that should never outlive the process. A named volume is durable storage that survives the container, for data you actually want to keep. Either way, every writable location is now written down in the run spec where you can audit it, instead of "the whole filesystem is writable and nobody knows what changed." Finding the paths is easy. Run the container once under --read-only and read the errors it throws at you. Nginx, for one, will tell you it needs /var/cache/nginx and /var/run before it agrees to boot. Give each one a tmpfs or a volume and move on.

# the tmpfs we mounted for /tmp IS writable, on purpose
$ docker exec web sh -c 'echo scratch > /tmp/work && cat /tmp/work'
scratch
 
# confirm what actually took effect: noexec and nosuid are on
$ docker exec web sh -c "grep ' /tmp ' /proc/mounts"
tmpfs /tmp tmpfs rw,nosuid,nodev,noexec,relatime,size=65536k 0 0

The paths read-only doesn't cover

Here is the gap attackers actually use. The --read-only flag governs the image layers and nothing else. Named volumes and bind mounts (a bind mount is a directory from the host machine plugged straight into the container) are separate mounts, and they stay writable unless you say otherwise. Worse, they are executable. A volume gets no noexec the way a tmpfs does, so an attacker who reaches a writable data directory can drop a binary there and run it, straight past your immutable rootfs. If that mount is a bind of a host path, they can now write to the host. If it is the Docker socket, the game is over: anyone who can write to /var/run/docker.sock can start a new privileged container and own the whole machine, and read-only does nothing about it. Treat every mount the way you treat the rootfs. If the app only reads a mount, append :ro so nothing can write to it. Detection fits on one line. List your mounts with their RW flag, and anything writable that didn't need to be is a finding.

# a normal write to the volume still works; --read-only never touched it
$ docker exec web sh -c 'echo cache > /var/lib/app/data && cat /var/lib/app/data'
cache
 
# same script, two mounts: the tmpfs blocks exec, the volume runs it
$ docker exec web sh -c 'printf "#!/bin/sh\necho ran\n" > /tmp/s; chmod +x /tmp/s; /tmp/s'
sh: /tmp/s: Permission denied          # tmpfs is noexec: dead end
$ docker exec web sh -c 'printf "#!/bin/sh\necho ran\n" > /var/lib/app/s; chmod +x /var/lib/app/s; /var/lib/app/s'
ran                                    # volume has no noexec: it executes
 
# audit every mount's writability
$ docker inspect --format \
  '{{ range .Mounts }}{{ .Destination }} rw={{ .RW }}{{ println }}{{ end }}' web
/var/lib/app rw=true
 
# fix: if the app only reads it, mount the volume read-only
$ docker run -d --name web --read-only -v webdata:/var/lib/app:ro alpine:3.20 sleep infinity
$ docker exec web sh -c 'echo x > /var/lib/app/x' 2>&1
sh: can't create /var/lib/app/x: Read-only file system

Read-only rootfs is cheap enough to require, not hope for. Kill one myth first. The Restricted Pod Security Standard in Kubernetes (the strictest of the three built-in pod security levels) does not switch it on for you, so don't assume the platform has your back. Set securityContext.readOnlyRootFilesystem: true on your pods yourself, and block anything that ships without it with a policy engine like Kyverno or OPA Gatekeeper (OPA = Open Policy Agent, a rules engine that says yes or no to a config before it ever reaches the cluster). That is the same check the CIS Docker Benchmark makes in control 5.12 (CIS = Center for Internet Security, which publishes hardening checklists). Stack read-only with a non-root user and dropped Linux capabilities (the individual root powers you can hand back one by one), and an attacker who lands inside has no writable binaries and nowhere to stage a payload that survives a restart. That is a lot of persistence gone for the price of one boolean.

Malware expects to write. It reaches for /usr or /var/tmp, and on an immutable rootfs it gets an error instead. Anything it does manage to write has to land in the tmpfs and volumes you allowed on purpose, and those are exactly the places you already size and watch. A quiet compromise becomes a noisy one.

Some apps insist on writing next to their own binary, usually because nobody ever told them not to. The fix is a clean split: code on the read-only layers, state on a mounted volume. Change the app, not the control. Switching read-only off because one log file sits in the wrong directory trades a whole class of protection for the five minutes of work you were avoiding.

Pair it with a non-root user. Root on a read-only rootfs can still do real damage through mounts and capabilities, so this is not a force field. What it removes is the casual persistence: the dropped binary, the edited startup script, the cron entry (cron = the Linux job scheduler that runs commands on a timer) that quietly brings the attacker back tomorrow.

Run the check again after every change window. Confirm the flag is still set, paste the docker inspect line and its output into the ticket, and refuse to close the change if the reading drifted from what you expected. Pick the tightest set of writable paths the workload can actually live with. On one container that is housekeeping. Across every host and every pipeline it is the difference between a fleet you can vouch for and one you hope is fine.

Try this

Prove both halves yourself in about thirty seconds. Start a container with --read-only and watch a write to / fail, then add a tmpfs on /tmp and watch the path you approved work.

$ docker run --rm --read-only alpine sh -c 'touch /x 2>&1; echo status:$?'
touch: /x: Read-only file system
status:1
$ docker run --rm --read-only --tmpfs /tmp alpine sh -c 'touch /tmp/x && echo tmp-ok'
tmp-ok

Takeaway

Make the root filesystem immutable, then open writable paths only where the app genuinely has to keep something. Treat the failed writes as a feature. They turn silent persistence into an error you can see, log and page on.

Related