>_SecOpsLog_
CoursesDocker in depthStorage drivers & the layered filesystem

Storage drivers & the layered filesystem

overlay2 and copy-on-write.

Intermediate12 min · lesson 10 of 30

A storage driver is the piece of Docker that turns a stack of image layers into a filesystem a container can actually read from and write to. Layers work like sheets of tracing paper stacked on a desk. Each sheet adds or redraws a few lines, and when you look straight down through the pile you see one combined drawing. The container only ever sees that combined drawing. It has no idea the picture is spread across a dozen separate sheets underneath.

On any current kernel the driver doing that work is overlay2, a union filesystem (a filesystem that stacks several directories and presents them to you as one). It calls the read-only image layers the lowerdir, hands each container a single writable layer called the upperdir, and blends the two into the view the container works in, the merged directory. Older drivers such as btrfs, zfs and vfs are still around for odd corners, but you will almost never pick one on purpose. Docker dropped aufs and devicemapper a few releases back, so those are not even on the menu.

Find out what your host is actually running

terminal
$ docker info -f '{{ .Driver }}'
overlay2
$ docker info -f '{{ json .DriverStatus }}'
[["Backing Filesystem","extfs"],["Supports d_type","true"],["Using metacopy","false"]]

Supports d_type has to say true. d_type is short for directory entry type, a small piece of information the backing filesystem returns when overlay lists a directory, and overlay uses it to tell a real file apart from the marker left behind by a deleted one. On ext4 you get d_type for free. On xfs you only get it if the filesystem was formatted with ftype=1, and when it is missing overlay2 corrupts data quietly instead of failing loudly. Every layer lives as its own directory under /var/lib/docker/overlay2. You can move that whole tree onto a dedicated disk with the data-root setting in /etc/docker/daemon.json, which keeps images and volumes off the root partition.

Reads are the easy direction. Overlay looks down the stack from the top and hands back the first copy of a file it finds, so a file in the upperdir shadows the same path in every layer below it. That is how a later layer replaces a file without the original ever being deleted. Writes are where the interesting behaviour lives.

What overlay2 does on the first write to a file
Container writes to /path/file
first write
Brand-new path, nothing below
File is created straight in the upperdir
no copy
Path exists only in a lower layer
Whole file is copied up to the upperdir, then edited
copy-up cost
Path already in the upperdir
File is edited in place
cheap

Watch copy-on-write happen

Copy-on-write is the rule that the lower sheets never get touched. The first time a container changes a file that came from a lower layer, overlay copies that entire file up into the upperdir and edits the copy. Brand-new files are simpler. They get created straight in the upperdir, because there is nothing below to copy. Both are easy to prove. Start a container, write a fresh file, then go find it on the host.

terminal
$ docker run -d --name web nginx:1.27
b7f3a9c2e1d4
$ docker exec web sh -c 'echo "written inside the container" > /note.txt'
$ docker inspect -f '{{ .GraphDriver.Data.UpperDir }}' web
/var/lib/docker/overlay2/3d9c.../diff
$ sudo cat /var/lib/docker/overlay2/3d9c.../diff/note.txt
written inside the container

That file sits in no image layer at all. It exists only in the diff directory, which is the upperdir overlay2 handed this container. Delete the container and the file goes with it. Now edit a file that did ship inside the image, and watch the copy-up land in exactly the same place.

terminal
$ docker exec web sh -c 'echo "# tweaked" >> /etc/nginx/nginx.conf'
$ sudo ls /var/lib/docker/overlay2/3d9c.../diff/etc/nginx/
nginx.conf
$ docker diff web
A /note.txt
C /etc
C /etc/nginx
C /etc/nginx/nginx.conf

nginx.conf ships inside the nginx image, so before your write it lived only in a lowerdir. Appending one line forced overlay to copy the whole file into the upperdir first, which is why it now shows up under diff. For a config file of a few kilobytes that cost is nothing. For a multi-gigabyte database file it is a stall, and it hits on the very first write. docker diff spells the damage out: A for added, C for changed. Newer kernels offer a metacopy mode (the false in that DriverStatus line) which copies only a file's metadata up for cheap changes like a chmod, but rewriting the contents still pays the full copy.

This is why databases go on volumes

A volume steps around the union stack completely. It is a directory on the host's own filesystem that Docker mounts straight into the container, so writes go to disk directly with no copy-up and no merged view in the path. It also outlives the container, which the writable layer never does. Run Postgres on one and check where its data really lands.

terminal
$ docker run -d --name pg \
    --mount type=volume,source=pgdata,target=/var/lib/postgresql/data \
    -e POSTGRES_PASSWORD=secret postgres:16
f0a1b2c3d4e5
$ docker inspect -f '{{ range .Mounts }}{{ .Destination }} -> {{ .Source }}{{ end }}' pg
/var/lib/postgresql/data -> /var/lib/docker/volumes/pgdata/_data
$ docker exec pg psql -U postgres -c \
    "CREATE TABLE t(id int); INSERT INTO t VALUES (42); SELECT * FROM t;"
 id 
----
 42
(1 row)

The mount source points into /var/lib/docker/volumes, not an overlay2 diff directory, so Postgres never pays copy-on-write on its data files. The row you inserted goes to the backing filesystem directly. Stop the container, remove it, start a fresh one on the same pgdata volume, and the table with its 42 in it is still sitting there.

Deleting a file from a lower layer does not shrink your image
When a container or a later build step deletes a file that lives in a lower layer, overlay2 cannot truly remove it. Those bytes are read-only. What it writes instead is a whiteout marker, a zero-size character device in the upperdir whose only job is to hide that path from the merged view. In a running container that is harmless. In a Dockerfile it bites: running rm bigfile.tar in a later RUN, after an earlier layer added the file, leaves the full file sitting in that earlier layer and your image stays fat. Delete the file in the same RUN that created it, or use a multi-stage build so the junk never enters the final image.

The copy-on-write tax, and who ends up paying it

The tax is charged per file, once, on the first write, and it is charged in full. A container that appends a single line to a 40 MB log file it inherited from its image copies all 40 MB before writing byte one. Ten containers started from that same image each pay it separately, because each one gets its own upperdir. docker diff is the fastest way to spot who is paying: point it at a container that is supposed to be immutable, and every C line is a file that got copied up behind your back.

Disk pressure under data-root shows up in two numbers, and most dashboards only watch one. Bytes are the obvious one. Inodes are the quiet one. An inode is the bookkeeping entry a filesystem allocates per file or directory, and overlay2 creates a directory per layer plus an entry for every file in it, so an image full of thousands of tiny files burns through inodes far faster than it burns through gigabytes. You can run out of inodes on a disk that still reports plenty of free space. Keep data-root on a fast disk, watch both counters, and keep big short-lived files (build scratch, unpacked tarballs, caches) out of the writable layer.

If a host reports a driver you were not expecting, docker info tells you before you waste an afternoon. Everything in this lesson is overlay2 behaviour: the lowerdir and upperdir layout, the file-level copy-up, the d_type requirement. None of it transfers cleanly to btrfs or zfs, which handle layers with their own snapshot machinery. Match the documentation to the machine in front of you first.

Changing the driver or moving data-root is a maintenance window, not an afternoon tweak. The daemon has to stop, and images and containers under the old root do not follow you to the new one. Before you touch it, note the Driver and DriverStatus lines you saw, which host you ran them on, and what you would do to put it back. A one-line record saying that Supports d_type read true on the old ext4 root and reads true on the new xfs volume too is what saves you at 2am when containers start losing files. If a colleague cannot replay the change from the ticket alone, add the exact daemon.json keys you set and the docker info output you expected on a healthy host.

Try this

Run these on a lab engine (Docker 24 or newer is fine). Read the sample output first, so you know what a healthy result looks like before you lean on any of it in production.

terminal
$ docker info --format 'driver={{.Driver}} root={{.DockerRootDir}}'
driver=overlay2 root=/var/lib/docker
$ docker run -d --name cow alpine:3.20 sleep 300
$ docker exec cow sh -c 'echo hi > /etc/motd'
$ docker diff cow
C /etc
C /etc/motd
# STATUS: copy-up created a writable-layer change; prefer volumes for heavy writes

Takeaway

The rule to carry out of here: any file your workload rewrites often, or any file bigger than a config file, belongs on a volume rather than the container's writable layer. Watch bytes and inodes under data-root, and run docker diff against anything that claims to be immutable.

Quick check
01Your container appends one line to a 2 GB file that came from an image layer. What does overlay2 do on that very first write?
Correct — overlay2 unions whole files rather than blocks, so the first change to a lower-layer file drags the whole thing up before editing. That is why big-file rewrites stall on write number one.
Incorrect — No. overlay2 works file by file, not block by block. It has nowhere to keep a partial copy, so the whole file has to come up first.
Incorrect — No. Lower layers are strictly read-only and overlay never writes to them. That restriction is the entire reason copy-up exists.
Incorrect — No. The write succeeds. The lower layer being immutable is exactly why the file gets copied into the writable layer first.
02docker info reports Supports d_type: false on an overlay2 host backed by xfs. Why should that worry you?
Incorrect — No. d_type is a correctness feature, not a speed one. Without it overlay2 can mangle data rather than merely slow down.
Incorrect — No. ext4 supplies d_type for free and overlay2 runs fine on it. A false here points at xfs that was formatted without ftype=1.
Correct — Overlay needs d_type to separate real files from whiteout markers, and xfs only reports it when the filesystem was made with ftype=1.
Incorrect — No. That mixes up d_type with metacopy. They are two separate fields in DriverStatus.
03A Dockerfile adds app.tar in one layer, a later RUN rm /app.tar deletes it, and yet the finished image is still enormous. What happened, and what fixes it?
Correct — A delete in a later layer cannot touch read-only bytes underneath it, so the file stays in its original layer and the image stays fat.
Incorrect — No. The whiteout hides the path from the merged view, so the running container cannot see the file. The problem is image size, not what the container sees.
Incorrect — No. A delete writes a whiteout marker rather than a copy-up, and image layers are not the container's writable upperdir.
Incorrect — No. Pruning volumes clears unused volumes. It does nothing about bytes baked into image layers.
Volumes, bind mounts & tmpfsThe container network model &…

Related

What Docker is, and whyDocker for beginnersNamespaces from first principlesAdvanced container securityDocker CLI cheat sheetCheat sheetDocker vs PodmanComparisonDocker interview questionsInterview prep