Where Docker keeps data

The containerd image store, snapshotters and the classic graph drivers.

This lesson answers an operational question: where a Docker host keeps images and container filesystems, and what each layout costs on disk. On a host installed fresh with Docker 29 the images are not under /var/lib/docker at all, so a du -sh /var/lib/docker that reports a few hundred megabytes on a full disk proves nothing. Which directory holds them depends on which image store the daemon runs, and that depends on how the host was installed. Start by asking the daemon:

overlayfs with driver-type io.containerd.snapshotter.v1 is the containerd image store, the default for fresh Docker Engine 29 installs. Image content and container filesystems belong to containerd and live under /var/lib/containerd. A host upgraded from Docker 28 or older reports overlay2 here instead, with lines such as Backing Filesystem and Supports d_type: that is the classic graph driver, which keeps everything under /var/lib/docker and stays in place after the upgrade until someone migrates the host. You will meet both for years, so the rest of this lesson takes them apart side by side.

A container's filesystem on the containerd store

Build the lab image, whose Dockerfile adds two files of random bytes (100 MB and 50 MB) so that copies are easy to see in sizes and timings, and start a container from it:

FROM alpine:3.22
# two files of random bytes, big enough that a copy-up shows in timings and sizes
RUN head -c 100000000 /dev/urandom > /data.bin \
 && head -c 50000000 /dev/urandom > /meta.bin
CMD ["sleep", "infinity"]

Old runbooks read .GraphDriver.Data.UpperDir to find a container's files on the host. On the containerd store GraphDriver is null, and the new Storage field only names the snapshotter. The paths come from the mount itself. dockerd mounts each container's root filesystem at /var/lib/docker/rootfs/overlayfs/<id> ("Engine architecture on Docker 29" found it the same way), and containerd can describe the snapshot behind it:

A snapshot is containerd's name for one filesystem layer on disk. The overlayfs snapshotter stores each one as a numbered directory, and an overlay mount stacks them. lowerdir lists the read-only snapshots, top first: 1741 is Docker's init layer (placeholder files such as /etc/hosts and /etc/resolv.conf that Docker fills in per container), 1738 is the image's RUN layer and 1 is alpine's base layer, which every image built on this alpine shares. upperdir is the container's writable layer. ctr shows the same chain from containerd's side: an Active (writable) snapshot named after the container ID, whose parent is the -init snapshot. The numbers are allocation order on this VM; yours differ.

Copy-on-write, and what the first write costs

Lower layers are never modified. The first time a container changes a file that exists in a lower layer, overlayfs copies the whole file into the upper directory and changes the copy. docker ps -s shows the writable layer's size, so the cost is visible without sudo:

Appending two bytes to /data.bin took 0.17 s the first time and nothing the second, and the writable layer went from 4.1kB to 100MB: the entire file was copied up before one byte changed. Timings depend on the disk; the size jump does not. virtual adds the image's size. Each container pays this separately, because each has its own upper directory, so ten containers that touch the same large file store ten copies.

Since Linux 4.19, overlayfs can copy only metadata for changes such as chmod (the metacopy feature), but it is off unless the module parameter or the metacopy=on mount option enables it, and Docker does not set the option. The module parameter on this kernel:

N means off, and a permission change on the 50 MB meta.bin copied all of it: 150MB now. This is why the rule from "Volumes and bind mounts" in Docker for beginners matters in practice. Database files, logs, caches and anything rewritten often belong on a volume, which bypasses the overlay mount entirely.

Deletes are whiteouts

docker diff reports the change set: A added, C changed, D deleted. On disk, the upper directory holds the two copied-up files, the new note.txt, and etc/issue as a character device with device number 0, 0. That is an overlayfs whiteout: it tells the merged view to hide /etc/issue from the lower layer, whose bytes are untouched. The same mechanism in an image build is why deleting a file in a later RUN does not shrink the image; "Slimming images and layer hygiene" in Advanced container security deals with that. Files in a writable layer also count as inodes, not only bytes, and a filesystem can run out of inodes with gigabytes free, so watch both on the filesystem that holds /var/lib/containerd:

Here both are far from full. On a host that runs many images with tens of thousands of small files (language runtimes, node_modules), IUse% is the number to alert on. Clean up the main VM part:

Inside /var/lib/containerd

The rest of the lesson runs on secopslog-docker-sec, where your login is not in the docker group, so every command uses sudo. Fetch the lesson files into ~/lab/storagedrivers on that VM too (the lesson files box shows the command) and work there. A recreated sec VM may not have alpine yet, so pull it first. Then look at where containerd keeps its data and how that is configured:

containerd config dump prints the effective configuration: root holds persistent data, state (under /run) holds sockets and runtime state that disappear at reboot. The file Docker's packages install, /etc/containerd/config.toml, has both lines commented out, so the defaults apply. It also has no version line, which is why containerd 2.x warns that it migrated the config from version 0 in memory; the warning is harmless, and containerd config migrate prints an upgraded file if you want to silence it.

The content store is content-addressed: every blob is a file named after its own SHA-256 digest. The image ID that Docker 29 prints for alpine:3.22 is such a digest, and the file with that name is the image index, a small JSON document listing one manifest per platform ("Image anatomy: index, manifest, config and layers" reads it in detail). unknown entries are attestation manifests, not a platform.

Two copies of every layer

The containerd store keeps each layer twice: compressed in the content store, as pulled or built, and unpacked in a snapshot that containers can use. Measure what one 50 MB layer of random bytes (which does not compress) costs:

FROM alpine:3.22
# 50 MB of random bytes: compression cannot shrink it, so disk growth is easy to read
RUN head -c 50000000 /dev/urandom > /blob.bin

The content store grew by 48 MB: the layer as a compressed blob, which for random bytes is as large as the data. The snapshot directory grew by 105 MB, about twice the layer, because the build left more than one snapshot of it. docker buildx du reports the build-cache record for the RUN step at 100MB and counts everything under Shared, meaning the cache and the image use the same snapshots, so docker builder prune cannot free them while lab-blob:1 exists. That is about 150 MB of disk for a 50 MB layer. An image you pull rather than build costs the compressed blob plus one unpacked snapshot; Docker's documentation gives that double storage as the reason the containerd store needs more disk than the legacy drivers. Size disks for it, and expect different absolute numbers on your VM, which holds other images.

The legacy graph driver, side by side

Switch this VM to the legacy store with the procedure from "Configuring the daemon safely": back up the current daemon.json (an empty {} stands in when there is none, as on this VM), merge the new key into the backup with jq, validate, restart, check. Merging instead of overwriting keeps whatever the host already had in the file, such as the MTU settings the lab kit writes on VPN machines. Images do not move between the stores, so save one first to load on the other side:

{
  "features": { "containerd-snapshotter": false }
}

docker info now reports overlay2 and the classic graph-driver details, and the image list was empty until docker load put alpine into the legacy store. Build the same 50 MB image here:

/var/lib/docker grew from 10 to 58 MB, 48 MB for the same layer, and containerd's directories did not move. The legacy store keeps one unpacked copy and no compressed blob, which is also why it reported CONTENT SIZE 0B in "Inspecting, exporting and cleaning up images". Now look at a container on this store:

On the legacy store .GraphDriver.Data names every directory: LowerDir (here the -init layer and alpine), UpperDir, MergedDir and WorkDir, all under /var/lib/docker/overlay2/<id>/. The whiteout looks exactly as before, because the kernel's overlayfs does the work on both stores. Each layer directory holds diff (the layer's files), link (a short name), lower (the short names of its parents), and for containers merged and work. The short names under l/ are symlinks that keep the overlay mount options under the kernel's page-size limit when an image has many layers. Remove what you created on this store before you leave it:

Switching back, and the data left behind

Restoring the backup, which has no features key, does not bring the containerd store back once legacy data exists: the daemon finds /var/lib/docker/overlay2 and treats the host as an upgraded one ("Configuring the daemon safely" and "Inspecting, exporting and cleaning up images" both show that trap). Ask for the containerd store explicitly:

{
  "features": { "containerd-snapshotter": true }
}

The containerd store's images are visible again, including the lab-blob:1 built earlier, and /var/lib/docker/overlay2 is still there. Switching stores hides the other store's data; it never deletes it. On a real host upgraded from Docker 28 this is the disk-usage trap of the migration: after "containerd-snapshotter": true, every old image and container still occupies /var/lib/docker/overlay2 and /var/lib/docker/image/overlay2, invisible to docker system df. Migrate in this order: list what the host needs, switch, pull or rebuild those images (or docker save them before and docker load after), recreate the containers, and only then remove the old directories with Docker stopped. Docker also has an experimental automatic migration ("containerd-migration": true in features) that switches only when no containers exist and the images are below a size threshold; the docs recommend starting fresh instead. This lab VM never held anything important in the legacy store, so it removes the directories at once and restores the backup:

One daemon option rules the containerd store out entirely: userns-remap disables it, and a host that needs user-namespace remapping runs the legacy store ("User-namespace remapping" in Advanced container security covers the trade-off). Rootless Docker keeps its own data under the user's home directory; "Rootless Docker" in Advanced container security covers it.

Moving containerd's root

data-root in daemon.json moves only dockerd's directory, as "Configuring the daemon safely" showed. To put images and container filesystems on another disk, move containerd's root as well. Stop or remove every container first, and check that docker ps -q prints nothing: with live-restore on, stopping docker.service leaves containers running on their shims, and the copy would race their writes to the upper directories. Then both daemons must be stopped, and the copy must keep hard links, ACLs and extended attributes, which overlay layers use (rsync -aHAX). Plan it as a maintenance window. In production /srv/containerd would be the mount point of the new disk:

After the restart the images are still listed, and the new container's upperdir is under /srv/containerd. Until you delete it, the old /var/lib/containerd is your rollback, with one cost: switching back loses every image pulled and every container change made since the move. Once the new location has run for a while, remove it to free the space. On a real disk, make sure containerd never starts before the disk is mounted, or it creates a fresh, empty root on / and fills the root filesystem; a systemd drop-in does that:

[Unit]
RequiresMountsFor=/srv/containerd

Each daemon needs its own root: never point two containerd instances (or two hosts over NFS) at the same directory. This lab moves back:

d_type and XFS

Overlayfs needs the backing filesystem to report each directory entry's file type (d_type); it uses it to recognise whiteouts. ext4 always supports it. XFS supports it only when formatted with ftype=1, which mkfs.xfs has defaulted to for years, but old RHEL and CentOS 7 era hosts were often formatted with ftype=0. The classic overlay2 check is what prints Supports d_type: true in docker info. To see what happens without it, the lab formats an XFS image with ftype=0 (it needs the deprecated V4 format to allow that) and points each store at it:

Nothing failed, and that is the problem. With ftype=0 under data-root, dockerd did not select overlay2, tried fuse-overlayfs (not installed, the one error line) and settled on vfs, which logs at info level only. vfs has no copy-on-write: every layer and every container is a full copy of everything below it, so disk use multiplies and container starts slow down. On this store the daemon gives up on overlay2 without a d_type message in its log, so the place to catch it is docker info after provisioning: alert on any driver other than the one you expect. Docker's deprecation notes add the other half: since Engine 23 a daemon told explicitly to use overlay2 (storage-driver in daemon.json) on such a filesystem refuses to start; automatic selection, as here, skips it. Stop the daemon and restore the backup before the next test, so dockerd starts on its defaults again:

Now the containerd store on the same filesystem. Point containerd's root at the XFS mount, restart containerd, and start dockerd:

containerd itself starts, but its overlayfs snapshotter plugin is in state error, and the kernel log says why: upper fs needs to support d_type. dockerd then refuses to start. Its journal names the cause in a warning (does not support d_type ... reformat with ftype=1), and the line that ends the start is an unrelated-looking buildkit error about a missing leases service, which is what you would see first in systemctl status. So the containerd store fails loudly where the legacy store degrades quietly. The fix is the same for both: check xfs_info <mountpoint> | grep ftype before putting Docker or containerd data on XFS, and reformat with ftype=1 (the default in current mkfs.xfs) if it says 0. The restore step puts containerd's original config back, removes the test filesystem and starts both daemons.

Clean up the lab directory on the sec VM:

Try this

Work through “d_type and XFS” yourself on a sandbox you can throw away, following the commands above in order. Then break one step deliberately and re-run, so you have seen the failure before it finds you.

Takeaway

If you keep one thing from where docker keeps data, keep “d_type and XFS”. Decide now which check you will run when this shows up on a live system, and write it somewhere your team will find it.

Related