Filesystem internals: ext4 and XFS

Inodes, extents, journaling and repair.

After a power cut, a settings file exists but is empty; after a disk error, a directory's files turn up in lost+found under numbers; fsck reports an XFS volume clean although it is damaged. Each of these follows from how the filesystem stores files and when it writes them, and this lesson reproduces all three. Ubuntu installs its root filesystem as ext4 and RHEL installs XFS, so those are the two you will build, on loop devices where a file stands in for a disk. You will see how each stores a file (inodes and extents), when written data actually reaches the disk (page cache, writeback, delayed allocation and fsync), what the journal protects and what it costs, and how to check and repair a filesystem that you damage on purpose. The previous lesson covered the VFS objects above the filesystem. Mount options as a security control are covered in the Linux hardening course (optional), in "Mount options for /tmp and friends".

Two filesystems in files

truncate creates sparse files: 256 MiB and 512 MiB long, but ls -s shows 0 blocks allocated until something is written. deploy can run mkfs on a file it owns; on a real disk the same command needs sudo. mke2fs made 65,536 blocks of 4 KiB and 65,536 inodes. An inode is the on-disk record of one file, and ext4 fixes how many there are when the filesystem is made, here one per 4 KiB of space (the small profile in /etc/mke2fs.conf, which mke2fs uses for filesystems from 3 MiB up to 512 MiB). Blocks are grouped into block groups of 32,768, a backup copy of the superblock sits in group 1, and the journal takes 4,096 blocks.

XFS divides the space into allocation groups (agcount=4), each managing its own free space and inodes, so parallel writers do not contend for one allocator. Inodes are 512 bytes (isize) and are created in chunks as files appear; imaxpct=25 only caps how much space they may take. crc=1 means checksummed metadata, reflink=1 allows files to share extents (cp --reflink), and the internal log has 16,384 blocks.

mount -o loop attached each file to the first free loop device (here /dev/loop0 and /dev/loop1; on your machine the numbers depend on what else is attached) and mounted it with the default options. Such a loop device is detached again automatically when the filesystem is unmounted. From here on the kernel treats them like any other disk.

Inodes and extents

stat reads the inode: number 13, one link (one name), 8 blocks of 512 bytes (one 4 KiB block reserved) and four timestamps; Birth is the creation time that ext4 stores. filefrag -v asks the filesystem where the data lives, and the answer is nowhere yet: the flags unknown_loc,delalloc mean ext4 has reserved space but not chosen a location. That is delayed allocation. ext4 and XFS pick the physical blocks when the data is written back, when they know how much there is, so a file written in many small pieces can still get one contiguous extent.

The first sync flushed everything else, so Dirty in /proc/meminfo started near zero. After dd had written 16 MiB and exited, Dirty read about 15.5 MB and Writeback (pages being written out right now) 8 MB. Together that is more than the 16 MiB written, because both counters cover the whole machine, and a loop device writes its image file through the page cache of /var/tmp in turn: the same data is counted once for big on the test filesystem and again for the image file under it. On a real disk you would see only the first. Writeback of big had already started, so ext4 had allocated its blocks, and filefrag shows one extent. After sync, Dirty is close to zero and small has a real block (4171).

debugfs reads the image directly. Inode 14 has flag 0x80000 (it uses extents), Blockcount in 512-byte sectors, timestamps to the nanosecond and a checksum. EXTENTS maps logical blocks 0 to 4095 of the file to physical blocks 38912 to 43007: one extent, a start and a length, where ext2 and ext3 kept a list of every single block.

xfs_bmap counts in 512-byte sectors, so [0..32767] is the whole 16 MiB file in one extent, with the allocation group it sits in and its offset inside that group. The root directory is inode 128 on this XFS filesystem and inode 2 on ext4; XFS inode numbers encode the allocation group and position of the inode.

Writeback and fsync

write() returns once the data is in the page cache. The kernel's flusher threads write dirty pages back when they are older than vm.dirty_expire_centisecs (3000, 30 seconds) or when too much memory is dirty (vm.dirty_background_ratio 10 and vm.dirty_ratio 20 percent). sync() writes back everything. fsync(fd) writes one file's data and metadata and waits until the device confirms them; fdatasync() skips metadata that is not needed to read the data back. Before relying on the sync command, check which call it makes.

Ubuntu 26.04's Rust sync given a file name still calls sync(), flushing every filesystem; GNU sync (installed as gnusync) calls fsync() on that file. The Rust version's sync --data FILE calls fdatasync(). The next test uses dd conv=fsync, which calls fsync() on its output with either implementation. xfs_io -x -c shutdown then stops the filesystem as a power cut would: nothing more is written, not even the journal. It works on ext4 as well as XFS.

Before the shutdown the two files looked the same. After it, any write fails with Input/output error, and the kernel log shows the requested shutdown and the aborted journal (loop0-8 names the journal by its device and the journal's inode, 8; the thread that writes it is jbd2/loop0-8). Unmount, mount again and look.

Mounting replayed the journal (recovery complete). synced has its contents. unsynced exists but is empty: its creation was committed to the journal when the fsync of synced flushed the running transaction, but its data was still waiting for delayed allocation in memory. That is the classic empty-file-after-a-crash pattern, and the fix is in the program: write a temporary file, fsync it, rename it over the old name, and fsync the directory.

Journaling modes

The journal records metadata changes before they are written in place, so after a crash the filesystem replays it at mount instead of scanning everything. ext4 has three modes, set with data= at mount. In ordered, the default, only metadata goes through the journal, but a file's data blocks are written before the metadata that points to them is committed, so a crash never exposes old contents from freed blocks. writeback drops that ordering: faster, and after a crash a file can contain stale data. journal sends data through the journal as well.

lifetime_write_kbytes counts what the filesystem sent to its device. In ordered mode 32 MiB of data cost about 32 MiB of writes plus a little metadata; in journal mode it cost twice that, because every block went to the journal first and to its place later. The ext4 documentation adds that data=journal disables delayed allocation and direct I/O. XFS journals metadata only and has no data= option; its log is the internal log in the mkfs.xfs output.

Checking and repairing

The journal covers crashes. Corruption from a failing disk, a bug or a stray write needs a checker, and the checker needs the filesystem unmounted. To have something to repair, fill a directory, unmount, find the directory's data block and overwrite it with zeros.

Always look before you repair. -n opens the filesystem read-only and answers no to every question; here it stopped at the corrupted directory, because it cannot go further without changing something, and exited 12: 4 (errors left uncorrected) plus 8 (operational error). -f forces a full check even when the superblock says the filesystem is clean.

A modifying repair cannot be undone, and the next run shows it throwing names away. On real data, take a copy of the device first, a block-level image or a snapshot, which is your only way back; e2image (ext4) and xfs_metadump (XFS) save just the metadata when a full copy is too large. Here the image file is the lab copy.

With -y e2fsck salvaged the directory, recreated its . and .. entries, and reconnected the twenty files it found with no name into lost+found; exit status 1 means errors were corrected. Run the check again until it is clean, then mount and look.

docs is empty, and lost+found holds the files under names made from their inode numbers, contents intact. The names lived in the block you zeroed; the inodes and data did not.

XFS does not check itself at mount and has no working fsck: its repair tool is xfs_repair. It also refuses to run while the log holds changes. Simulate a crash again, then attach the image to a loop device yourself so the tools see a block device. losetup --find --show picks the first free loop device and prints its name; keep the name in a shell variable, loop, and use "$loop" in the commands that follow rather than typing a number, because the number depends on your machine. The later commands expect the same shell; in a new one, loop=$(losetup --noheadings --output NAME --associated /var/tmp/st-fs-xfs.img) finds the device again.

xfs_repair stopped at the dirty log. Its advice is the right order: mount to replay the log, unmount, then repair. -L zeroes the log and throws away the metadata changes in it, so it is the last resort for a filesystem that will not mount.

The mount replayed the log and all twenty files are there. Now damage the directory block, whose sector range xfs_bmap showed above, and try the obvious tool first.

fsck succeeded without looking at anything. fsck.xfs is a script that runs xfs_repair only when called with -f from a non-interactive boot (fsck.mode=force on the kernel command line); otherwise it prints this pointer and exits 0. A clean fsck exit on XFS proves nothing.

xfs_repair -n found the bad checksum and the zeroed directory block and listed what it would do, exiting 1. The real run did it, and chose differently from e2fsck: it cleared the damaged directory inode and removed the docs entry from the root, then moved the disconnected files into lost+found under their inode numbers. Either way the data survives and the names do not.

The root filesystem cannot be unmounted on a running system. To check it, add fsck.mode=force to the kernel command line for one boot, from the boot menu (the boot lesson covers the kernel command line). systemd-fsck then checks the root filesystem during boot. On ext4 the default fsck.repair=preen fixes only what is safe to fix without asking, and fsck.repair=yes answers yes to every question unattended, so read the result of a preen boot in the journal before you consider that. On XFS the forced fsck.xfs runs a full xfs_repair straight away, with no read-only pass, so the copy of the disk comes first there. The boot takes as long as the check, which on a multi-terabyte filesystem can be hours with only the console to watch. For anything bigger than a quick check, boot rescue media and repair the unmounted device by hand.

Take the lab apart: unmount both filesystems, detach the loop device you attached yourself (the ones mount -o loop made went away at unmount), and delete the images.

Try this

Make a 64 MiB ext4 image with only 64 inodes (mkfs.ext4 -q -N 64), mount it and create empty files in a loop until one fails. Predict first: touch fails at file 53 with No space left on device, df -h still shows the filesystem 1% used, and df -i shows all 64 inodes used: your 52 files and 12 the filesystem uses itself (reserved inodes and lost+found among them). An XFS filesystem would keep creating inodes until its space or imaxpct ran out. Clean up with umount, rmdir and rm.

Takeaway

Treat data as durable only after fsync has returned, and treat a filesystem check as a sequence: look with -n, copy the device, repair it unmounted (replaying an XFS log by mounting first), and check again until it is clean.

Related