Disks, filesystems and mounts

lsblk, findmnt, fstab and full disks.

A server keeps its files on one or more disks, but you never work with a disk directly: you see a single directory tree that starts at /. This lesson shows how to find which disks and filesystems a server has and where each is attached, how to add a filesystem that is mounted at every boot without putting the boot or its services at risk, and what to check, in order, when the disk is full, one of the most common causes of an outage.

Disks, partitions and filesystems

Three layers are involved. A block device is a disk as the kernel presents it: a long row of numbered blocks that can be read and written. A partition is a slice of a disk that behaves like a disk of its own. A filesystem is the structure written onto a device that turns its blocks into files and directories with names, owners and permissions. lsblk -f shows all three:

vda is the lab machine's virtual disk (virtio, the disk driver most virtual machines use); physical servers and many clouds show sda or nvme0n1. The tree lines show its three partitions. FSTYPE is the filesystem on each: ext4 for / and /boot, vfat for /boot/efi, the partition the firmware reads at power-on. LABEL and UUID are names stored inside the filesystem, FSAVAIL and FSUSE% show free space, and MOUNTPOINTS where each filesystem appears. vdb (cidata) is a small disc the virtualisation tool attaches with first-boot settings; a real server does not have it.

findmnt answers the reverse question: which filesystem holds this directory? Ubuntu formats its root filesystem as ext4, while RHEL and Rocky Linux use XFS:

Both are mature journaling filesystems: they record changes to their own bookkeeping (the metadata) in a journal before making them, so a crash does not leave the structure half-updated. Day to day the difference is mainly the tools (e2fsck and resize2fs for ext4, xfs_repair and xfs_growfs for XFS). OPTIONS lists the mount options in force; rw means read-write.

One tree, built from mounts

Mounting attaches a filesystem to a directory, its mount point, so that the directory shows the filesystem's contents. The root filesystem is mounted at / first and every other filesystem is attached somewhere below it. df -hT lists what is mounted, with sizes in K, M and G (-h) and each filesystem's type (-T):

Only three of these rows are disk partitions. The tmpfs rows are filesystems kept in RAM, empty at every boot: /run, /dev/shm, /tmp on Ubuntu 26.04, and small ones under /run/credentials/ in which systemd hands settings or secrets to a service (the rows left out are more of these). efivarfs exposes the firmware's variables. To see real storage only, leave out the in-memory types:

/dev/vdb is the virtualisation tool's settings disc again. Use% is the column to watch: a filesystem at 100% refuses new data from ordinary programs.

Mount options change how the kernel treats a whole filesystem. nosuid ignores set-user-ID and set-group-ID bits and file capabilities, nodev ignores device files, and noexec refuses to run programs stored there. On Ubuntu 26.04, /tmp gets the first two:

An ordinary user cannot create a set-user-ID root file, so nosuid here guards against a SUID program left or planted in this world-writable place by a privileged process or an intruder who briefly had root ("SUID, SGID and the sticky bit" explains more). size= caps /tmp at half the memory. RHEL 10 has no tmpfs on /tmp, and the hardening course applies these options to other directories.

Adding a filesystem

A new disk appears as a new, empty block device, such as /dev/sdb or /dev/nvme1n1. The lab has no spare disk, so it makes one from a file: a loop device is a block device whose blocks live in an ordinary file, and from then on the commands are the same as for a real disk.

truncate created a sparse file: the 0 at the start of the ls -s line is the space really allocated, because blocks are only taken as data is written. losetup --find --show attached the file to the first free loop device and printed its name (yours may have another number), which has no filesystem yet. mkfs.ext4 writes a new, empty ext4 filesystem onto a device, and -L gives it a label:

The output gives the size in 4 KiB blocks, the number of inodes (an inode is the record of one file's owner, permissions and data location) and the journal size. Formatting destroys what the device held, so check the name with lsblk -f first. Now create a mount point, mount the filesystem and look at it:

The filesystem holds 56M rather than 64M because the journal and the inode tables take space. Avail is also smaller than Size minus Used: ext4 reserves 5% of the blocks for root, so that services running as root keep working after ordinary users have filled the disk. The top directory of a new filesystem belongs to root, so hand it to the account that will use it:

In sudo chown deploy: /mnt/ess-data, the colon with no group after it sets the group to the user's login group. lost+found is where e2fsck puts pieces of files it recovers after a crash.

Mounting at boot: /etc/fstab

A mount command lasts until the next reboot. Filesystems that must come back at boot are listed in /etc/fstab, one per line. systemd reads the file early at boot, and again whenever it is reloaded, and turns each line into a mount unit. On the Ubuntu cloud image it contains:

Each line has six fields: what to mount, where, the filesystem type, the mount options, a field for the old dump backup tool that is nearly always 0, and the order in which fsck checks filesystems at boot (1 for the root filesystem, 2 for the others, 0 for never). The first field names each filesystem by its label rather than as /dev/vda1: the kernel names devices in the order it finds disks, so names can change, while a label or UUID moves with the filesystem. RHEL uses UUIDs:

A mistake in /etc/fstab is one of the few configuration errors that can stop a server from booting. systemd waits up to 90 seconds for each listed device, and when a filesystem the boot requires cannot be mounted, local-fs.target fails and systemd starts emergency mode, a root shell on the console. Without console access, the machine is then out of reach. nofail makes a filesystem wanted rather than required, so the boot carries on without it. Use it for data disks the system can boot without, but together with the service setting in the next section: on its own it moves the problem to the services that use the disk.

The safe routine is to find the filesystem's UUID, add the line, and check the file before anything uses it. The lab works on a copy of /etc/fstab, so nothing it does can affect the next boot:

Here are the last two lines of the copy after adding the new line with an editor (fields can be separated by any spaces or tabs):

findmnt --verify reads an fstab file and checks every line: that the source can be found, that the mount point exists and that the type is one the kernel supports. --tab-file points it at a file other than /etc/fstab:

A UUID with one character wrong, on a line without nofail, is exactly the mistake that stops a boot, and the check catches it:

"Unreachable on boot required source" means the boot would wait for a device that will never appear. Last, prove that the line works by mounting from it. mount --fstab reads the copy, and given only a mount point it takes everything else from the matching line:

On a real server the same routine runs on /etc/fstab itself: edit it with sudoedit /etc/fstab, run sudo findmnt --verify, run sudo systemctl daemon-reload so that systemd rebuilds its mount units from the file, then sudo mount /mnt/data (or sudo mount -a for every line) to prove the line before the next reboot tests it for you.

When a nofail disk is missing

With nofail, a missing disk no longer stops the boot, but the services that use it still start, and the mount point is then an empty directory on the root filesystem. A program that runs as root, or finds its directory in place, starts a fresh data set there, fills /, and leaves two copies that disagree once the disk is back. Here is a stand-in database, a service that appends the date to a file under /mnt/ess-data/db:

[Unit]
Description=Stand-in database that keeps its data on /mnt/ess-data
 
[Service]
Type=oneshot
ExecStart=/bin/sh -c 'mkdir -p /mnt/ess-data/db && date >> /mnt/ess-data/db/data'

The mount unit systemd makes from an fstab line is named after the path, here mnt-ess\x2ddata.mount (\x2d stands for -). The lab created that unit directly under /run/systemd/system, to leave the real /etc/fstab alone, from the line LABEL=ess-data /mnt/ess-data ext4 defaults,nofail 0 2; to try it yourself, add that line to /etc/fstab, run sudo systemctl daemon-reload, and remove it afterwards. Now the disk disappears and the service starts:

findmnt -T names the filesystem that holds a path: the data went to /dev/vda1, the root filesystem. Remove it, and tie the service to the mount with a drop-in (sudo systemctl edit ess-storage-db opens an editor for it):

RequiresMountsFor= makes the service require, and start after, every mount unit needed to reach that path. With the disk still missing, the service now refuses to start:

systemctl status also accepts a mount point instead of a unit name. The start waited 90 seconds, systemd's default time for a device to appear, then the mount failed and the service with it, and nothing was written. When the disk is back, starting the service mounts the filesystem first:

So the rule is nofail on the disk and RequiresMountsFor= on every service that keeps data on it.

When the disk is full

When a filesystem fills up, writes fail with No space left on device. On the lab filesystem a stand-in application, a small service running as deploy, writes its log to /mnt/ess-data/logs/app.log, and the log has filled the filesystem:

Work down from the filesystem to the directory. du adds up the files under a directory: -x keeps it on one filesystem, -h prints readable sizes and --max-depth=1 gives one line per subdirectory:

The log is the culprit. Deleting it looks like the fix, but the space does not come back:

df still reports 100% while du finds 24K. A file's space is released only when its last name is removed and no process has it open, and the application still has the log open. lsof +aL1 lists open files with no name left (a link count below 1) on the filesystem you give it:

The process app.sh, running as deploy, holds the file on descriptor 11, open for writing (11w). SIZE/OFF is the 53 MB it still occupies and NLINK 0 means no name points to it. Restarting the service makes it let go. If the restart has to wait, empty the file through the process's own descriptor: /proc/PID/fd/11 still leads to the open file although it has no name, and pgrep -x app.sh supplies the PID:

The space is back at once, but the process still holds a deleted file. Restart it when you can, and it opens a new log:

Better still, never delete a log that a program has open: truncate -s 0 FILE empties it in place, and logrotate rotates logs on a schedule (its copytruncate option loses the lines written between the copy and the truncation). Emptying in place works cleanly only for a program that writes in append mode, as this one does; one that keeps its own write position carries on at the old offset, and ls -l still shows the old size.

On a real server the search starts at /, where -x matters most: without it du would also wander into /proc, /run, /tmp and every other filesystem mounted below the root. sort -h orders the readable sizes and tail keeps the largest:

There is a second way to run out of space. ext4 creates a fixed number of inodes when the filesystem is made (16384 on this small one), and a program that writes huge numbers of tiny files, such as a cache, can use them all while blocks remain. df -i counts inodes instead of blocks:

The for loop runs touch for f1, f2 and so on up to f20000, and || break stopped it at the first failure: No space left on device, with 51M still free according to df -h. IUse% 100% is the real reason. XFS creates inodes as it needs them, which makes it much less prone to this.

To detach a filesystem, unmount it. That fails while any process still uses it, and fuser -vm names the processes:

In the ACCESS column, F is a file open for writing and c would be a process whose current directory is inside the mount, which is the usual culprit when the process is your own shell. Stop or move the process, then unmount; losetup -d detaches the loop device, and the empty losetup -a confirms nothing is attached.

Try this

Build the loop filesystem from this lesson, mount it, give it to your account and put a small script on it: printf '#!/bin/sh\necho hello from the new disk\n' > /mnt/ess-data/hello.sh, then chmod +x /mnt/ess-data/hello.sh. Run it, remount with sudo mount -o remount,noexec /mnt/ess-data and run /mnt/ess-data/hello.sh again: it fails with Permission denied, and findmnt -no OPTIONS /mnt/ess-data shows why. Now run sh /mnt/ess-data/hello.sh: it prints its message, because noexec stops the kernel from executing files there but not a program that reads the file as its input. Remount with exec, then clean up with sudo umount /mnt/ess-data, sudo losetup -d /dev/loop0 and rm -r /var/tmp/ess-storage (and remove any line you added to /etc/fstab).

Takeaway

Check every change to /etc/fstab with findmnt --verify, and give a data disk nofail only together with RequiresMountsFor= on the services that use it. When a disk fills up, compare df with du: if they disagree, a deleted file is still open, and lsof +aL1 names the process to restart.

Related